raghav-reglobe opened a new pull request, #70465:
URL: https://github.com/apache/doris/pull/70465
### Background
An Arrow Flight SQL client calls GetFlightInfo on an FE and gets back
endpoints, each a ticket plus the locations where the client redeems it with
DoGet. A query result produced by BEs is held on a BE, and its endpoint names
that BE (or `public_host` / `arrow_flight_sql_proxy_port` from be.conf, for
clients outside the cluster network). A result the FE holds itself (a statement
it answers without a BE, such as SHOW, USE or SET, and every metadata request:
GetCatalogs, GetDbSchemas, GetTables, GetTableTypes, GetSqlInfo, ...) gets its
endpoint in `DorisFlightSqlProducer.getFlightInfoForSchema`, which always names
the FE's own address: the host `FrontendOptions` reports and
`arrow_flight_sql_port`.
### What problem does this PR solve?
Issue Number: #62538 (the same report, on 4.0.5 in Kubernetes; closed as
stale, not fixed; tracked in #65615)
Related PR: None
PR Type: Bugfix
Problem Summary:
A client that reaches the FEs through a load balancer or a proxy often
cannot resolve or reach the FE's own address. It logs in and reads BE-held
query results through `public_host`, but every FE-answered call fails: the ADBC
Flight SQL driver reports `name resolver error: produced zero addresses`, the
Flight SQL JDBC driver `UNAVAILABLE: unresolved address`, so a BI tool cannot
list databases or tables. A public address cannot stand in for the FE's,
because the result exists only in the session on the FE that answered, and a
load balancer in front of several FEs would send the DoGet to any of them. This
PR adds a mutable FE config, `arrow_flight_fe_result_empty_location` (default
`false`, today's behaviour). Set to `true`, the FE sends these endpoints with
no location, which the Flight protocol defines as redeemable only on the
service that issued the ticket, so the client fetches over the connection it
already holds to that FE. Existing users see no change after an upgrade until
they set it.
### Details
- **Frequency:** Every FE-answered call, every time, from a client that
cannot resolve or reach the FE's own host (typically any client outside the
cluster network).
- **Root cause:** `getFlightInfoForSchema` always puts the producer's
`location` (this FE's host and `arrow_flight_sql_port`) in the endpoint. The BE
side has `public_host` / `arrow_flight_sql_proxy_port` for this case; the FE
side has nothing, and one public address could not serve several FEs anyway
(see above).
- **Why existing tests missed it:** The unit tests and the regression suites
reach the FE directly, where its own address resolves.
- **Fix:** `getFlightInfoForSchema` builds its endpoint with a new
`localResultEndpoint(ticket)`: `new FlightEndpoint(ticket)` (no location) when
`Config.arrow_flight_fe_result_empty_location` is true, `new
FlightEndpoint(ticket, location)` otherwise. The config is read for each
request, so `ADMIN SET FRONTEND CONFIG` applies from the next GetFlightInfo
without a restart. It is per FE (not master-only), so set it on every FE, and
in fe.conf to keep it across restarts.
- **Other paths and branches:** Endpoints of BE-held results are unchanged,
with the config on or off.
### Testing
- **Unit tests**, `DorisFlightSqlProducerTest` (+3):
- A metadata request's endpoint names the FE by default and carries no
location with the config on. The check follows a change between two requests on
the same producer, with the ticket unchanged.
- An FE-answered statement's endpoint behaves the same way.
- A BE-held result keeps the BE's address with the config on.
These ran in an FE unit-test run of 65 classes, every Arrow Flight test
class among them (`DorisFlightSqlProducerTest` 15/15; 701 tests, all passing).
That run was on a branch where `DorisFlightSqlProducer.java` and
`DorisFlightSqlProducerTest.java` are byte-identical to this PR's. This exact
head is left to CI.
- **Manual, one FE, external client.** Clients ran on a machine where the
FE's host does not resolve, reaching the FE and the BE through port forwards:
adbc-driver-flightsql 1.13.0, the Flight SQL JDBC driver 15.0.2 and 19.0.0, and
raw pyarrow Flight calls.
- Before: every FE-answered call failed with the errors above. Those calls
were SqlInfo, SHOW DATABASES, USE, SET, get_objects / getCatalogs / getSchemas
/ getTables / getTableTypes and getColumns. BE-held SELECT results were fine.
- After setting the config to `true` at runtime: all of those calls passed
on every client, with `locations=[]` and the DoGet going over the current
connection. BE-held results were unchanged.
- Setting it back to `false` restored the old behaviour at once.
- An invalid value is refused (`type mismatch`).
- **Manual, three FEs.** The FEs sat behind a TLS-terminating TCP load
balancer, with the config on every FE. A Tableau workbook over the Flight SQL
JDBC driver, run from outside the cluster network, connects and runs its
queries.
### Notes for Reviewers
- **Why no location rather than `Location.reuseConnection()`**
(`arrow-flight-reuse-connection://?`): #67578 (A2) names the reuse URI as the
standard answer to the reachability problem behind #62538. For a result the FE
holds, an empty list means the same thing, "redeem on the service that issued
the ticket", and it is the older form. The clients tested above (ADBC 1.13.0,
Flight SQL JDBC 15.0.2 and 19.0.0) all redeem it on the current client; JDBC
does so in `ArrowFlightSqlClientHandler.getStreams`. A client that does not
recognise the reuse scheme would instead dial its empty host. If you prefer the
reuse URI, switching to it is a one-line change in `localResultEndpoint`.
- **Why the default stays `false`:** a client that assumes `locations[0]`
exists would break on an empty list. I am happy to flip the default if you
prefer the spec form everywhere.
- **Why not an FE `public_host`, as #62538 asks:** the result exists only in
the session on the FE that answered. A single public address in front of
several FEs would send the DoGet to whichever FE the load balancer picks.
- **Out of scope:** a Flight SQL JDBC driver below 15.0.0 ignores endpoint
locations altogether, so it cannot read BE-held results either way.
### Release note
New FE config `arrow_flight_fe_result_empty_location` (mutable, default
false). Set it to true when Arrow Flight SQL clients reach the FEs through a
load balancer or a proxy. FE-answered results (SHOW, USE, SET and the metadata
calls such as GetCatalogs and GetTables) are then fetched over the client's own
connection instead of from the FE's internal address.
### Check List (For Author)
- Behavior changed: No (opt-in; default unchanged)
- Does this need documentation: Yes, a line on the Arrow Flight SQL
connection page next to `public_host` / `arrow_flight_sql_proxy_port`, in a
doris-website follow-up once this is settled.
- Pre-submission status: Partially verified: the unit tests ran on a branch
with byte-identical copies of the two Arrow Flight files (Config.java there
differs only in unrelated entries); this head's own run is CI's.
### AI Disclosure
- Tool: Claude Code (Claude Opus 5.5)
- Generated: the code, the unit tests and this description
- Human review: the author ran the change on a production cluster (three FEs
behind a load balancer) and confirmed an external Tableau client works with the
config on
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
