yashmayya opened a new pull request, #19759:
URL: https://github.com/apache/pinot/pull/19759
Stacked on #19757 (phase 1). Only the last commit, 715997d105, is new:
review that commit.
## Problem
A query can filter on an IdSet of millions of ids, for example
`IN_ID_SET(col, '<16 MB literal>') = 1`. With a literal that large, both
engines have three problems:
- **Parsing is quadratic.** The generated JavaCC char stream grows its
buffer by 2,048 chars at a time, and copies the buffer every time. To parse a
literal of 1M chars, the broker allocates 2.6 GB. A 6 MB literal takes 2.5 s
and 84 GiB.
- **Each segment decodes its own copy of the IdSet.**
- **The message size limits are fixed.** MSE servers accept dispatched plans
of up to 64 MB. The broker gRPC query endpoint accepts requests of up to 4 MB,
which is the gRPC default. No config changes these limits.
## Changes
- New `GrowingCharStream`: the SQL parser reads through this subclass of the
generated `SimpleCharStream`. When a token outgrows the 4,096-char buffer, the
buffer grows one time, to the length of the SQL.
- A literal of 1M chars now parses in 13 ms with 12 MB.
- Tokens of up to 2,048 chars never grow the buffer, so most queries
allocate as before. For example, a 1M-value IN list allocates 526 MB, as before.
- The cost is in a long SQL with one longer token. A token of more than
2,048 chars can grow the buffer, and one of more than 4,096 chars always does.
For example, a 5,000-char literal in an 8M-char IN list query allocates an 80
MB buffer. Doubling the buffer would avoid that, but each long literal would
then allocate 2 to 3 times as much.
- `InIdSetTransformFunction` decodes the IdSet one time per server request,
and all segments use it. IdSet lookups only read, so many threads can share it.
- `TransformFunction` gets an `init` overload that takes the
`QueryContext`. The factory calls it, and its default calls the existing `init`.
- `QueryContext` now keeps its shared values in its own map. Before, a
`ClassValue` kept them: a value stayed in the cache of its class after the
query, until about 21 later queries used the same class. For IdSets, that kept
tens of MB for each old query. This also frees the shared literals and IN lists
of other queries sooner. `MemoizedClassAssociation` had no other user, so it is
deleted.
- New configs for the gRPC message size limits. The defaults do not change.
Each value must be positive, or the server or broker fails at startup.
| Config | Default | Limits |
|---|---|---|
| `pinot.query.multistage.query.server.max.inbound.message.size.bytes` | 64
MB | Plans that brokers dispatch to MSE servers. The plans carry the literals. |
| `pinot.query.multistage.dispatch.channel.max.inbound.message.size.bytes` |
64 MB | Messages that MSE servers send to brokers, for example stats and
`EXPLAIN` plans. |
| `pinot.broker.grpc.max.inbound.message.size.bytes` | 4 MB | Requests to
the broker gRPC query endpoint. |
No change to the wire protocol or to serialized formats.
## Results
- With a literal of about 16M chars (an IdSet for 10M ids at 1% false
positives), the integration test query takes about 0.7 s in each engine. MSE
planning of a query with this literal takes about 0.4 s and 352 MB.
- With a literal of 32M chars (20M ids at 1%), the query takes about 1.3 s
in each engine through the broker gRPC endpoint, with
`pinot.broker.grpc.max.inbound.message.size.bytes` set to 64 MB. (I ran this
one time locally. It is not in the tests.)
## Known limits
- **HTTP queries:** the broker parses the request body with Jackson's
default limit of 20M chars for a string. So over HTTP, the SQL can have a
literal of up to about 20M chars: an IdSet of about 12M ids at 1% false
positives, or 16M at 3%. Larger sets must use the gRPC endpoint. The same
Jackson limit applies to clients that read an `IDSET` result. This is not new,
and SSE has the same limit.
## Tests
- `GrowingCharStreamTest`:
- The new stream reads the same tokens as the generated stream, with the
same line and column numbers, in the lexical state of the SQL parser. The SQL
has string literals, quoted identifiers, numbers and comments of up to 100K
chars, at offsets that make the circular buffer wrap around. The test also
makes sure that each SQL has its long token.
- Only a long token grows the buffer, and only one time.
- `CalciteSqlParserTest`: a literal of 2M chars parses with less than 50
bytes per char. The generated stream allocates 10.8 GB for it. An empty query
fails as before.
- `InIdSetTransformFunctionTest`: the segments of one query share one
decoded IdSet. Another IdSet, or another query, gets its own. The decoded IdSet
is collected after the query.
- `QueryServerTest`, `DispatchClientTest` and `BrokerGrpcServerTest`: each
config sets the limit of a real gRPC server or channel. A larger message fails
with `RESOURCE_EXHAUSTED`. Zero and negative values fail.
- `MultiStageBrokerRequestHandlerTest`: the broker gives the dispatch config
to its dispatcher.
- `IdSetIntegrationTest`: both engines build an IdSet literal of about 16M
chars and filter with it.
Each new test fails when its change is reverted. I did this check for each
change.
## Next phases
- **Phase 3, single-query forms in MSE:** `IN_SUBQUERY(expr, 'sql')` and
uncorrelated `IN_ID_SET(expr, (SELECT IDSET(...) ...))`. The broker runs the
subquery first, so the set never goes through the parser, the client or the
HTTP limit.
- **Phase 4 (optional, measure first):** one shared Bloom filter per query
for `IDSET`, and shorter literals in `EXPLAIN` (#8009).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
