yashmayya opened a new pull request, #19757:
URL: https://github.com/apache/pinot/pull/19757
## Problem
`IDSET` and `IN_ID_SET` only partly work in the multi-stage engine (MSE):
- `IDSET` works on INT columns only. Every other type, and every Bloom
filter set, fails with `Cannot deserialize BloomFilter from ByteBuffer not
backed by an accessible byte array` (or the same error for
`Roaring64NavigableMap`). MSE hands intermediate results over in read-only
buffers, which have no accessible array.
- `IN_ID_SET` works only when the planner pushes it into a leaf stage. In a
filter over a join, in `HAVING` or in a join condition it fails with
`Unsupported function: INIDSET`, because it has no scalar implementation.
- When the set and the column have different types (for example a LONG set
on an INT column), both engines fail with an exception that has no message, or
with a `ClassCastException`.
## Changes
- `BloomFilterIdSet` and `Roaring64NavigableMapIdSet` read from any
`ByteBuffer`. A buffer without an accessible array is read through a stream
over a duplicate. A buffer with an array keeps the current path.
- New `inIdSet` scalar function, for MSE stages that are not leaf stages.
Leaf stages keep `InIdSetTransformFunction`, because transform functions take
precedence. Each call site gets its own instance, which caches the deserialized
set.
- `IdSets.validateValueType` checks the value type against the set once, in
both the transform function and the scalar function. A mismatch now fails with,
for example, `Cannot look up LONG values in an IdSet built from INT values`.
- `IN_ID_SET` is now typed as nullable in the planner, because it returns
NULL for a NULL value. With the old NOT NULL type, Calcite removed `IS NOT
TRUE` and `COALESCE` around it, which gave wrong results over outer joins.
- An empty or NULL set gives FALSE, even for a NULL value, as `IN` over an
empty subquery does. `IDSET` returns NULL over zero rows when null handling is
on. This changes existing results in both engines: with null handling on,
`IN_ID_SET(nullableCol, <empty set>) = 0` (or `NOT IN_ID_SET(...)`) now keeps
the NULL rows, as `x NOT IN (empty subquery)` does. This includes the
single-stage `IN_SUBQUERY` when the subquery matches no rows.
- The scalar function is not evaluated at compile time (`isDeterministic =
false`). The planner passes literals in types the lookup cannot use, such as
BigDecimal for an INT literal, so an all-literal call is left to the runtime,
as before.
- The single-stage engine now also accepts `IN_ID_SET` after aggregation,
for example in `HAVING`. Aggregations such as `MAX` return DOUBLE there, so
they must be cast to the type the set was built from.
No change to the serialized format or the wire protocol.
## Known limits
- A Bloom filter set records only its hash funnel. INT and FLOAT values
share one funnel, and LONG and DOUBLE values share another, so a mismatch
inside those pairs is not detected.
- A call site whose set changes from row to row (for example a correlated
`IDSET` subquery, which has one set per group) deserializes the set on every
change.
## Tests
- `IdSetTest`: every set type from heap, read-only, direct and offset
buffers, and the value type matrix.
- `IdSetFunctionsTest`: every value type (including BOOLEAN, TIMESTAMP and
UUID, through their stored types), NULL and empty sets, malformed sets, cache
reuse and registration.
- `IdSetQueriesTest`: `IN_ID_SET` on every single-value type in a leaf, in
`HAVING`, and the mismatch error.
- `IdSet.json` (MSE, both planners):
- `IDSET` on every type, as a scalar subquery, a CTE and a correlated
subquery.
- `IN_ID_SET` in leaf filters, post-join filters, `HAVING` and join
conditions.
- NULL values from outer joins and nullable columns, and empty subqueries
with null handling on and off.
- A 9 MB set that crosses the mailbox in more than one chunk.
- `IDSET` results compared byte for byte with sets built directly from the
values.
- `IdSetIntegrationTest` (2 servers): both engines return the same `IDSET`
bytes, and a set built by either engine filters the same rows in both.
Without the main changes, 56 of the first 66 `IdSet.json` cases fail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
