yashmayya opened a new pull request, #19874:
URL: https://github.com/apache/pinot/pull/19874
## Problem
A lookup join fails to plan when its left input is an intermediate stage,
for example an aggregation:
```sql
SELECT /*+ joinOptions(join_strategy='lookup') */ d.name, t.cnt
FROM (SELECT dimKey, COUNT(*) AS cnt FROM fact GROUP BY dimKey) t
JOIN dim d ON t.dimKey = d.id
```
The error is `Cannot invoke "java.util.Map.size()" because
"workerIdToServerInstanceMap" is null`.
`WorkerManager` assigns workers in two passes: leaf stages first, then
intermediate stages. The lookup join stage scans the dimension table, so it is
a leaf stage. It copies the workers of its input. When the input is an
intermediate stage, the input has no workers yet in the leaf pass.
## Fix
- A lookup join stage over an intermediate input is now assigned in the
intermediate pass, after its input. This also applies to a chain of lookup
joins.
- Its workers come from `getCandidateServersForReplicatedLeaf()`. By
default, this hook returns the workers of the input. So the exchange stays
local, the same as for a lookup join over a leaf input.
- The join probes the dimension table in server memory, so it must run on a
server. A subclass that runs intermediate stages on instances that cannot scan
segments can return servers from this hook.
- A lookup join over a leaf input does not change.
## Tests
- `WorkerManagerTest`:
- Lookup joins over an aggregation, with the default hook and with an
override that runs intermediate stages on a non-server instance.
- A lookup join over a leaf input, with that override.
- The two intermediate-input tests fail without the fix.
- `LookupJoin.json`: one lookup join and a chain of two lookup joins over an
aggregation, end to end.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
