GitHub user vinothchandar edited a comment on the discussion: Native SQL DDL
support for Hudi table creation across engines (Trino, Presto etc.)
Quick Trino connector SPI research says,
```
source SELECT ──┐
├── join on merge condition ← engine-planned exchanges
(index lookup
target scan ────┘ (t.uuid = s.uuid) can be done by pushing
filter to target scan)
│
merge processor: tags each row INSERT / DELETE / UPDATE(→delete+insert),
attaches $row_id
│
═══ remote exchange: merge partitioning scheme ═══ ← Only one
connector-controlled shuffle allowed
│ (our challenge is doing optimizing file size/layout and writing
MDT RLI etc from here)
▼
ConnectorMergeSink per writer task → writes files → finish() → fragments →
sent to coordinator for commit
```
I think we cannot easily support RLI or SI writes in Trino. We should
probably design the V1 without those. Then get creative later on.
Hope that helps! this was the limitation I was referring to at the Hudi
meetup talk.
GitHub link:
https://github.com/apache/hudi/discussions/19484#discussioncomment-18001578
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
