ekas-7 commented on issue #871: URL: https://github.com/apache/wayang/issues/871#issuecomment-6857474931
Hi @zkaoudi, Thank you for the +1 on this. The implementation is now up in #877, and I would be grateful if you could take a look whenever you have time. A short summary of what the PR contains: - `wayang-platforms/wayang-clickhouse`, following the Trino module layout: TableSource, Filter, Projection, Join, GlobalReduce, ReduceBy, Sort and TableSink. - TableSink overwrite mode emits `CREATE TABLE ... ENGINE = MergeTree ORDER BY tuple() AS SELECT ...`, with the engine overridable per sink. - 41 unit tests that need no server, plus `ClickHouseOperatorsIT` (11 tests), which I ran against ClickHouse 24.12 and which skips when no server is reachable. - Module README, `ClickHouseDemo` and `clickhouse.md` in `wayang-applications`, and the docs index entry. No docker-compose or setup folders, per #796. One thing I should flag: in this issue I proposed splitting the work into several small PRs, but #877 combines PRs 1–4. The pieces turned out to be hard to test in isolation (the integration test needs the sink, and the sink needs the other operators), so I kept them together. I realise that makes it a larger review. If you would prefer it split into smaller PRs, I am very happy to do that; just let me know. The questions from the issue description are still open, and I will follow whatever you decide: 1. Is the Trino/Presto/BigQuery module structure the right reference? 2. Is a configurable sink engine acceptable, or should the sink only `INSERT INTO` pre-created tables? 3. Is an integration test that skips without a server fine, or would you prefer Testcontainers? Cost calibration (`ClickHouseCostPilotIT`) is not included and can follow separately, as noted under "out of scope". It also looks like the CI workflows on the PR are waiting for approval to run. Could you approve them when convenient? Thank you for your time, and for any feedback. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
