AliRana30 opened a new issue, #875:
URL: https://github.com/apache/wayang/issues/875
### Description
Wayang currently supports embedded execution via `wayang-java` (Java
Streams) and `wayang-sqlite3` (row-oriented embedded SQL), as well as
server-based platforms (Spark, Flink, Postgres, BigQuery, and the proposed
ClickHouse platform in #871).
However, Wayang currently lacks an **in-process, vectorized columnar OLAP
platform**.
I propose adding a new platform module: `wayang-platforms/wayang-duckdb`.
DuckDB is widely regarded as the "SQLite of OLAP". Adding DuckDB enables
Wayang to execute analytical queries (aggregations, joins, filter pushdowns)
with vectorized columnar speed directly in-process, without requiring users to
launch or maintain a remote cluster (like Spark) or external database daemon
(like Postgres or ClickHouse).
### Value & Motivation
1. **High-Speed In-Process OLAP**: Gives Wayang a fast, local columnar
execution tier for medium-sized analytical workloads that are too large for
simple Java Streams but too small to justify cluster overhead.
2. **Zero Infrastructure & Reliable CI**: Uses the official
`org.duckdb:duckdb_jdbc` driver (MIT license). Because DuckDB is in-process,
full end-to-end integration tests (`DuckDBOperatorsIT`) run directly in CI
without requiring external servers or Docker containers.
3. **Native Parquet/CSV Compatibility**: Lays the foundation for pushing
file-based scans directly into a vectorized SQL engine.
### Proposed Architecture & Layout
Following the architecture of `wayang-sqlite3` and `wayang-jdbc-template`:
| Component | Class / File | Purpose |
|---|---|---|
| Entry point | `DuckDB.java` | Static helper for `plugin()`, `platform()` |
| Platform | `DuckDBPlatform extends JdbcPlatformTemplate` | Platform
definition, configuration ID `duckdb` |
| Operators |
`DuckDB{TableSource,Filter,Projection,Join,GlobalReduce,ReduceBy,Sort,TableSink}Operator`
| Extend corresponding `Jdbc*` operators |
| Mappings | `Mappings.java` (operator mappings + `Mappings.ALL`) |
Translates Wayang operators to DuckDB physical operators |
| Channels | `ChannelConversions` | Bridges DuckDB SQL -> Java Streams /
Spark RDDs |
| Config | `wayang-duckdb-defaults.properties` | JDBC URL (`jdbc:duckdb:`
in-memory or file-backed), driver class, cost parameters |
| Build | `wayang-platforms/pom.xml` | Register new
`<module>wayang-duckdb</module>` |
### Dialect & Engine Specifics
- **Connection handling**: Default to in-process in-memory database
(`jdbc:duckdb:`) or user-configured database file paths
(`wayang.duckdb.jdbc.url`).
- **Table Sink**: DuckDB fully supports standard `CREATE TABLE <t> AS SELECT
...` and `INSERT INTO <t> SELECT ...`, cleanly matching `JdbcTableSinkOperator`.
- **Quoting & Identifiers**: Case preservation and identifier escaping
conform to standard ANSI SQL quotes (`"identifier"`).
### Proposed Phased Implementation
To keep code reviews concise and self-contained:
1. **PR 1: Module Skeleton & Basic Operators**
- Create `wayang-platforms/wayang-duckdb` module.
- Add `DuckDBPlatform`, `DuckDB.java`, plugin registration, and
configuration properties.
- Implement `DuckDBTableSource`, `DuckDBFilterOperator`, and
`DuckDBProjectionOperator`.
- Unit tests covering SQL clause generation.
2. **PR 2: Complex Operators & Integration Tests**
- Implement `Join`, `GlobalReduce`, `ReduceBy`, `Sort`, and `TableSink`
operators.
- Add `DuckDBOperatorsIT` running real queries against an in-process
DuckDB instance (100% CI compatible).
3. **PR 3: Documentation & Application Demo**
- Add module `README.md`, an example demo in `wayang-applications`, and
update the platform list in `wayang-docs`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]