weiqingy commented on issue #1264:
URL: https://github.com/apache/auron/issues/1264#issuecomment-3940226421
Hi @Tartarus0zm @x-tong ! I've been looking into the Flink integration
effort and wanted to share a summary of where things stand, and ask about what
tasks are available to pick up.
Current state of auron-flink-extension:
```
┌─────────────────────────────────────────────────────────────────────┬──────────────┐
│ Component │
Status │
├─────────────────────────────────────────────────────────────────────┼──────────────┤
│ FlinkArrowUtils (Flink LogicalType → Arrow type conversion) │
Done (#1850) │
├─────────────────────────────────────────────────────────────────────┼──────────────┤
│ FlinkArrowUtilsTest │
Done │
├─────────────────────────────────────────────────────────────────────┼──────────────┤
│ Module structure (runtime / planner / assembly) │
Done (#1852) │
├─────────────────────────────────────────────────────────────────────┼──────────────┤
│ Test infrastructure (AuronFlinkTableTestBase, AuronFlinkCalcITCase) │
Done (#1387) │
└─────────────────────────────────────────────────────────────────────┴──────────────┘
```
What remains for Phase 1 (Calc operator), per the AIP:
1. FlinkArrowFieldWriter / FlinkArrowWriter — Write Flink RowData into
Arrow VectorSchemaRoot (Part 2/3 from
#1850)
2. FlinkArrowFieldReader / FlinkArrowReader — Read Arrow vectors back into
Flink RowData
3. FlinkArrowFFIExporter — Export Arrow batches to the native engine via C
Data Interface + JNI (Part 3/3 from
#1850)
4. Planner rule — Intercept StreamExecCalc and replace with a native
execution node
5. Native-side expression support — Convert Flink expressions to
DataFusion equivalents in Rust
I'd like to start contributing. Item 1 (FlinkArrowWriter) seems like the
most natural next step since it's self-contained, testable, and blocks the
downstream components. Is that available, or are there subtask issues already
being tracked that I should look at?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
