weiqingy opened a new issue, #2416: URL: https://github.com/apache/auron/issues/2416
Follow-up to #2329. When a native source feeds more than one Calc (for example `UNION ALL` over one table under Flink source reuse), the source is not fused and each Calc runs as a standalone native operator over the shared source stream. This is correct, but it repeats work per consumer. The source emits `AuronColumnarRowData`, a zero-copy view over the native Arrow batch. Each standalone Calc then re-materializes that row back into a freshly allocated Arrow `VectorSchemaRoot`, column by column, before running its native plan (`FlinkAuronCalcOperator` -> `FlinkArrowFFIExporter` -> `FlinkArrowWriter`). So for N consumers the Kafka read and decode happen once, but the row-to-Arrow rebuild is paid N times, and it is a rows x cols scalar copy each time. `AuronColumnarRowData` exposes only per-field getters, so there is no zero-copy re-export of the underlying batch today, and the exporter has no already-Arrow-backed fast path. Idea: hand Arrow batches across the operator edge instead of a per-row RowData stream, so a shared native source can feed multiple native Calc operators without rebuilding the columnar buffers per consumer. This is a batch-oriented operator exchange and is larger than the multi-consumer-safety scope of #2329, hence a separate issue. Magnitude is unmeasured. A profiler run on a `source -> single Calc` job (parallelism 1) would quantify the CPU share spent in `FlinkArrowWriter.write` / `ArrowFieldWriter` versus Kafka read and native compute, to confirm the win is worth the change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
