Ma77Ball opened a new issue, #8684: URL: https://github.com/apache/texera/issues/8684
### Task Summary > **In one sentence:** now that the columnar (Arrow) path makes downstream operators fast, the CSV scan's text parsing is the dominant cost, and a measured ~20x gap between parsing CSV and reading an equivalent Arrow file shows CSV ingestion is the next lever. **Context.** Part of the columnar execution work (parent #8556, discussion #8571). The columnar scan already parses CSV straight into Arrow batches (no per-row `Tuple`), and everything downstream (filter, aggregate, join) runs 4-6x faster. But scanning is now the bottleneck: turning CSV text into typed columns is inherently expensive, and unlike an already-columnar file the parse cannot be skipped. **Measured (why this is the next lever).** The same 2M rows written once as a CSV file and once as an Arrow columnar file, scanned and summing one column (JDK 17, x86, monotonic clock): | Scan 2M rows, sum one column | Time | vs CSV | | --- | --- | --- | | CSV (read + parse text) | 254 ms | baseline | | Arrow file (read, no parse) | 13 ms | ~20x | | Arrow file (mmap in place) | 12.6 ms | ~20x | The ~20x is the parse, not I/O: memory-mapping the Arrow file added only ~3%. So the win is in the parse itself, not zero-copy tricks. **Proposed directions (any subset).** 1. Parallelize CSV parsing by byte-range split into Arrow batches (a CSV analogue of the existing JSONL byte-range work), so multiple workers parse disjoint file regions. 2. Faster per-field parsing straight into Arrow column buffers, avoiding intermediate boxing. 3. Offer or automate a one-time CSV to Arrow/Parquet conversion so repeated queries hit the ~20x fast path via the columnar Arrow-file source operator. **Scope.** A performance improvement to the source layer that complements the columnar series and the scan/ingest work; not a drive-by. Related: #8556, #8571. ### Task Type - [x] Performance -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
