Amer-Mukhtar opened a new pull request, #8907:
URL: https://github.com/apache/texera/pull/8907
Description:
### What changes were proposed in this PR?
The CSV scan spends most of its time converting text fields to typed values,
mostly timestamps. User-facing behavior is unchanged: same values, same errors,
same warnings.
- `AttributeTypeUtils.parseFields` fills an array in a plain loop instead of
building a collection per row.
- Timestamp parsing uses one `DateParser` per thread instead of
`DateParserUtils.parseDate`, which is `static synchronized` and serializes all
workers in the JVM.
- New fast path for zone-less ISO timestamps (`yyyy-MM-dd[(
|T)HH:mm[:ss[.f]]]`). It sets the same `Calendar` fields the library does, so
results are identical; any other string still goes to the library.
- `CSVScanSourceOpExec` and `ParallelCSVScanSourceOpExec` build the column
type array once per scan instead of once per row.
| Full CSV scan, 2M rows | Before | After |
|---|---|---|
| int, double, string, bool, long | 876 ms | 553 ms |
| + ISO timestamp column | 3593 ms | 1026 ms |
### Any related issues, documentation, discussions?
Related to #8684 (direction 2, faster per-field parsing). The Arrow parts of
that issue depend on #8558.
### How was this PR tested?
Added tests to `AttributeTypeUtilsSpec` (short and null rows, parse-error
message and cause, timestamps compared against the date-parser library on
~4,800 valid and invalid strings, parsing from 8 threads at once) and to
`CSVScanSourceOpExecSpec` (each field comes out as its inferred type). A
one-off check also compared the fast path with the library on 137,760 strings
in 7 time zones, with 0 mismatches.
sbt "WorkflowCore/testOnly
org.apache.texera.amber.core.tuple.AttributeTypeUtilsSpec"
"WorkflowOperator/testOnly org.apache.texera.amber.operator.source.scan.csv.*"
sbt WorkflowCore/test WorkflowOperator/test
sbt "WorkflowCore/scalafixAll --check" "WorkflowOperator/scalafixAll
--check" WorkflowCore/scalafmtCheckAll WorkflowOperator/scalafmtCheckAll
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
