kz930 opened a new pull request, #8597: URL: https://github.com/apache/texera/pull/8597
### What changes were proposed in this PR? Type Casting to a timestamp read text with `pd.to_datetime(col, errors="coerce")` in the exported script. That is wrong in two ways, and both empty a cell the run itself filled. pandas parses into nanoseconds, which reach only 1677-09-21 to 2262-04-11. The engine holds a `java.sql.Timestamp`, where `2500-01-01 00:00:00` is an ordinary moment: it reads it, and the exported script answered with an empty cell. The same for `1500-06-15 08:30:00` and `9999-12-31 23:59:59`. pandas also infers one format for the whole column and coerces every row that does not match it. The engine hands `DateParserUtils` one field at a time, so a row states its own format. A column holding `2024-03-05 14:09:07` and `March 5, 2024` kept the first and emptied the second. The text branch now reads the column cell by cell and holds the result at microsecond resolution, which covers the years the engine covers. It still coerces: the engine accepts a set of formats no single pandas call states, so text neither side can read leaves an empty cell rather than ending an exported run halfway. That part is unchanged, and a test now says so. The number branch is left alone. A LONG is read as epoch milliseconds and has the same window, but it also disagrees about the calendar before 1582-10-15, where `new Timestamp(long)` is Julian and pandas is proleptic Gregorian. Widening the window there would answer with a moment eight days off instead of an empty cell, so what that branch should do is a separate decision. The issue records it. ### Any related issues, documentation, discussions? Closes #8595. #8480 closed the same nanosecond gap for Extract Date/Time Fields, which parses strictly and could take the one-line form this operator cannot. ### How was this PR tested? Two tests in `TypeCastingOpDescSpec`, both running the generated Python and comparing cell by cell against `AttributeTypeUtils.parseField`, which is how that spec already compares the other casts. The first reads eight moments in one column: three ordinary ones written in three different formats, the two rows either side of the nanosecond edge, and `2500-01-01`, `1500-06-15` and `9999-12-31`. It fails on the old generator, which empties five of the eight. The second says the cast still answers unreadable text with an empty cell, where the engine refuses it. The whole `WorkflowOperator` module passes, 2567 tests. `scalafmtCheckAll` and `scalafixAll --check` are clean. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
