kz930 opened a new pull request, #8704:
URL: https://github.com/apache/texera/pull/8704
### What changes were proposed in this PR?
The JSON Lines and Arrow sources implement `StandaloneCodeGenerator`, so an
exported script starts from the same data the workflow did.
Arrow is read into pandas' nullable dtypes rather than its numpy ones. Arrow
says of every value whether it is there, and a numpy column has nowhere to put
that: a missing double and a stored NaN both land on NaN, and a missing integer
costs the column its integer type. The engine reads the file's own answer, so a
set operation keeps a NaN row a null row does not match, a holed integer column
is still made of integers, and a long past 2^53 is not rounded on its way
through a float. The other sources state the types they need one at a time,
their formats having no way to say it once.
A file also states the width and the sign of each of its numbers, and Texera
has no column narrower than a double or a 32-bit integer. A single-precision or
16-bit column was arriving null, the parse having no case for the Float and
Short those vectors hand back, and an unsigned one arrived as the -1 its
storage counts down to. Each now reads as the Texera type that holds it, an
unsigned 32-bit column as a long and an unsigned 64-bit one refused for want of
a wider one, and the script normalizes the same widths so a later sum comes to
the same number on both sides. This is the rule the Parquet source already
follows. The end of an exported Arrow window is counted in Long, as the CSV
readers count theirs.
A timestamp is read as the engine reads one. The engine hands each field to
DateParserUtils, so a row states its own format, any year reads, an offset is
moved to the local zone, and the reading is cut to the millisecond. pandas'
date parsing does none of that. So JSONL reads the column as text and converts
it with the helper #8597 adds.
This PR also changes the engine. `ArrowUtils` handed a TIMESTAMP column to a
Python worker as an Arrow millisecond timestamp labelled UTC. A Python operator
saw an aware datetime cut to the millisecond, and a year past 2262 overflowed.
It is now a zoneless microsecond timestamp. That is what Texera's own Python
side declares for a TIMESTAMP (`pa.timestamp("us")`), and what an exported
script holds. Two things a user can see change. A Python UDF now receives a
naive datetime instead of one in UTC. A result downloaded as an Arrow file now
carries a zoneless microsecond timestamp column instead of a UTC millisecond
one. A TIMESTAMP parameter a Python UDF declares is now parsed the same way, as
a naive wall clock with any offset moved to the machine's zone, so it compares
with a row's instead of raising.
### Any related issues, documentation, discussions?
Part of #8325, step 12b; that issue lists the set in order. It needs #8341
for the shared source code and #8597 for the timestamp helpers, so both merge
first.
Closes #8702, the task this change is the whole of.
It also finishes #8602, where a scan window of no rows left the type
inference nothing to read and JSONL declared a schema of no columns. #8341
fixes the CSV half.
Closes #8602.
### How was this PR tested?
Each operator asserts the block it emits in its own spec, and the Arrow spec
runs the engine's own executor and the generated script over the same file. The
engine change is covered in both ArrowUtilsSpec files, which include a round
trip on either side of the nanosecond range, and in the Python tests of the
timestamp parser. Once the verification lands, every operator is also run both
ways on every configuration its schema offers; this branch is cut from main and
does not carry that machinery, so those runs are not on this diff's CI.
Arrow is the one worth calling out. Its column of timestamps only matches
once #7672 is in: before that the engine read the file's numbers through the
JVM's zone while `pd.read_feather` read the wall clock the file states, and the
two differed by whatever the machine was set to. That change has since landed.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
