kz930 opened a new pull request, #7672: URL: https://github.com/apache/texera/pull/7672
### What changes were proposed in this PR? Every other source hands the engine a wall clock. CSV and JSONL carry no zone at all, so what the file states is what the operator gets, and `TypeCasting` and the time-series plot both parse with a plain `pd.to_datetime`. A Texera TIMESTAMP has no zone to record one either. Arrow was the exception: its fields carry a zone, and `ArrowUtils` reconciled that through the value's own epoch, which reads the wall clock in the JVM's zone and puts the machine's setting where the file cannot record it. So one file read out different values on servers in different places, with nothing to account for the difference: | Server timezone | Value produced, before | After | | --- | --- | --- | | America/Los_Angeles | 2024-01-07 00:00:00 | 2024-01-07 00:00:00 | | UTC | 2024-01-07 08:00:00 | 2024-01-07 00:00:00 | | Asia/Tokyo | 2024-01-07 17:00:00 | 2024-01-07 00:00:00 | | Europe/Berlin | 2024-01-07 09:00:00 | 2024-01-07 00:00:00 | Writing was the same mismatch from the other end: the number stored was the local instant of the wall clock while the label beside it said UTC, so every reader other than a Texera in that same zone saw the value moved. Reading that file with `pandas.read_feather` reported 08:00 before and reports 00:00 now, which is what Texera itself shows. Both directions now go through the label the field already carries, which puts Arrow on the same footing as the rest: what the file states is what the engine gets, wherever it runs. Zoneless columns, which is what an ordinary `.arrow` file written by pandas holds, are untouched. Those hand back a `LocalDateTime`, already the wall clock itself. One thing for anyone holding existing files: bytes written before this are read by the new rule, so a timestamp in them shifts by the offset of the zone that wrote it. ### Any related issues, documentation, discussions? Closes #7666 ### How was this PR tested? `ArrowUtilsSpec` covers both halves and one assertion moved with the change: it stated the identity between the stored long and the value's own epoch, which is machine-dependent, and now states the wall clock explicitly. The round-trip case needed no change, preserving the wall clock being the property both before and after. Beyond the suite, the table above is one file written once and read by four JVMs started with different `-Duser.timezone`, and a zoneless file written by pandas was read by three more, reporting the wall clock pandas wrote in each. `workflow-core` (759 tests) and `workflow-operator` (2214) pass, as do the Python-channel specs that share this conversion, `PythonProxyServerSpec` and `PythonProxyClientSpec`. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
