kz930 commented on code in PR #8597:
URL: https://github.com/apache/texera/pull/8597#discussion_r4060369563
##########
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/StandaloneHelpers.scala:
##########
@@ -122,6 +122,49 @@ object StandaloneHelpers {
| return max(-2147483648, min(2147483647, int(value)))
|
|
+ |def _texera_text_to_timestamp(s):
+ | # Read cell by cell, the way the engine reads a column:
DateParserUtils
+ | # is handed one field at a time, so a row states its own format and
the
+ | # rest of the column has no say in it.
+ | #
+ | # Held at microsecond resolution and not the nanoseconds pandas
parses
+ | # into by default, which reach 1677 to 2262: the engine holds a
+ | # java.sql.Timestamp, where the year 2500 is an ordinary moment and
+ | # emptying it would answer for a row the run itself had no trouble
with.
+ | #
+ | # Still coerced, which the strict cast is not: the engine accepts a
set
+ | # of formats no single pandas call states, so text neither can read
is
+ | # answered with an empty cell rather than by ending the run.
+ | #
+ | # A reading that states an offset is moved to the zone the machine
is
+ | # set to and then holds that wall clock, which is what the engine
does
+ | # with one: DateParserUtils reads the offset and java.sql.Timestamp
+ | # keeps no zone of its own. Left alone, the offset travels as far as
+ | # the conversion below and stops the cast on a value the run reads.
+ | # tzlocal() and not a fixed offset, so each instant gets the one in
+ | # force when it happened.
+ | from dateutil.parser import parse as _parse_date
+ | from dateutil.tz import tzlocal
+ |
+ | if pd.api.types.is_datetime64_any_dtype(s):
+ | if getattr(s.dtype, "tz", None) is not None:
+ | s = s.dt.tz_convert(tzlocal()).dt.tz_localize(None)
+ | return s.astype("datetime64[us]")
Review Comment:
Confirmed and fixed in ca67ce12b. The `astype` was meant for the text path
and was also catching an already-parsed column; that branch now returns the
column at the resolution it arrived in, and a test compares your two values
against `parseField` cell by cell.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
