carloea2 commented on code in PR #8597:
URL: https://github.com/apache/texera/pull/8597#discussion_r4253762583
##########
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/StandaloneHelpers.scala:
##########
@@ -153,20 +235,5 @@ object StandaloneHelpers {
| return s.map(lambda x: None if pd.isna(x) else ("true" if x
else "false"))
| if pd.api.types.is_integer_dtype(s):
| return s.map(lambda x: None if pd.isna(x) else str(int(x)))
- | if pd.api.types.is_datetime64_any_dtype(s):
- | # java.sql.Timestamp.toString: trailing zeros dropped from the
- | # fraction, but never all of them. Python's own str() writes no
- | # fraction at all on a whole second and six digits otherwise.
- | def _ts(x):
- | if pd.isna(x):
- | return None
- | text = x.strftime("%Y-%m-%d %H:%M:%S.%f").rstrip("0")
- | return text + "0" if text.endswith(".") else text
- |
- | return s.map(_ts)
- | return s.map(
- | lambda x: None
- | if pd.isna(x)
- | else ("true" if x else "false") if pd.api.types.is_bool(x) else
str(x)
- | )""".stripMargin
+ | return s.map(lambda x: None if pd.isna(x) else
_texera_java_text(x))""".stripMargin
Review Comment:
Stored NaN is still converted to null here. With a nullable Arrow Float64
column containing NaN, null and 1.0, the generated cast returns null, null and
1.0 as text; the native AttributeTypeUtils cast returns NaN, null and 1.0. I
reproduced this with the generated TypeCastingOpDesc code and Arrow validity
mask intact. Infinities and signed zero agree. Please preserve the column's
null mask so a valid NaN reaches _texera_java_text, and add this parity case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
