HyukjinKwon commented on code in PR #59212:
URL: https://github.com/apache/spark/pull/59212#discussion_r4169374476
##########
python/pyspark/sql/variant_utils.py:
##########
@@ -413,10 +415,73 @@ def handle_array(value_pos_list: List[int]) -> str:
if isinstance(value, bytes):
# decoding simply converts byte array to string
return '"' + base64.b64encode(value).decode("utf-8") + '"'
- if isinstance(value, (datetime.date, datetime.datetime)):
+ if isinstance(value, float):
+ return cls._float_to_json(value, type_info ==
VariantUtils.FLOAT)
+ if isinstance(value, decimal.Decimal):
+ return cls._decimal_to_json(value)
+ if isinstance(value, datetime.datetime):
+ return '"' + cls._timestamp_to_json(value) + '"'
+ if isinstance(value, datetime.date):
return '"' + str(value) + '"'
return str(value)
+ # The `_*_to_json` helpers below format scalars the same way as
`Variant.toJsonImpl` on the
Review Comment:
**Nit (P3):** This comment says `VariantVal.toJson` and `to_json` "produce
the same output", but finite doubles and FLOATs still use Python's notation.
For example, `1e7` renders as `10000000.0` here vs `1.0E7` on the JVM, and
`1e20` as `1e+20` vs `1.0E20`. The PR description says this is intentional, but
that note won't live next to the code. Could the comment state value-level
parity and name this exception? Something like: "... format scalars like
`Variant.toJsonImpl` on the JVM. Finite doubles and floats keep Python's
notation (e.g. `1e+20` vs the JVM's `1.0E20`), which denotes the same value."
**Verification:**
- **Inspection:** The revised comment accurately reflects
review-frame:exponent-notation-accepted and the JVM Double/Float.toString
behavior, and the diff touches only comment lines.
- **Regression:** The existing variant toJson tests, including the
JVM-comparison test, still pass unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
