ashrithb commented on code in PR #53377:
URL: https://github.com/apache/spark/pull/53377#discussion_r2596757779
##########
python/pyspark/pandas/frame.py:
##########
@@ -2699,8 +2699,17 @@ def to_feather(
# Make sure locals() call is at the top of the function so we don't
capture local variables.
args = locals()
+ pdf = self._to_internal_pandas()
+ # SPARK-54068: Clear non-serializable attrs before feather write.
+ # PyArrow >= 22.0.0 serializes DataFrame.attrs to JSON metadata, but
+ # PlanMetrics/PlanObservedMetrics objects from Spark Connect are not
+ # JSON serializable. We clear these attrs since they are internal
+ # execution metadata not needed in the output file.
+ pdf.attrs = {k: v for k, v in pdf.attrs.items()
+ if k not in ("metrics", "observed_metrics")}
Review Comment:
> Maybe, can we do this only for `if LooseVersion(pa.__version__) <
LooseVersion("22.0.0"):` safely in order to avoid any regressions?
Hmm, yeah it may be better to err on the side of caution here even though
the change is for internal metadata, I'll add this logic in then!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
