xianzhe-databricks commented on code in PR #52467:
URL: https://github.com/apache/spark/pull/52467#discussion_r2398636023
##########
python/pyspark/sql/connect/dataframe.py:
##########
@@ -1823,7 +1823,14 @@ def collect(self) -> List[Row]:
assert schema is not None and isinstance(schema, StructType)
- return ArrowTableToRowsConversion.convert(table, schema)
+ return ArrowTableToRowsConversion.convert(
+ table, schema, binary_as_bytes=self._get_binary_as_bytes()
+ )
+
+ def _get_binary_as_bytes(self) -> bool:
+ """Get the binary_as_bytes configuration value from Spark session."""
+ conf_value =
self._session.conf.get("spark.sql.execution.pyspark.binaryAsBytes", "true")
+ return (conf_value or "true").lower() == "true"
Review Comment:
The method `self._session.conf.get` returns only an `Optional[str]`. I think
the conf is retrieved from Spark Connect Server. ([code
pointer](https://github.com/apache/spark/blob/master/python/pyspark/sql/connect/conf.py#L68))
This way, mypy does not complain about type incompatibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
