itholic commented on code in PR #45699:
URL: https://github.com/apache/spark/pull/45699#discussion_r1538764195
##########
python/pyspark/sql/pandas/conversion.py:
##########
@@ -600,15 +608,39 @@ def _create_from_pandas_with_arrow(
)
import pyarrow as pa
+ infer_pandas_dict_as_map = (
+
str(self.conf.get("spark.sql.execution.pandas.inferPandasDictAsMap")).lower()
== "true"
Review Comment:
Ah, I added tests into connect test suite to directly compare non-connect
and connect behavior like:
```python
self.assertEqual(
sdf.withColumn("test", SF.col("dict_col")[SF.col("str_col")]).collect(),
cdf.withColumn("test", CF.col("dict_col")[CF.col("str_col")]).collect(),
)
```
Would it be better to move those tests into non-connect and reuse it?
##########
python/pyspark/sql/pandas/conversion.py:
##########
@@ -600,15 +608,39 @@ def _create_from_pandas_with_arrow(
)
import pyarrow as pa
+ infer_pandas_dict_as_map = (
+
str(self.conf.get("spark.sql.execution.pandas.inferPandasDictAsMap")).lower()
== "true"
Review Comment:
Ah, I added tests into connect test suite because I wanted to directly
compare non-connect and connect behavior like:
```python
self.assertEqual(
sdf.withColumn("test", SF.col("dict_col")[SF.col("str_col")]).collect(),
cdf.withColumn("test", CF.col("dict_col")[CF.col("str_col")]).collect(),
)
```
Would it be better to move those tests into non-connect and reuse it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
