Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/20678#discussion_r170763146
--- Diff: python/pyspark/sql/tests.py ---
@@ -3493,19 +3495,42 @@ def create_pandas_data_frame(self):
data_dict["4_float_t"] = np.float32(data_dict["4_float_t"])
return pd.DataFrame(data=data_dict)
- def test_unsupported_datatype(self):
- schema = StructType([StructField("map", MapType(StringType(),
IntegerType()), True)])
- df = self.spark.createDataFrame([(None,)], schema=schema)
- with QuietTest(self.sc):
- with self.assertRaisesRegexp(Exception, 'Unsupported type'):
- df.toPandas()
+ @contextmanager
+ def arrow_fallback(self, enabled):
--- End diff --
I think it would be best to disable fallback for all the tests on
setup/teardown. That way if something goes wrong elsewhere, the tests won't
start passing due to falling back. For the test where it is enabled, you could
do that explicitly. What do you think?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
