itholic commented on code in PR #45699:
URL: https://github.com/apache/spark/pull/45699#discussion_r1538534299
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -4392,6 +4392,15 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val INFER_PANDAS_DICT_AS_MAP =
buildConf("spark.sql.execution.pandas.inferPandasDictAsMap")
Review Comment:
I'm not sure if we can reuse
`spark.sql.pyspark.inferNestedDictAsStruct.enabled` for this case?
Because IIRC `spark.sql.pyspark.inferNestedDictAsStruct.enabled` was defined
for inferring "nested" `dict` as `StructType` whereas current case is inferring
`dict` (non-nested) as `MapType`, so the purpose of two options sounds a bit
different for me.
For example, if someone wants to convert a nested `dict` to a `StructType`
(should set an option `True`) but at the same time doesn't want to convert the
`dict` to a `MapType` (should set an option `False` or `None`), shouldn't there
be two separate options?
Please correct me if I missed something!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
