sadhen commented on a change in pull request #32332:
URL: https://github.com/apache/spark/pull/32332#discussion_r619935979
##########
File path: python/pyspark/sql/session.py
##########
@@ -695,9 +704,10 @@ def prepare(obj):
prepare = lambda obj: obj
Review comment:
See
https://github.com/apache/spark/blob/6f782efb044403ab3ca79662fdeca7f1f906e1bf/python/pyspark/sql/pandas/conversion.py#L312
@HyukjinKwon That's why we need to do verification after the schema is
inferred.
##########
File path: python/pyspark/sql/session.py
##########
@@ -695,9 +704,10 @@ def prepare(obj):
prepare = lambda obj: obj
Review comment:
See
https://github.com/apache/spark/blob/6f782efb044403ab3ca79662fdeca7f1f906e1bf/python/pyspark/sql/pandas/conversion.py#L312:L314
@HyukjinKwon That's why we need to do verification after the schema is
inferred.
##########
File path: python/pyspark/sql/session.py
##########
@@ -695,9 +704,10 @@ def prepare(obj):
prepare = lambda obj: obj
Review comment:
See
https://github.com/apache/spark/blob/6f782efb044403ab3ca79662fdeca7f1f906e1bf/python/pyspark/sql/pandas/conversion.py#L312-L314
@HyukjinKwon That's why we need to do verification after the schema is
inferred.
##########
File path: python/pyspark/sql/session.py
##########
@@ -695,9 +704,10 @@ def prepare(obj):
prepare = lambda obj: obj
Review comment:
1. create from pandas, schema is a list.
2. create directly without schema, schema is None
##########
File path: python/pyspark/sql/session.py
##########
@@ -695,9 +704,10 @@ def prepare(obj):
prepare = lambda obj: obj
Review comment:
it is not schema inference, it just extracts the name.
I think doing verification after the schema is inferred is good idea.
Otherwise, we need to change when to infer schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
