[
https://issues.apache.org/jira/browse/SPARK-24358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488090#comment-16488090
]
Joel Croteau commented on SPARK-24358:
--------------------------------------
This may be trickier than I first thought. In Python 2, bytes is an alias for
str, so a bytes object resolves as a StringType. In Python 3, they are
different types, and not in general freely convertible, as an str in Python 3
is unicode, and an arbitrary byte string as represented by a bytes may not be
valid unicode. This means that a bytes in Python 2 will need to be resolved as
a different schema from a bytes in Python 3. Not sure how significant that is.
> createDataFrame in Python 3 should be able to infer bytes type as Binary type
> -----------------------------------------------------------------------------
>
> Key: SPARK-24358
> URL: https://issues.apache.org/jira/browse/SPARK-24358
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.3.0
> Reporter: Joel Croteau
> Priority: Minor
> Labels: Python3
>
> createDataFrame can infer Python 3's bytearray type as a Binary. Since bytes
> is just the immutable, hashable version of this same structure, it makes
> sense for the same thing to apply there.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
