Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/7262#discussion_r34081685
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlSerializer2.scala
---
@@ -179,23 +179,38 @@ private[sql] object SparkSqlSerializer2 {
/**
* Check if rows with the given schema can be serialized with
ShuffleSerializer.
+ * Right now, we do not support a schema having complex types or UDTs,
or all data types
+ * of fields are NullTypes.
*/
def support(schema: Array[DataType]): Boolean = {
if (schema == null) return true
+ var allNullTypes = true
var i = 0
while (i < schema.length) {
schema(i) match {
- case udt: UserDefinedType[_] => return false
- case array: ArrayType => return false
- case map: MapType => return false
- case struct: StructType => return false
+ case NullType => // Do nothing
+ case udt: UserDefinedType[_] =>
+ if (allNullTypes) allNullTypes = false
+ return false
+ case array: ArrayType =>
+ if (allNullTypes) allNullTypes = false
+ return false
+ case map: MapType =>
+ if (allNullTypes) allNullTypes = false
+ return false
+ case struct: StructType =>
+ if (allNullTypes) allNullTypes = false
+ return false
case _ =>
+ if (allNullTypes) allNullTypes = false
--- End diff --
pretty minor, but this is the same as just setting `allNullTypes = false`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
