Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19943#discussion_r160016446
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala
---
@@ -110,4 +107,23 @@ object OrcUtils extends Logging {
}
}
}
+
+ /**
+ * Return a fixed ORC schema with data schema information, if needed.
+ * The schema inside old ORC files might consist of invalid column names
like '_col0'.
+ */
+ def getFixedTypeDescription(
--- End diff --
Do we really need this? The ORC schema is used to create the ORC batch, for
the batch data I think we only care about the data types not field names.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
