sivabalan narayanan created HUDI-3596:
-----------------------------------------
Summary: Understand why NULLSchemaProvider is used for empty batch
in InputBatch instead of empty schema
Key: HUDI-3596
URL: https://issues.apache.org/jira/browse/HUDI-3596
Project: Apache Hudi
Issue Type: Task
Components: deltastreamer
Reporter: sivabalan narayanan
In
[InputBatch|https://github.com/apache/hudi/blob/548000b0d635067acf7574c4cf5122759e79b52b/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/InputBatch.java#L57]
we use NullSchemaProvider when batch is empty. this will return null schema (
Schema.create(Schema.Type.NULL)). We need to understand why can't we return an
empty schema here (Scehma.create(Collections.emptyList()). Thats the schema for
an empty spark dataframe and we should stick to that.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
