[
https://issues.apache.org/jira/browse/PARQUET-1945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Shao updated PARQUET-1945:
--------------------------------
Description:
Right now, Parquet Writer throws out an exception:
{{Parquet record is malformed: empty fields are illegal, the field should be
ommited completely instead}}
when an empty field (array or struct or map I guess?) is written.
The suggestion here is to add an option "auto_convert_empty_fields_to_null"
that convert empty fields to null automatically on write.
The LOC to change is
[here:|https://sourcegraph.com/github.com/apache/parquet-mr/-/blob/parquet-column/src/main/java/org/apache/parquet/io/MessageColumnIO.java#L328]
{quote}{{if (emptyField) {}}
{{ {{ throw new ParquetEncodingException("empty fields are illegal, the field
should be ommited completely instead");}}}}
{{}}}{quote}
was:
Right now, Parquet Writer throws out an exception:
{{Parquet record is malformed: empty fields are illegal, the field should be
ommited completely instead}}
when an empty field (array or struct or map I guess?) is written.
The suggestion here is to add an option "auto_convert_empty_fields_to_null"
that convert empty fields to null automatically on write.
The LOC to change is
[here:|https://sourcegraph.com/github.com/apache/parquet-mr/-/blob/parquet-column/src/main/java/org/apache/parquet/io/MessageColumnIO.java#L328]
{{ if (emptyField) {}}
{{ throw new ParquetEncodingException("empty fields are illegal, the field
should be ommited completely instead");}}
{{ }}}
> Add an option to allow auto conversion from empty fields to NULL
> ----------------------------------------------------------------
>
> Key: PARQUET-1945
> URL: https://issues.apache.org/jira/browse/PARQUET-1945
> Project: Parquet
> Issue Type: Improvement
> Components: parquet-format
> Reporter: Zheng Shao
> Priority: Minor
>
> Right now, Parquet Writer throws out an exception:
> {{Parquet record is malformed: empty fields are illegal, the field should be
> ommited completely instead}}
> when an empty field (array or struct or map I guess?) is written.
> The suggestion here is to add an option "auto_convert_empty_fields_to_null"
> that convert empty fields to null automatically on write.
> The LOC to change is
> [here:|https://sourcegraph.com/github.com/apache/parquet-mr/-/blob/parquet-column/src/main/java/org/apache/parquet/io/MessageColumnIO.java#L328]
> {quote}{{if (emptyField) {}}
> {{ {{ throw new ParquetEncodingException("empty fields are illegal, the
> field should be ommited completely instead");}}}}
> {{}}}{quote}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
