[
https://issues.apache.org/jira/browse/BEAM-11742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenbing Bai updated BEAM-11742:
-------------------------------
Fix Version/s: 2.30.0
Assignee: Wenbing Bai
Resolution: Fixed
Status: Resolved (was: Open)
https://github.com/apache/beam/commit/c97e17eba04e56f9f7bdcec991e179c003ae128b#diff-33b0b6b112036df96f341aa83b88efba9215ec14dfabc9db9e9ffe66a23154a2
> ParquetSink fails for nullable fields
> -------------------------------------
>
> Key: BEAM-11742
> URL: https://issues.apache.org/jira/browse/BEAM-11742
> Project: Beam
> Issue Type: Improvement
> Components: io-py-parquet
> Reporter: Wenbing Bai
> Assignee: Wenbing Bai
> Priority: P3
> Fix For: 2.30.0
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> Before pyarrow 0.15, it is not possible to create pyarrow record batch with
> schema.
> So in apache_beam.io.parquetio._ParquetSink, when creating pyarrow record
> batch we use
>
> {code:java}
> rb = pa.RecordBatch.from_arrays(arrays, self._schema.names){code}
> Error is raised that the parquet table to be created (record batch schema)
> has a different schema with the schema specify (self._schema).
> For example, when schema specified with "is not null", the record batch
> schema doesn't indicate that, the error will be raised.
>
> The fix is to use schema instead of names in pa.RecordBatch.from_arrays
> {code:java}
> rb = pa.RecordBatch.from_arrays(arrays, schema=self._schema){code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
