mattcasters opened a new pull request, #9211: URL: https://github.com/apache/hop/pull/9211
Parquet file output writes one file with `FileIO.write().via(ParquetIO.sink)`. The filename, including date and time, is resolved when the pipeline is submitted. Copy number and split number are not extra shards. Partitioned output and a field that sets its own Parquet type are refused. Dictionary page size and the writer version stay on the local engine. Parquet file input reads the file named by each incoming row and appends the configured columns. A field list or a metadata filename is required. A null filename is skipped. An empty incoming collection emits one null row when that option is set. `ParquetOutput.avroSchema` is now public so the handler and the local writer share the nullable Avro record. `beam-sdks-java-io-parquet` is a runtime dependency, resolved to the same Parquet 1.18.1 the Parquet plugin already uses. `BeamParquetFileIOTest` (6) and `ParquetIOViabilityTest` (5) passed on the Direct runner. Remote runners were not exercised. fixes #2357 ------------------------ Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Run `mvn clean install apache-rat:check` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically. - [x] If you have a group of commits related to the same change, please squash your commits into one and force push your branch using `git rebase -i`. - [x] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. To make clear that you license your contribution under the [Apache License Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0) you have to acknowledge this by using the following check-box. - [x] I hereby declare this contribution to be licensed under the [Apache License Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0) - [ ] In any other case, please file an [Apache Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
