saLeox opened a new pull request, #21149: URL: https://github.com/apache/flink/pull/21149
## What is the purpose of the change From the improvement [FLINK-23715](https://issues.apache.org/jira/browse/FLINK-23715), Flink use a collection named `unknownFieldsIndices` to track the nonexistent fields, and it is kept inside the `ParquetVectorizedInputFormat`, and applied to all parquet files under given path. However, some fields may only be nonexistent in some of the historical parquet files, while exist in latest ones. And based on `unknownFieldsIndices`, flink will always skip these fields, even thought they are existing in the later parquets. As a result, the value of all fields will become empty when they are nonexistent in some historical parquet files. This PR is going to make `unknownFieldsIndices` separate for each ParquetReader. ## Brief change log - *Make `unknownFieldsIndices` as a member variable of ParquetReader, which is an inner class of ParquetVectorizedInputFormat* ## Verifying this change This change added tests and can be verified as follows: - *Added junit test that validates `unknownFieldsIndices` can differ from each parquets* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): ( no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
