maytasm opened a new pull request, #13612: URL: https://github.com/apache/druid/pull/13612
Fix Parquet Parser missing column when reading parquet file ### Description If you are ingesting Parquet using hadoop task, then you would set Parquet Hadoop Parser (https://druid.apache.org/docs/latest/ingestion/data-formats.html#parquet-hadoop-parser) Looking at how things work, there is a DruidParquetReadSupport class that select the columns from the parquet schema that are used in the schema of the ingestion job. This is use to read in Parquet data for the map/reduce job. DruidParquetReadSupport#getPartialReadSchema only consider the fields in the parseSpec->dimensionsSpec->dimensions, the ts field from parseSpec->timestampSpec->column, and the metrics’ requiredFields from dataSchema->metrics->fieldName The issue: you have a transformSpec or a flattenSpec which creates a field that has a different name which does not exist in your parquet schema and you do not include the base field (that exist in parquet schema) in the dimensionsSpec. For example, your parquet file has column country, then you add a transformSpec that does something like "transforms": [ { "type": "expression", "name": "countryUpper", "expression": "upper(country)" } . In the parseSpec->dimensionsSpec->dimensions, you only have countryUpper (as you don’t want the original country field in your Druid datasource). The problem is that the DruidParquetReadSupport will not read in the country field (as country is not in the dimensionsSpec). As a result, you will not get countryUpper in your Druid datasource as Druid (map/reduce) job did not read in the country field. This PR fix the above issue by determining the columns that the reader needs to read from parquet file by also looking at the flattenSpec and transformSpec. This PR has: - [ ] been self-reviewed. - [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.) - [ ] added documentation for new or modified features or behaviors. - [ ] a release note entry in the PR description. - [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links. - [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md) - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader. - [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met. - [ ] added integration tests. - [ ] been tested in a test Druid cluster. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
