aokolnychyi opened a new issue #125: ParquetMetrics corrupts lower/upper bounds for small decimals URL: https://github.com/apache/incubator-iceberg/issues/125 It seems `ParquetMetrics` corrupts lower/upper bounds for decimals that are represented as ints in Parquet. Consequently, Iceberg can skip wrong files. Create a Parquet file with one row: ``` Schema schema = new Schema( optional(1, "decimalCol", Types.DecimalType.of(10, 2)) ); GenericData.Record record = new GenericData.Record(AvroSchemaUtil.convert(schema.asStruct())); record.put(0, new BigDecimal("3.50")); File parquetFile = newParquetFile(); try (FileAppender<GenericData.Record> writer = Parquet.write(localOutput(parquetFile)) .schema(schema) .build()) { writer.add(record); } ``` Explore this file using parquet-tools (min/max values are correct): ``` file schema: table -------------------------------------------------------------------------------- decimalCol: OPTIONAL INT64 O:DECIMAL R:0 D:1 row group 1: RC:1 TS:75 OFFSET:4 -------------------------------------------------------------------------------- decimalCol: INT64 GZIP DO:0 FPO:4 SZ:91/75/0.82 VC:1 ENC:BIT_PACKED,PLAIN,RLE ST:[min: 3.50, max: 3.50, num_nulls: 0] ``` Create an Iceberg table: ``` String tableLocation = newTableLocation(); Configuration conf = new Configuration(); HadoopTables hadoopTables = new HadoopTables(conf); Table table = hadoopTables.create(schema, tableLocation); ``` Append the Parquet file to the table: ``` DataFile dataFile = DataFiles.builder(PartitionSpec.unpartitioned()) .withFileSizeInBytes(parquetFile.length()) .withPath(parquetFile.toString()) .withMetrics(fromInputFile(localInput(parquetFile))) .build(); table.newAppend().appendFile(dataFile).commit(); ``` Query the table via Spark: ``` Dataset<Row> df = spark.read() .format("iceberg") .load(tableLocation); df.show(); +----------+ |decimalCol| +----------+ | 3.50| +----------+ df.where("decimalCol = 3.50").show(); +----------+ |decimalCol| +----------+ +----------+ df.where("decimalCol = CAST(3.50 AS DECIMAL(10,2))").show(); +----------+ |decimalCol| +----------+ +----------+ ``` This happens because `ParquetMetrics` uses `ParquetConversions$fromParquetPrimitive`, which assumes that decimals are always represented as binary. In this particular case, `org.apache.parquet.column.statistics.Statistics$genericGetMin` will return an integer.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
