zhongyujiang opened a new pull request #3910:
URL: https://github.com/apache/iceberg/pull/3910
This PR fixs jmh Spark parquet benchmark, now the
SparkParquetWritersFlatDataBenchmark throws the following exception:
```
# Warmup Iteration 2: <failure>
org.apache.iceberg.exceptions.AlreadyExistsException: File already exists:
/Users/zhongyujiang/projects/github/iceberg/spark/v3.2/spark/build/tmp/jmh/parquet-flat-data-benchmark6286636693589332554.parquet
at org.apache.iceberg.Files$LocalOutputFile.create(Files.java:58)
at
org.apache.iceberg.parquet.ParquetIO$ParquetOutputFile.create(ParquetIO.java:148)
at
org.apache.parquet.hadoop.ParquetFileWriter.<init>(ParquetFileWriter.java:329)
at
org.apache.parquet.hadoop.ParquetFileWriter.<init>(ParquetFileWriter.java:305)
at
org.apache.parquet.hadoop.ParquetFileWriter.<init>(ParquetFileWriter.java:284)
at
org.apache.iceberg.parquet.ParquetWriter.ensureWriterInitialized(ParquetWriter.java:114)
at
org.apache.iceberg.parquet.ParquetWriter.flushRowGroup(ParquetWriter.java:200)
at org.apache.iceberg.parquet.ParquetWriter.close(ParquetWriter.java:233)
at
org.apache.iceberg.spark.data.parquet.SparkParquetWritersFlatDataBenchmark.writeUsingIcebergWriter(SparkParquetWritersFlatDataBenchmark.java:105)
at
org.apache.iceberg.spark.data.parquet.jmh_generated.SparkParquetWritersFlatDataBenchmark_writeUsingIcebergWriter_jmhTest.writeUsingIcebergWriter_ss_jmhStub(SparkParquetWritersFlatDataBenchmark_writeUsingIcebergWriter_jmhTest.java:416)
at
org.apache.iceberg.spark.data.parquet.jmh_generated.SparkParquetWritersFlatDataBenchmark_writeUsingIcebergWriter_jmhTest.writeUsingIcebergWriter_SingleShotTime(SparkParquetWritersFlatDataBenchmark_writeUsingIcebergWriter_jmhTest.java:371)
```
This PR set the level of `TearDown` to `Level.Invocation` to clean up
`dataFile` for each benchmark method execution.
@aokolnychyi could you take a look? thx.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
