bersprockets commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r774270256
##########
File path:
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -103,9 +104,14 @@ public void initialize(InputSplit inputSplit,
TaskAttemptContext taskAttemptCont
fileReader.setRequestedSchema(requestedSchema);
String sparkRequestedSchemaString =
configuration.get(ParquetReadSupport$.MODULE$.SPARK_ROW_REQUESTED_SCHEMA());
- this.sparkSchema =
StructType$.MODULE$.fromString(sparkRequestedSchemaString);
+ StructType sparkRequestedSchema =
StructType$.MODULE$.fromString(sparkRequestedSchemaString);
+ ParquetToSparkSchemaConverter converter = new
ParquetToSparkSchemaConverter(configuration);
+ this.parquetColumn = converter.convertParquetColumn(requestedSchema,
Review comment:
This line has a performance impact on parquet files with very wide
records.
Take, for example, with a parquet dataset with:
- 6000 bigint columns
- 25 files
- 5000 records per file (total of 125000 records)
Reading all the records in this parquet dataset with a single executor
thread is 42% slower with this PR than with the baseline (commit f361ad8d2e on
the master branch).
The main culprit is the n**2 issue
[here](https://github.com/apache/spark/blob/f361ad8d2eb067ec86483a294771f1976f31a981/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala#L95)
in ParquetToSparkSchemaConverter#convertInternal, which was not exercised
anywhere except by unit tests until your PR (as far as I can tell).
By the way, even with a patch for the above n**2 issue, this PR is still 10%
slower than the baseline when testing the wide row case. I haven't confirmed
it, but I think this is due to the extra OnHeapColumnVector instances created
in ParquetColumnVector, for which you already have a TODO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
