parthchandra commented on code in PR #1187:
URL: https://github.com/apache/parquet-mr/pull/1187#discussion_r1428231805
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -123,10 +123,19 @@ public class ParquetFileReader implements Closeable {
public static String PARQUET_READ_PARALLELISM =
"parquet.metadata.read.parallelism";
+ private ParquetMetricsCallback metricsCallback;
+
private final ParquetMetadataConverter converter;
private final CRC32 crc;
+ /**
+ * set a callback to send back metrics info
+ */
+ public void initMetrics(ParquetMetricsCallback callback) {
Review Comment:
That's a good idea, but it may require changes to some of the public
constructors which are in use by others. For instance Spark has this
https://github.com/apache/spark/blob/15e99cf676d9de02c54ca5ebe9a2bc6a3ce014e5/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java#L104
In this case, the constructor is deprecated and it looks like Spark could
benefit from a constructor that takes a ParquetReadOptions as a parameter. I
can introduce a new constructor but wasn't sure that we would want to do so.
Having an additional API to set the metrics callback seemed like the simpler
option.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
