rangareddy commented on issue #17026:
URL: https://github.com/apache/hudi/issues/17026#issuecomment-5391448107
This issue was reviewed as part of the JIRA-migrated backlog triage
(HUDI-9443).
**Findings: confirmed, the reflection is still there and it is on the hot
path.**
In
`hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/spark/sql/execution/datasources/parquet/HoodieVectorizedParquetRecordReader.java`:
```java
:59 private Field batchIdxField;
...
:161 return columnarBatch == null ? super.getCurrentValue() :
columnarBatch.getRow(batchIdxFromSuper() - 1);
:164 private int batchIdxFromSuper() {
:166 if (batchIdxField == null) {
:167 batchIdxField =
VectorizedParquetRecordReader.class.getDeclaredField("batchIdx");
:168 batchIdxField.setAccessible(true);
```
The field handle is cached, so the per-row cost is a reflective `get` rather
than a lookup - but `getCurrentValue()` is called per row, so it is still on
the hot path.
Worth adding a second argument to the ticket beyond performance: reading a
**private Spark field by name** is fragile across Spark versions, and the
project now spans Spark 3.4 through 4.2 (with 3.3 reintroduced by #18956). A
rename or removal of `batchIdx` in any of those lines breaks this at runtime
rather than at compile time.
Keeping this open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
