mderoy opened a new pull request, #8177:
URL: https://github.com/apache/iceberg/pull/8177
Problem:
Our application is using the following to read from the table
```
IcebergGenerics.ScanBuilder scanBuilder =
IcebergGenerics.read(iceberg_table);
CloseableIterable<Record> records =
scanBuilder.useSnapshot(snapshotId).build();
List<List<String>> output = new ArrayList<List<String>>();
for (Record record : records) {
```
however when I apply a case insensitive filter like
```
Expression filterExpr = ExpressionParser.fromJson(m_scanFilter);
scanBuilder = scanBuilder.where(filterExpr).caseInsensitive();
```
I get the below error despite setting the scan to caseInsensitive
```
2023-07-28 11:17:45.000208 EDT INFO: Scan of 'test.usethestats' using
provided filter: '{"type":"eq","term":"V1","value":1}'
Records in test.usethestats :
Error processing the request: Cannot find field 'V1' in struct: struct<1:
v1: optional int>
```
Solution:
The GenericReader is initializing a ParquetReader but not propagating the
caseSensitivity argument down to that reader...the same is true for the Orc
reader (and avro but I do not see that the avro reader has such an argument).
I've tested the fix with Parquet file using our application...I don't have a
good way to test with orc.
Testing:
```
With Filter:
2023-07-28 11:19:16.000684 EDT INFO: Scan of 'test.usethestats' using
provided filter: '{"type":"eq","term":"V1","value":1}'
Records in test.usethestats :
1
Output Without Filter (just to verify)
Records in test.usethestats :
1
20
200
```
let me know what additional things are needed to land this. I'm assuming
that the CI will run all the standard tests...maybe I should add a test for
this somewhere?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
