WangGuangxin opened a new pull request, #36328:
URL: https://github.com/apache/spark/pull/36328
### What changes were proposed in this pull request?
Push down StringEndsWith/Contains to Parquet so that we can leverage Parquet
Dictionary Filtering
### Why are the changes needed?
Improve performance.
FilterPushDownBenchmark:
```
Java HotSpot(TM) 64-Bit Server VM 1.8.0_181-b13 on Mac OS X 10.16
Intel(R) Core(TM) i7-1068NG7 CPU @ 2.30GHz
StringEndsWith filter: (value like '%10'): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------
Parquet Vectorized 6724 6938
283 2.3 427.5 1.0X
Parquet Vectorized (Pushdown) 643 714
55 24.5 40.9 10.5X
Native ORC Vectorized 7364 8120
452 2.1 468.2 0.9X
Native ORC Vectorized (Pushdown) 8060 8586
567 2.0 512.4 0.8X
```
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
added UT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
