beliefer commented on issue #26875: [SPARK-30245][SQL] Add cache for Like and RLike when pattern is not static URL: https://github.com/apache/spark/pull/26875#issuecomment-601999002 @HyukjinKwon After many tests, I found that this modification did not bring a clear performance improvement. ``` spark-sql> select count(*) from bigtable; 360014110 ``` I test the sql show below: `select count(*) from bigtable where string(ws_ship_customer_sk) like '/%SystemDrive/%//Users%';` Spark3.1.0 Time | Min | 25th percentile | Median | 75th percentile | Max -- | -- | -- | -- | -- | -- 2.4 min | 4 s | 9 s | 13 s | 32 s | 2.2 min 1.3 min | 6 s | 9 s | 12 s | 17 s | 56 s 1.4 min | 6 s | 11 s | 13 s | 17 s | 1.1 min 1.2 min | 6 s | 11 s | 14 s | 20 s | 40 s 1.1 min | 4 s | 10 s | 12 s | 15 s | 38 s 2.6 min | 5 s | 9 s | 12 s | 18 s | 2.5 min Spark3.1.0 apply this PR Time | Min | 25th percentile | Median | 75th percentile | Max -- | -- | -- | -- | -- | -- 1.1 min | 5 s | 9 s | 12 s | 16 s | 40 s 46 s | 7 s | 9 s | 11 s | 14 s | 37 s 1.4 min | 7 s | 11 s | 14 s | 18 s | 1.0 min 2.2 min | 5 s | 11 s | 13 s | 19 s | 1.9 min 1.0 min | 4 s | 10 s | 12 s | 14 s | 50 s 2.0 min | 6 s | 9 s | 12 s | 15 s | 1.9 min So I think we should not make this change and I will close https://github.com/apache/spark/pull/27497
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
