viirya commented on PR #59879: URL: https://github.com/apache/spark/pull/59879#issuecomment-6509837795
Thanks for the review. The anchored threshold was changed in round 2: a pattern that starts with `^` or `\A`, outside `(?m)` and without a top-level `|`, now uses the threshold of 3. On the streaming question, I agree that checkpoints written by 4.2.0 or later should keep the old placement. This PR adds an internal config, `spark.sql.optimizer.estimateRegexPatternCost`, which defaults to `true`. When it is `false`, every LIKE and RLIKE is expensive, as in 4.2.0. The config is recorded in the offset log. A checkpoint without it uses `false`, the same way `pruneFiltersCanPruneStreamingSubplan` is handled. A query resumed from a 4.2.x or 4.3.x checkpoint therefore keeps its filter above the join. One from a 4.1 checkpoint also takes the 4.2.0 placement. That moves the filter up rather than down, which does not change results. With this in place the fix is also safe to backport to `branch-4.2` and `branch-4.3`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
