panbingkun commented on PR #43751: URL: https://github.com/apache/spark/pull/43751#issuecomment-1968709145
> This is a hard decision. Technically the behavior of LIKE in many commands (`SHOW TABLES LIKE ...`) relies on the underlying catalog, which can be HMS of different versions, or a Hive-compatible metastore service. This is out of Spark's control. > > From @panbingkun's [investigation](https://github.com/apache/spark/pull/43751#issuecomment-1953619718), the Hive behavior is actually very weird and different from other main-stream SQL systems (they follow the same behavior of the LIKE expression). Hive 4.0 also switches to the more common behavior. > > There are some commands that we implement the LIKE filtering by our own, following the Hive behavior. Now we are in a hard position: > > 1. If we do nothing, then Spark's behavior of LIKE in commands is non-standard and different from other databases. We may also hit future behavior changes if we upgrade to Hive 4.0. > 2. If we change the LIKE filtering behavior now, it's a breaking change, and also lead to inconsistent behaviors as some commands use Hive to do LIKE filtering. > > cc @srielau After efforts, all commands that support syntax `Like <pattern>` have been changed. By adding a configuration `spark.sql.legacy.useVerticalBarAndStarAsWildcardsInLikePattern` (default value `false`), when it is `true`, the wildcards supported by the `pattern` of `Like` are consistent with the semantics supported before `Hive version 4` (use '*' for any character(s) and '|' for a choice as wildcards). If it is false, their behavior is consistent with the semantics of `SQL Like` (use '%' for any character(s) and '_' for a single character as wildcards), and `the document` has also been updated synchronously. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
