maropu commented on a change in pull request #26965:
URL: https://github.com/apache/spark/pull/26965#discussion_r411792481
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
##########
@@ -101,8 +101,9 @@ private[this] object JsonPathParser extends RegexParsers {
private[this] object SharedFactory {
val jsonFactory = new JsonFactoryBuilder()
- // Enabled for Hive compatibility
+ // The two options below enabled for Hive compatibility
.enable(JsonReadFeature.ALLOW_UNESCAPED_CONTROL_CHARS)
+ .enable(JsonReadFeature.ALLOW_SINGLE_QUOTES)
Review comment:
> I am afraid that end users expect all our JSON functions have the same
behavior. That means, either all the functions support SINGLE QUOTE or none of
them supports it. Make the behavior configurable? For reading the external JSON
data sources, users can control it
If we make it configurable, could we make all the behaivours configurable by
using `JSONOptions` here as max suggested in
https://github.com/apache/spark/pull/26965/files#r360562916 ? Some users might
expect the same behaivours between json functions and json external reading,
too.
> BTW, will this change cause the query result difference?
Ah, I think yes; after this PR, some query results have changed from `null`
to a correct answer. We need to update the migration guide for 3.0.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
