MaxGekk commented on a change in pull request #26563: [SPARK-29758][SQL][2.4]
Fix truncation of requested string fields in `json_tuple`
URL: https://github.com/apache/spark/pull/26563#discussion_r347236196
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
##########
@@ -468,11 +468,13 @@ case class JsonTuple(children: Seq[Expression])
new GenericInternalRow(row) :: Nil
}
+ private val optimizeStringCopy =
SQLConf.get.getConf(SQLConf.OPTIMIZE_STRING_COPY_IN_JSON_TUPLE)
+
private def copyCurrentStructure(generator: JsonGenerator, parser:
JsonParser): Unit = {
parser.getCurrentToken match {
// if the user requests a string field it needs to be returned without
enclosing
// quotes which is accomplished via JsonGenerator.writeRaw instead of
JsonGenerator.write
- case JsonToken.VALUE_STRING if parser.hasTextCharacters =>
+ case JsonToken.VALUE_STRING if optimizeStringCopy &&
parser.hasTextCharacters =>
Review comment:
Initially, I thought of removing the optimization but just hesitated that it
could impact on users who have small fields. ok, let's remove it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
