cloud-fan commented on a change in pull request #26563: [SPARK-29758][SQL][2.4]
Fix truncation of requested string fields in `json_tuple`
URL: https://github.com/apache/spark/pull/26563#discussion_r347229794
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
##########
@@ -468,11 +468,13 @@ case class JsonTuple(children: Seq[Expression])
new GenericInternalRow(row) :: Nil
}
+ private val optimizeStringCopy =
SQLConf.get.getConf(SQLConf.OPTIMIZE_STRING_COPY_IN_JSON_TUPLE)
+
private def copyCurrentStructure(generator: JsonGenerator, parser:
JsonParser): Unit = {
parser.getCurrentToken match {
// if the user requests a string field it needs to be returned without
enclosing
// quotes which is accomplished via JsonGenerator.writeRaw instead of
JsonGenerator.write
- case JsonToken.VALUE_STRING if parser.hasTextCharacters =>
+ case JsonToken.VALUE_STRING if optimizeStringCopy &&
parser.hasTextCharacters =>
Review comment:
I'd simply remove this optimization. correctness is critical.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
