MaxGekk opened a new pull request #26563: [SPARK-29758][SQL][2.4] Fix truncation of request string field in `json_tuple` URL: https://github.com/apache/spark/pull/26563 ### What changes were proposed in this pull request? In the PR, I propose new SQL config `spark.sql.optimizeStringCopyInJsonTuple.enabled` to control copying of requested string field: - If it is enabled, `json_tuple` uses the underlying array of character in copying requested string field - otherwise it request Jackson Core to allocate a String while coping the field By default, the config is set to `false` that disables the optimization. ### Why are the changes needed? Spark 2.4 uses Jackson Core 2.6.7 which has a bug in copying string. This bug may lead to truncation of results in some cases. The bug has been already fixed by the commit https://github.com/FasterXML/jackson-core/commit/554f8db0f940b2a53f974852a2af194739d65200 which is a part of Jackson Core since the version 2.7.7. Upgrading Jackson Core up to 2.7.7 or later version is risky. That's why this PR propose to avoid using buggy methods of Jackson Core 2.6.7. ### Does this PR introduce any user-facing change? No ### How was this patch tested? By new test added to `JsonFunctionsSuite`
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
