David Mollitor created SPARK-59543:
--------------------------------------
Summary: Avoid a per-record SQLConf.get in the JSON charset-decode
path
Key: SPARK-59543
URL: https://issues.apache.org/jira/browse/SPARK-59543
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.1.0
Reporter: David Mollitor
h2. Summary
When reading JSON with an explicit {{encoding}} option, Spark re-materializes
SQLConf *once per input record*. {{CreateJacksonParser.getStreamDecoder}}
builds a {{CharsetDecoder}} per record via {{{}CharsetProvider.newDecoder(enc,
caller = "Jackson Parser"){}}}, and {{{}newDecoder{}}}'s two default arguments
each call {{{}SQLConf.get{}}}:
{code:scala}
def newDecoder(charset: String,
legacyCharsets: Boolean = SQLConf.get.legacyJavaCharsets,
legacyErrorAction: Boolean = SQLConf.get.legacyCodingErrorAction,
caller: String = "decode"): CharsetDecoder
{code}
On an executor task thread {{SQLConf.get}} constructs a fresh
{{ReadOnlySQLConf}} +
{{ConfigReader}} + {{HashMap}} each call, so this is pure per-record churn to
read two flags that are constant for the whole task.
h2. Details
The single-line/text read path is affected:
{{CreateJacksonParser.text/internalRow/bytes(enc, ...)}}
call {{{}getStreamDecoder{}}}. (The multi-line path uses {{new
InputStreamReader(is, enc)}} and does not hit {{{}newDecoder{}}}, so it is
unaffected.)
The fix resolves the two flags once per reader in {{JSONOptions}} and passes
them explicitly to {{{}newDecoder{}}}, so no {{SQLConf.get}} runs per record.
This mirrors what CSV already does: {{CSVOptions}} resolves
{{legacyJavaCharsets}} at construction via
{{{}CharsetProvider.forName(_, SQLConf.get.legacyJavaCharsets, ...){}}}.
* {{{}JSONOptions{}}}: add {{val legacyJavaCharsets}} and {{{}val
legacyCodingErrorAction{}}}, resolved from
{{SQLConf.get}} at construction (matching the existing SQLConf-derived fields
such as {{{}writeNullIfWithDefaultValue{}}}).
* {{{}CreateJacksonParser{}}}: thread the two flags through
{{getStreamDecoder}} and the {{{}text{}}}, {{{}internalRow{}}}, and {{bytes}}
encoding variants into {{{}CharsetProvider.newDecoder{}}}.
* {{{}TextInputJsonDataSource{}}}: pass the resolved flags from the options at
the three closure sites.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
