[
https://issues.apache.org/jira/browse/SPARK-59551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59551:
-----------------------------------
Labels: pull-request-available (was: )
> Avoid recomputing existence default values per row in the Parquet, ORC, and
> CSV readers
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-59551
> URL: https://issues.apache.org/jira/browse/SPARK-59551
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.1.0
> Reporter: David Mollitor
> Priority: Minor
> Labels: pull-request-available
>
> h2. Summary
> When a row-based reader materializes each record it applies "existence
> default values" (the value used for a column that is missing from the data
> but has a DEFAULT in the schema). The readers do this by calling
> {{ResolveDefaultColumns.applyExistenceDefaultValuesToRow(schema, row, bitmask
> }}and {{{}resetExistenceDefaultsBitmask(schema, bitmask){}}}) once per
> record, and those methods call {{existenceDefaultValues(schema)}} internally:
> {code:scala}
> def existenceDefaultValues(schema: StructType): Array[Any] =
> schema.fields.map(getExistenceDefaultValue) // allocates a new
> Array[Any]; for any field with
> // a default, parses +
> evaluates its expression
> {code}
> This is a pure function of the schema (it depends only on field metadata,
> never on row data) and is not cached, so it is recomputed on every record.
> {{applyExistenceDefaultValuesToRow}} recomputes it
> and also calls {{{}hasExistenceDefaultValues(schema){}}}, which recomputes it
> again. The utility even documents this hazard: "The caller should avoid using
> such methods in a loop for efficiency." Because the recompute happens whether
> or not the schema actually has any defaults, even the common no-default read
> pays the per-record array allocation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
