David Mollitor created SPARK-59551:
--------------------------------------
Summary: Avoid recomputing existence default values per row in the
Parquet, ORC, and CSV readers
Key: SPARK-59551
URL: https://issues.apache.org/jira/browse/SPARK-59551
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.1.0
Reporter: David Mollitor
h2. Summary
When a row-based reader materializes each record it applies "existence default
values" (the value used for a column that is missing from the data but has a
DEFAULT in the schema). The readers do this by calling
{{ResolveDefaultColumns.applyExistenceDefaultValuesToRow(schema, row, bitmask
}}and {{{}resetExistenceDefaultsBitmask(schema, bitmask){}}}) once per record,
and those methods call {{existenceDefaultValues(schema)}} internally:
{code:scala}
def existenceDefaultValues(schema: StructType): Array[Any] =
schema.fields.map(getExistenceDefaultValue) // allocates a new Array[Any];
for any field with
// a default, parses +
evaluates its expression
{code}
This is a pure function of the schema (it depends only on field metadata, never
on row data) and is not cached, so it is recomputed on every record.
{{applyExistenceDefaultValuesToRow}} recomputes it
and also calls {{{}hasExistenceDefaultValues(schema){}}}, which recomputes it
again. The utility even documents this hazard: "The caller should avoid using
such methods in a loop for efficiency." Because the recompute happens whether
or not the schema actually has any defaults, even the common no-default read
pays the per-record array allocation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
