[
https://issues.apache.org/jira/browse/SPARK-60945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lihang Shen updated SPARK-60945:
--------------------------------
Description:
When a table write triggers a cache refresh check, Spark compares the plan for
the updated table with cached plans. The comparison uses the canonicalized form
of both plans. With concurrent writes, different threads can end up
canonicalizing the same cached plan at the same time.
If the cached plan contains \{{from_protobuf(..., map(...))}}, canonicalization
can create new \{{FromProtobuf}} expressions when it renumbers column IDs.
These copies still share the same options expression. Canonicalization also
evaluates those options. When they use \{{CreateMap}}, the threads can modify
the same internal map builder concurrently, producing duplicate keys or
mismatched key/value array lengths. This causes \{{DUPLICATED_MAP_KEY}} or
\{{IllegalArgumentException: requirement failed}}.
The fix calls \{{options.freshCopyIfContainsStatefulExpression()}} before
evaluating options in both \{{FromProtobuf}} and \{{ToProtobuf}}, so each
evaluation uses its own map builder.
was:
{\{FromProtobuf.replacement}} and \{{ToProtobuf.replacement}} evaluate
\{{options}} directly. When options come from an expression like \{{map('mode',
'PERMISSIVE')}}, this reuses the mutable map builder inside \{{CreateMap}}.
When plan canonicalization changes attribute IDs, it creates a new protobuf
expression but keeps the same options child. Each copy has its own
\{{replacement}} lazy-val lock, so two threads can still evaluate the shared
\{{CreateMap}} at the same time. Both modify the same
\{{ArrayBasedMapBuilder}}, which can cause \{{DUPLICATED_MAP_KEY}} or
\{{IllegalArgumentException: requirement failed}} in \{{ArrayBasedMapData}}.
> from_protobuf/to_protobuf can corrupt a shared options map when the plan is
> canonicalized concurrently
> ------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60945
> URL: https://issues.apache.org/jira/browse/SPARK-60945
> Project: Spark
> Issue Type: Bug
> Components: Protobuf, SQL
> Affects Versions: 4.1.0, 4.0.0, 4.2.0
> Reporter: Lihang Shen
> Priority: Major
>
> When a table write triggers a cache refresh check, Spark compares the plan
> for the updated table with cached plans. The comparison uses the
> canonicalized form of both plans. With concurrent writes, different threads
> can end up canonicalizing the same cached plan at the same time.
> If the cached plan contains \{{from_protobuf(..., map(...))}},
> canonicalization can create new \{{FromProtobuf}} expressions when it
> renumbers column IDs. These copies still share the same options expression.
> Canonicalization also evaluates those options. When they use \{{CreateMap}},
> the threads can modify the same internal map builder concurrently, producing
> duplicate keys or mismatched key/value array lengths. This causes
> \{{DUPLICATED_MAP_KEY}} or \{{IllegalArgumentException: requirement failed}}.
> The fix calls \{{options.freshCopyIfContainsStatefulExpression()}} before
> evaluating options in both \{{FromProtobuf}} and \{{ToProtobuf}}, so each
> evaluation uses its own map builder.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
