Karen Coppage created HIVE-24001:
------------------------------------
Summary: Don't cache MapWork in tez/ObjectCache during query-based
compaction
Key: HIVE-24001
URL: https://issues.apache.org/jira/browse/HIVE-24001
Project: Hive
Issue Type: Bug
Reporter: Karen Coppage
Assignee: Karen Coppage
Query-based major compaction can fail intermittently with the following issue:
{code:java}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: One writer is
supposed to handle only one bucket. We saw these 2 different buckets: 1 and 6
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDFValidateAcidSortOrder.evaluate(GenericUDFValidateAcidSortOrder.java:77)
{code}
This is consistently preceded in the application log with:
{code:java}
[INFO] [TezChild] |tez.ObjectCache|: Found
hive_20200804185133_f04cca69-fa30-4f1b-a5fe-80fc2d749f48_Map 1__MAP_PLAN__ in
cache with value: org.apache.hadoop.hive.ql.plan.MapWork@74652101
{code}
Alternatively, when MapRecordProcessor doesn't find mapWork in tez/ObjectCache
(but instead caches mapWork), major compaction succeeds.
The failure happens because, if MapWork is reused,
GenericUDFValidateAcidSortOrder (which is called during compaction) is also
reused on splits belonging to two different buckets, which produces an error.
Solution is to avoid storing MapWork in the ObjectCache during query-based
compaction.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
