[
https://issues.apache.org/jira/browse/IMPALA-12463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17770671#comment-17770671
]
Csaba Ringhofer commented on IMPALA-12463:
------------------------------------------
>But it will be very hard to support referential integrity and transactions in
>the future if the event processor starts to reorder events.
I would separate external, ACID and Iceberg tables here.
External: having cross table consistency for external tables seems like a lost
cause to me. Anyone can write into them any time, we may or may not get an
event about that. Even if there is an event, after the resulting refresh
catalogd won't see only files created by that insert, but also later inserts to
the same partition.
Hive ACID: there will be a commit event, and according to the logic in
description that should "cut" batching of events ,so we won't batch events
before and after the commit together. This means that events for actually
different INSERTs won't be batched. I am not 100% sure about ALTER PARTITION
events, but generally altering should also have its own transaction, and as for
INSERT, the batching won't go past those.
Note that I didn't check how exactly we handle events for ACID tables - ideally
Impala should only do refresh after commit and only if it changed the
validWriteIds
Iceberg: I guess that catalogd won't see partition level events at all, so the
batching doesn't seem relevant.
> Allow batching of non consecutive metastore events
> --------------------------------------------------
>
> Key: IMPALA-12463
> URL: https://issues.apache.org/jira/browse/IMPALA-12463
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog
> Reporter: Csaba Ringhofer
> Assignee: Joe McDonnell
> Priority: Major
> Attachments: concurrent_metadata_load.py
>
>
> Currently Impala tries to batch events like partition insert/creation only if:
> 1. the next event is for the same table as the previous one
> 2. the next event's id is the previous one's + 1
> 3. the next event has the same type as the previous one
> (2 can be stricter than 1 if some events were filtered between the two)
> See
> https://github.com/apache/impala/blob/94f4f1d82461d8f71fbd0d2e9082aa29b5f53a89/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#L315
> Another limit is that only events in the same batch from HMS can be merged.
> Currently 1000 events are polled at the same time:
> https://github.com/apache/impala/blob/94f4f1d82461d8f71fbd0d2e9082aa29b5f53a89/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java#L218
> Making this configurable could be also useful.
> Event batching could be improved by batching all events to the current one if
> they modify the same table, unless they are "cut" by:
> a. an event on the same table but with a different type
> b. a rename table event where the original or the new name is the same as the
> current event
> If such an event occurs, the events after that can be only merged to a newer
> event.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
