[email protected] has posted comments on this change. ( http://gerrit.cloudera.org:8080/24324 )
Change subject: IMPALA-14936: [DOCS] Hierarchical metastore event processing ...................................................................... Patch Set 2: (14 comments) http://gerrit.cloudera.org:8080/#/c/24324/2/docs/topics/impala_metadata.xml File docs/topics/impala_metadata.xml: http://gerrit.cloudera.org:8080/#/c/24324/2/docs/topics/impala_metadata.xml@318 PS2, Line 318: values lower than 1 second : are also supported Instead, please recommend our current default value in HEP enabled case too. http://gerrit.cloudera.org:8080/#/c/24324/2/docs/topics/impala_metadata.xml@487 PS2, Line 487: and handle event : dependencies delete this highlighted text. http://gerrit.cloudera.org:8080/#/c/24324/2/docs/topics/impala_metadata.xml@489 PS2, Line 489: By default, the metastore event processor is single-threaded. HEP is enabled by default now. We try to describe the problem without HEP here. So need to rephrase it. May be something like - Without Hierarchical metastore event processing, metastore event processing is single-threaded... http://gerrit.cloudera.org:8080/#/c/24324/2/docs/topics/impala_metadata.xml@494 PS2, Line 494: <p>Waiting for a lock or file metadata loading of a table can slow event processing and Merge this with above paragraph. http://gerrit.cloudera.org:8080/#/c/24324/2/docs/topics/impala_metadata.xml@496 PS2, Line 496: for Hive Metastore (HMS) events Remove highlighted text. http://gerrit.cloudera.org:8080/#/c/24324/2/docs/topics/impala_metadata.xml@499 PS2, Line 499: <p>You can convert existing metastore event processing into multi-level event processing by : using the <parmname>enable_hierarchical_event_processing</parmname> flag. This method : partitions events based on their dependency, maintains the order of events within that : dependency, and processes them independently when possible. </p> Suggest change to - Hierarchical event processing routes events to database and table executors and uses synchronization barriers where strict ordering is required. This maintains event order within a table and across events that have explicit dependencies, while allowing independent tables to be processed in parallel. http://gerrit.cloudera.org:8080/#/c/24324/2/docs/topics/impala_metadata.xml@504 PS2, Line 504: <li>Event dispatcher – Dispatches metastore events and maintains a fixed pool of : database event executors. Event polling and dispatch - Polls HMS for notification events and dispatches them to database event executors. http://gerrit.cloudera.org:8080/#/c/24324/2/docs/topics/impala_metadata.xml@506 PS2, Line 506: <li>Database event executor – Manages table event dispatching, processes database : events, and manages table event executors. Database event executor — Processes database events and dispatches table events to table event executors. http://gerrit.cloudera.org:8080/#/c/24324/2/docs/topics/impala_metadata.xml@508 PS2, Line 508: Processes events for multiple tables by using table : processors. Processes table events http://gerrit.cloudera.org:8080/#/c/24324/2/docs/topics/impala_metadata.xml@524 PS2, Line 524: li>A transaction synchronization barrier is used when the system creates pseudo-commit : and pseudo-abort transaction events for each table involved, because : <cmdname>COMMIT_TXN</cmdname> and <cmdname>ABORT_TXN</cmdname> can involve multiple : tables. These pseudo events are then processed independently at their respective table : processors. </li> Remove the highlighted part. Because COMMIT_TXN and ABORT_TXN events do not need any synchronization. However, a single transaction event may require processing on multiple tables involved in the transaction. To allow parallel processing, the event processor decouples each such event into multiple pseudo-events, one per table. Each pseudo-event is then processed independently on its respective table processor, without waiting for pseudo-events from the same original event on other tables. http://gerrit.cloudera.org:8080/#/c/24324/2/docs/topics/impala_metadata.xml@534 PS2, Line 534: parmname use codeph. Renders in monospaced font http://gerrit.cloudera.org:8080/#/c/24324/2/docs/topics/impala_metadata.xml@538 PS2, Line 538: <concept id="config_event_processor"> : <title>Catalogd configuration properties</title> I am not sure about adding this Catalogd configuration properties section here. Because, catalogd have many other configuration parameters. These are just specific to HEP. If we have all catalogd configuration properties at some place, we can just append to it. If you decide to keep it here, "Catalogd configuration properties" section should be under "Hierarchical metastore event processing" http://gerrit.cloudera.org:8080/#/c/24324/2/docs/topics/impala_metadata.xml@563 PS2, Line 563: -level Please remove this highlighted text -> (-level) http://gerrit.cloudera.org:8080/#/c/24324/2/docs/topics/impala_metadata.xml@567 PS2, Line 567: -level Please remove this highlighted text -> (-level) -- To view, visit http://gerrit.cloudera.org:8080/24324 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib57bc4df70edf513a826a715d5bf842e9a580a34 Gerrit-Change-Number: 24324 Gerrit-PatchSet: 2 Gerrit-Owner: Sanjana Malhotra <[email protected]> Gerrit-Reviewer: Anonymous Coward <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Sanjana Malhotra <[email protected]> Gerrit-Comment-Date: Thu, 02 Jul 2026 05:29:00 +0000 Gerrit-HasComments: Yes
