[
https://issues.apache.org/jira/browse/IMPALA-14628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18046715#comment-18046715
]
ASF subversion and git services commented on IMPALA-14628:
----------------------------------------------------------
Commit 1cc9ae42a8970b03a774d9e8e97ed9b4718bf1bf in impala's branch
refs/heads/master from Venu Reddy
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=1cc9ae42a ]
IMPALA-14628: Fix mismatch between greatest synced event time and last synced
event time
The greatest synced event id and its corresponding event time
must match the last synced event id and time once all outstanding
events have been fully processed.
Previously, for batch events, timestamp of the first event in
the batch is recorded when transitioning the batch event from
the in-progress log to the processed log in EventExecutorService.
As a result, the greatest synced event time incorrectly reflected
the first event’s timestamp, even though the greatest synced
event id corresponded to the last event in the batch, leading to
a mismatch.
This fix ensures that, for batch events, the timestamp of the
last event in the batch is recorded when the batch is marked as
processed. Consequently, the greatest synced event time now
correctly aligns with the last synced event id after all
outstanding events are processed.
Testing:
- Added an FE test and ran the existing tests.
Change-Id: I26550441bc0d5e7736c3b900c3e5178103157446
Reviewed-on: http://gerrit.cloudera.org:8080/23797
Reviewed-by: Quanlong Huang <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> FE tests in MetastoreEventsProcessorTest are flaky at greatestSyncedEventTime
> != lastSyncedEventTime
> ----------------------------------------------------------------------------------------------------
>
> Key: IMPALA-14628
> URL: https://issues.apache.org/jira/browse/IMPALA-14628
> Project: IMPALA
> Issue Type: Bug
> Reporter: Quanlong Huang
> Assignee: Venugopal Reddy K
> Priority: Critical
> Attachments:
> FeSupport.impala-ec2-redhat86-m6i-4xlarge-ondemand-0ede.vpc.cloudera.com.jenkins.log.INFO.20251215-041130.1530635.gz,
> FeSupport.quanlong-Precision-3680.quanlong.log.INFO.20251215-110610.791447
>
>
> FE test MetastoreEventsProcessorTest.testDisableEventSyncFlag is flaky. Here
> is an error in a downstream run:
> {noformat}
> java.lang.AssertionError: expected:<1765626451> but was:<1765626452>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.impala.catalog.events.SynchronousHMSEventProcessorForTests.verifyEventSyncedMetrics(SynchronousHMSEventProcessorForTests.java:78)
> at
> org.apache.impala.catalog.events.SynchronousHMSEventProcessorForTests.processEvents(SynchronousHMSEventProcessorForTests.java:53)
> at
> org.apache.impala.catalog.events.MetastoreEventsProcessorTest.testDDLOpUsingEvent(MetastoreEventsProcessorTest.java:2291)
> at
> org.apache.impala.catalog.events.MetastoreEventsProcessorTest.runDDLTestsWithFlags(MetastoreEventsProcessorTest.java:2145)
> at
> org.apache.impala.catalog.events.MetastoreEventsProcessorTest.testDisableEventSyncFlag(MetastoreEventsProcessorTest.java:2091)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:316)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:240)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:214)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:155)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:385)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
> at
> org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:507)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:495){noformat}
> The failed assertion is "greatestSyncedEventTime == lastSyncedEventTime" in
> the following code:
> {code}
> Assert.assertEquals(greatestSyncedEventId, lastSyncedEventId);
> Assert.assertEquals(greatestSyncedEventTime, lastSyncedEventTime); // failed
> here{code}
> https://github.com/apache/impala/blob/9d112dae23870b6729473047da94f1bc0ea89ceb/fe/src/test/java/org/apache/impala/catalog/events/SynchronousHMSEventProcessorForTests.java#L78
> This means the event ids are identical but the event time are different.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
