[
https://issues.apache.org/jira/browse/IMPALA-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821744#comment-17821744
]
Riza Suminto commented on IMPALA-12678:
---------------------------------------
Looping the test can hit this flakiness in few iterations. I think the test
expect REFRESH query in Impala to pull events before the event processor does.
But EventProcessor is faster than the REFRESH TABLE request.
[https://github.com/apache/impala/blob/3a6987e09b7f76b2b2c4c7c3b599cbdd4b509209/tests/custom_cluster/test_events_custom_configs.py#L416-L418]
{code:java}
I0227 21:32:22.598268 1552496 MetastoreEventsProcessor.java:909] Received 0
events. First event id : none.
I0227 21:32:22.598809 1552497 MetastoreEventsProcessor.java:1025] Latest event
in HMS: id=1270959, time=1709098339. Last synced event: id=1270959,
time=1709098334.
W0227 21:32:22.598910 1552497 MetastoreEventsProcessor.java:1028] Lag: 5s. 0
events pending to be processed.
I0227 21:32:27.605748 1552497 MetastoreEventsProcessor.java:1025] Latest event
in HMS: id=1270984, time=1709098347. Last synced event: id=1270959,
time=1709098339.
W0227 21:32:27.605820 1552497 MetastoreEventsProcessor.java:1028] Lag: 8s. 25
events pending to be processed.
I0227 21:32:27.606705 1552496 MetastoreEventsProcessor.java:909] Received 25
events. First event id : 1270960.
I0227 21:32:27.623741 1552496 MetastoreEvents.java:293] Total number of events
received: 25 Total number of events filtered out: 0
I0227 21:32:27.623901 1552496 MetastoreEvents.java:781] EventId: 1270983
EventType: ALTER_PARTITIONS Created a batch event for 24 events between 1270960
and 1270983
I0227 21:32:27.624006 1552496 CatalogServiceCatalog.java:1060] Not a self-event
since the given version is -1 and service id is empty
I0227 21:32:27.624522 1552496 MetastoreEvents.java:2610] Ignoring 24 events
between event id 1270960 and 1270983 since they modify parameters which can be
ignored
I0227 21:32:27.626438 1552496 MetastoreEventsProcessor.java:1191] Time elapsed
in processing event batch: 19.448ms
I0227 21:32:27.672039 1552673 JniUtil.java:166]
c143829f1529221c:32d4b46600000000] resetMetadata request: REFRESH TABLE
test_skipping_batching_events_b5611edb.test_batch_table issued by rsuminto
I0227 21:32:27.675635 1552673 CatalogServiceCatalog.java:2647]
c143829f1529221c:32d4b46600000000] Refreshing table metadata:
test_skipping_batching_events_b5611edb.test_batch_table
I0227 21:32:27.675818 1552581 JniUtil.java:166] getCatalogDelta request:
Getting catalog delta from version 1973
I0227 21:32:27.696142 1552673 HdfsTable.java:1251]
c143829f1529221c:32d4b46600000000] Reloading metadata for table definition and
all partition(s) of test_skipping_batching_events_b5611edb.test_batch_table
(REFRESH TABLE test_skipping_batching_events_b5611edb.test_batch_table issued
by rsuminto)
I0227 21:32:27.699833 1552673 HdfsTable.java:1886]
c143829f1529221c:32d4b46600000000] Loaded 13 columns from HMS. Actual columns:
13
I0227 21:32:27.722687 1552673 HdfsTable.java:3066]
c143829f1529221c:32d4b46600000000] Load Valid Write Id List Done. Time taken:
9.862us
I0227 21:32:27.730722 1552673 ParallelFileMetadataLoader.java:224]
c143829f1529221c:32d4b46600000000] Refreshing file and block metadata for 24
paths for table test_skipping_batching_events_b5611edb.test_batch_table using a
thread pool of size 5
I0227 21:32:27.750030 1552673 HdfsTable.java:821]
c143829f1529221c:32d4b46600000000] Loaded file and block metadata for
test_skipping_batching_events_b5611edb.test_batch_table partitions:
year=2010/month=7, year=2010/month=12, year=2010/month=3, and 21 others. Time
taken: 20.539ms
I0227 21:32:27.750385 1552673 HdfsTable.java:1289]
c143829f1529221c:32d4b46600000000] Incrementally loaded table metadata for:
test_skipping_batching_events_b5611edb.test_batch_table
I0227 21:32:27.750736 1552673 Table.java:259]
c143829f1529221c:32d4b46600000000] lastSyncedEventId_ for table:
test_skipping_batching_events_b5611edb.test_batch_table set from 1270957 to
1270984
I0227 21:32:27.750824 1552673 CatalogServiceCatalog.java:2708]
c143829f1529221c:32d4b46600000000] Refreshed table metadata:
test_skipping_batching_events_b5611edb.test_batch_table
I0227 21:32:27.750891 1552673 Table.java:1106]
c143829f1529221c:32d4b46600000000] last refreshed event id for table:
test_skipping_batching_events_b5611edb.test_batch_table set to: 1270984
I0227 21:32:27.757306 1552673 JniUtil.java:177]
c143829f1529221c:32d4b46600000000] Finished resetMetadata request: REFRESH
TABLE test_skipping_batching_events_b5611edb.test_batch_table issued by
rsuminto. Time spent: 86ms
I0227 21:32:27.757576 1552581 HdfsTable.java:2205] Skipped 24 partitions of
table test_skipping_batching_events_b5611edb.test_batch_table in the
incremental update
I0227 21:32:27.758225 1552581 catalog-server.cc:1158] Collected update:
1:TABLE:test_skipping_batching_events_b5611edb.test_batch_table, version=1974,
original size=4725, compressed size=2058
I0227 21:32:27.766546 1552581 catalog-server.cc:1158] Collected update:
1:CATALOG_SERVICE_ID, version=1974, original size=60, compressed size=58
I0227 21:32:27.766714 1552581 JniUtil.java:177] Finished getCatalogDelta
request: Getting catalog delta from version 1973. Time spent: 91ms
I0227 21:32:27.776623 1552589 catalog-server.cc:577] A catalog update with 2
entries is assembled. Catalog version: 1974 Last sent catalog version: 1973
I0227 21:32:27.844811 1554924 JniUtil.java:166] getEventProcessorSummary
request: Getting event processor summary
I0227 21:32:27.845082 1554924 JniUtil.java:177] Finished
getEventProcessorSummary request: Getting event processor summary. Time spent:
1ms {code}
> test_skipping_batching_events intermittent failure
> --------------------------------------------------
>
> Key: IMPALA-12678
> URL: https://issues.apache.org/jira/browse/IMPALA-12678
> Project: IMPALA
> Issue Type: Bug
> Reporter: Michael Smith
> Assignee: Sai Hemanth Gantasala
> Priority: Major
>
> custom_cluster.test_events_custom_configs.TestEventProcessingCustomConfigs.test_skipping_batching_events
> failed with
> {code}
> custom_cluster/test_events_custom_configs.py:408: in
> test_skipping_batching_events
> assert batch_events_2 == batch_events_1
> E assert 2 == 1
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
