[ https://issues.apache.org/jira/browse/IMPALA-8877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16918569#comment-16918569 ]
ASF subversion and git services commented on IMPALA-8877: --------------------------------------------------------- Commit 08ecb6d28cd7794a654f66fe41bb8ae727bc1a3e in impala's branch refs/heads/master from Anurag Mantripragada [ https://gitbox.apache.org/repos/asf?p=impala.git;h=08ecb6d ] IMPALA-8579: Ignore trivial alter table/partition events. Hive generates certain trivial alter events for eg: change only "transient_lastDdlTime". This is seen in the alter events accompanying an INSERT event. MetastoreEventProcessor should ignore such events as these trivial properties are not used by Impala and they cause unnecessary invalidates/refreshes. This change will also potentially reduce the likelihood of IMPALA-8877 as back to back invalidates after INSERT operations are avoided. Testing: Added tests respectively to testPartitionEvents() and testAlterTableEvent() in MetastoreEventProcessorTest class. Change-Id: I01a59d5170accc014f76f14eb526d96ddcf61f76 Reviewed-on: http://gerrit.cloudera.org:8080/14145 Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> > CatalogException during stress test: Table <foo> modified while operation was > in progress > ----------------------------------------------------------------------------------------- > > Key: IMPALA-8877 > URL: https://issues.apache.org/jira/browse/IMPALA-8877 > Project: IMPALA > Issue Type: Bug > Components: Catalog > Affects Versions: Impala 3.3.0 > Reporter: David Knupp > Assignee: Vihang Karajgaonkar > Priority: Critical > Attachments: catalogd.INFO.tar.gz, impalad.INFO.tar.gz > > > This was hit while running the stress tests to get a baseline on a deployed > cluster. > /* Mem: 12850 MB. Coordinator: quasar-mzmnbe-6.vpc.cloudera.com. */ > COMPUTE STATS catalog_sales > {noformat} > Query (id=924a50178a5a6146:29d58a7300000000) > Summary > Session ID: 5543fb9029e2b71f:f446381b1f59ed81 > Session Type: HIVESERVER2 > HiveServer2 Protocol Version: V6 > Start Time: 2019-08-19 01:26:07.292866000 > End Time: 2019-08-19 01:26:27.248053000 > Query Type: DDL > Query State: EXCEPTION > Query Status: CatalogException: Table > 'tpcds_300_decimal_parquet.catalog_sales' was modified while operation was in > progress, aborting execution. > Impala Version: impalad version 3.3.0-SNAPSHOT RELEASE (build > df3e7c051e2641524fc53a0cd07c2a14decd55f7) > User: syst...@vpc.cloudera.com > Connected User: syst...@vpc.cloudera.com > Delegated User: > Network Address: ::ffff:10.65.6.19:39174 > Default Db: tpcds_300_decimal_parquet > Sql Statement: /* Mem: 12850 MB. Coordinator: > quasar-mzmnbe-6.vpc.cloudera.com. */ > COMPUTE STATS catalog_sales > Coordinator: quasar-mzmnbe-6.vpc.cloudera.com:22000 > Query Options (set by configuration): > ABORT_ON_ERROR=1,MEM_LIMIT=13474201600,MT_DOP=4,EXEC_TIME_LIMIT_S=2147483647,TIMEZONE=America/Los_Angeles,DEFAULT_FILE_FORMAT=4,DEFAULT_TRANSACTIONAL_TYPE=1 > Query Options (set by configuration and planner): > ABORT_ON_ERROR=1,MEM_LIMIT=13474201600,MT_DOP=4,EXEC_TIME_LIMIT_S=2147483647,TIMEZONE=America/Los_Angeles,DEFAULT_FILE_FORMAT=4,DEFAULT_TRANSACTIONAL_TYPE=1 > DDL Type: COMPUTE_STATS > Query Compilation > Metadata of all 1 tables cached: 5.62s (5622372318) > Analysis finished: 5.62s (5622560027) > Authorization finished (noop): 5.62s (5622568284) > Retried query planning due to inconsistent metadata 7 of 40 times: > Catalog object TCatalogObject(type:TABLE, catalog_version:94204, > table:TTable(db_name:tpcds_300_decimal_parquet, tbl_name:catalog_sales)) > changed version between accesses.: 5.95s (5949859598) > Planning finished: 5.95s (5949861145) > Query Timeline > Query submitted: 0ns (0) > Planning finished: 5.95s (5950024020) > Child queries finished: 17.85s (17849072057) > Rows available: 19.82s (19825080035) > Unregister query: 19.95s (19955080560) > Frontend > - CatalogFetch.ColumnStats.Misses: 34 (34) > - CatalogFetch.ColumnStats.Requests: 34 (34) > - CatalogFetch.ColumnStats.Time: 0 (0) > - CatalogFetch.Config.Hits: 1 (1) > - CatalogFetch.Config.Requests: 1 (1) > - CatalogFetch.Config.Time: 0 (0) > - CatalogFetch.DatabaseList.Hits: 8 (8) > - CatalogFetch.DatabaseList.Requests: 8 (8) > - CatalogFetch.DatabaseList.Time: 0 (0) > - CatalogFetch.PartitionLists.Misses: 1 (1) > - CatalogFetch.PartitionLists.Requests: 1 (1) > - CatalogFetch.PartitionLists.Time: 7 (7) > - CatalogFetch.Partitions.Hits: 1837 (1837) > - CatalogFetch.Partitions.Misses: 1837 (1837) > - CatalogFetch.Partitions.Requests: 3674 (3674) > - CatalogFetch.Partitions.Time: 325 (325) > - CatalogFetch.RPCs.Bytes: 4.7 MiB (4936030) > - CatalogFetch.RPCs.Requests: 22 (22) > - CatalogFetch.RPCs.Time: 343 (343) > - CatalogFetch.TableNames.Hits: 4 (4) > - CatalogFetch.TableNames.Misses: 4 (4) > - CatalogFetch.TableNames.Requests: 8 (8) > - CatalogFetch.TableNames.Time: 0 (0) > - CatalogFetch.Tables.Misses: 8 (8) > - CatalogFetch.Tables.Requests: 8 (8) > - CatalogFetch.Tables.Time: 74 (74) > - InactiveTotalTime: 0ns (0) > - TotalTime: 0ns (0) > ImpalaServer > - CatalogOpExecTimer: 1.97s (1972007962) > - ClientFetchWaitTimer: 0ns (0) > - InactiveTotalTime: 0ns (0) > - RowMaterializationTimer: 0ns (0) > - TotalTime: 0ns (0) > Child Queries > Table Stats Query (id=db4821e4aa5bb04d:d4a5ae4500000000) > Column Stats Query (id=0444367557e3496d:f943511100000000) > {noformat} -- This message was sent by Atlassian Jira (v8.3.2#803003) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org