Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/12700 )
Change subject: IMPALA-7974: [DOCS] Document automatic invalidation via HMS notification ...................................................................... Patch Set 3: (8 comments) Thanks documenting the automatic invalidate feature! I left some suggestions for improvement below. http://gerrit.cloudera.org:8080/#/c/12700/3/docs/topics/impala_metadata.xml File docs/topics/impala_metadata.xml: http://gerrit.cloudera.org:8080/#/c/12700/3/docs/topics/impala_metadata.xml@175 PS3, Line 175: HMS events generated by updates done via other components, such as Spark, will not Can we rephrase this to something like below: Certain operations done via other components like Apache Hive (such as Apache Spark) which add new data to existing tables or partitions do not trigger HMS events currently. Hence such Impala tables are not invalidated. We recommend using LOAD DATA INPATH command in Hive to load data into existing tables and partitions to support such use-cases. In case, the ETL jobs only adds new partitions or tables, the HMS events are generated and hence the feature should work without the need to manually invalidate the tables. http://gerrit.cloudera.org:8080/#/c/12700/3/docs/topics/impala_metadata.xml@196 PS3, Line 196: for all databases and tables. You can disable the automatic invalidation via HMS May be rephrase this to something like: Users who wish to have fine-grained control on which tables or databases need to be synced using events can use "impala.disableHmsSync" property to disable the event processing at the table or database level. http://gerrit.cloudera.org:8080/#/c/12700/3/docs/topics/impala_metadata.xml@226 PS3, Line 226: <codeblock>CREATE DATABASE <name> WITH DBPROPERTIES ('impala.disableHmsSync'='true');</codeblock> Can we confirm this with someone else too? I am new to Impala and I relied on my limited knowledge for this. I know that alter database is not supported but not 100% sure on if create database with dbproperties. I will check with some other folks and confirm. http://gerrit.cloudera.org:8080/#/c/12700/3/docs/topics/impala_metadata.xml@231 PS3, Line 231: DATABASE</codeph> statement in Hive. Impala does not support <codeph>ALTER Not sure if the convention is to use "Hive" or "Apache Hive"? Same comment for Spark http://gerrit.cloudera.org:8080/#/c/12700/3/docs/topics/impala_metadata.xml@258 PS3, Line 258: latest. May as well add, that in such a case, the status of the event processor changes to "NEEDS_INVALIDATE" http://gerrit.cloudera.org:8080/#/c/12700/3/docs/topics/impala_metadata.xml@289 PS3, Line 289: Under the web UI, there are two pages that presents the metrics for HMS sync processor change sync processor to event processor. http://gerrit.cloudera.org:8080/#/c/12700/3/docs/topics/impala_metadata.xml@290 PS3, Line 290: automatic metadata sync I think it is more appropriate to say "automatic metadata invalidate" instead http://gerrit.cloudera.org:8080/#/c/12700/3/docs/topics/impala_metadata.xml@431 PS3, Line 431: or not. Possible states could be: STOPPED, ACTIVE, ERROR, NEEDS_INVALIDATE This might change to the following states after we merge IMPALA-7972 in 3.2 PAUSED, // event processor is paused because catalog is being reset concurrently ACTIVE, // event processor is scheduled at a given frequency ERROR, // event processor is in error state and event processing has stopped NEEDS_INVALIDATE, // event processor could not resolve certain events and needs a // manual invalidate command to reset the state STOPPED, // event processing has been shutdown. No events will be processed DISABLED // event processor is not configured to run -- To view, visit http://gerrit.cloudera.org:8080/12700 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6c8a4ec77152c839564b1d69c76dfa041a0ebd7f Gerrit-Change-Number: 12700 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni <[email protected]> Gerrit-Reviewer: Adrian Ng (389) Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Vihang Karajgaonkar <[email protected]> Gerrit-Comment-Date: Mon, 11 Mar 2019 19:16:25 +0000 Gerrit-HasComments: Yes
