[jira] [Comment Edited] (HIVE-20545) Ability to exclude potentially large parameters in HMS Notifications
[ https://issues.apache.org/jira/browse/HIVE-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640211#comment-16640211 ] Bharathkrishna Guruvayoor Murali edited comment on HIVE-20545 at 10/8/18 6:32 PM: -- Hi [~anishek] , An example would be when Impala writes stats information to Partition objects and accesses it, as shown : [Impala stats|https://github.com/apache/impala/blob/d48ffc2d45b2a9d4b9c730bba5677d3096311a25/fe/src/main/java/org/apache/impala/catalog/PartitionStatsUtil.java#L46] But this information is not relevant as a Partition metadata for any other purpose. Also, these parameters have considerably large-size. On running some preliminary tests, we observe that there is a performance hit on increasing message size of HMS Notifications, so we can use this configuration to filter parameters like above mentioned. Edited : Updated the link which was pointing to wrong one previously. was (Author: bharos92): Hi [~anishek] , An example would be when Impala writes stats information to Partition objects and accesses it, as shown : [Impala stats|https://github.com/apache/impala/blob/d48ffc2d45b2a9d4b9c730bba5677d3096311a25/fe/src/main/java/org/apache/impala/catalog/PartitionStatsUtil.java#L46] But this information is not relevant as a Partition metadata for any other purpose. Also, these parameters have considerably large-size. On running some preliminary tests, we observe that there is a performance hit on increasing message size of HMS Notifications, so we can use this configuration to filter parameters like above mentioned. > Ability to exclude potentially large parameters in HMS Notifications > > > Key: HIVE-20545 > URL: https://issues.apache.org/jira/browse/HIVE-20545 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.1.0, 4.0.0 >Reporter: Bharathkrishna Guruvayoor Murali >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Major > Attachments: HIVE-20545.1.patch, HIVE-20545.2.patch, > HIVE-20545.3.branch-3.patch, HIVE-20545.3.patch, HIVE-20545.4.patch, > HIVE-20545.6.patch, HIVE-20545.7.patch > > > Clients can add large-sized parameters in Table/Partition objects. So we need > to enable adding regex patterns through HiveConf to match parameters to be > filtered from table and partition objects before serialization in HMS > notifications. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20545) Ability to exclude potentially large parameters in HMS Notifications
[ https://issues.apache.org/jira/browse/HIVE-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640211#comment-16640211 ] Bharathkrishna Guruvayoor Murali edited comment on HIVE-20545 at 10/8/18 6:19 PM: -- Hi [~anishek] , An example would be when Impala writes stats information to Partition objects and accesses it, as shown : [Impala stats|https://github.com/apache/impala/blob/d48ffc2d45b2a9d4b9c730bba5677d3096311a25/fe/src/main/java/org/apache/impala/catalog/PartitionStatsUtil.java#L46] But this information is not relevant as a Partition metadata for any other purpose. Also, these parameters have considerably large-size. On running some preliminary tests, we observe that there is a performance hit on increasing message size of HMS Notifications, so we can use this configuration to filter parameters like above mentioned. was (Author: bharos92): Hi [~anishek] , An example would be when Impala writes stats information to Partition objects and accesses it, as shown : [Impala stats|http://github.mtv.cloudera.com/CDH/Impala/blob/6f2d928734a33ace15ec6abd5659651173b9e69e/fe/src/main/java/org/apache/impala/catalog/PartitionStatsUtil.java#L45] But this information is not relevant as a Partition metadata for any other purpose. Also, these parameters have considerably large-size. On running some preliminary tests, we observe that there is a performance hit on increasing message size of HMS Notifications, so we can use this configuration to filter parameters like above mentioned. > Ability to exclude potentially large parameters in HMS Notifications > > > Key: HIVE-20545 > URL: https://issues.apache.org/jira/browse/HIVE-20545 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.1.0, 4.0.0 >Reporter: Bharathkrishna Guruvayoor Murali >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Major > Attachments: HIVE-20545.1.patch, HIVE-20545.2.patch, > HIVE-20545.3.branch-3.patch, HIVE-20545.3.patch, HIVE-20545.4.patch, > HIVE-20545.6.patch, HIVE-20545.7.patch > > > Clients can add large-sized parameters in Table/Partition objects. So we need > to enable adding regex patterns through HiveConf to match parameters to be > filtered from table and partition objects before serialization in HMS > notifications. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20545) Ability to exclude potentially large parameters in HMS Notifications
[ https://issues.apache.org/jira/browse/HIVE-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640211#comment-16640211 ] Bharathkrishna Guruvayoor Murali edited comment on HIVE-20545 at 10/5/18 6:51 PM: -- Hi [~anishek] , An example would be when Impala writes stats information to Partition objects and accesses it, as shown : [Impala stats|http://github.mtv.cloudera.com/CDH/Impala/blob/6f2d928734a33ace15ec6abd5659651173b9e69e/fe/src/main/java/org/apache/impala/catalog/PartitionStatsUtil.java#L45] But this information is not relevant as a Partition metadata for any other purpose. Also, these parameters have considerably large-size. On running some preliminary tests, we observe that there is a performance hit on increasing message size of HMS Notifications, so we can use this configuration to filter parameters like above mentioned. was (Author: bharos92): Hi [~anishek] , An example would be when Impala writes stats information to Partition objects and accesses it, as shown : [Impala reading stats |http://github.mtv.cloudera.com/CDH/Impala/blob/6f2d928734a33ace15ec6abd5659651173b9e69e/fe/src/main/java/org/apache/impala/catalog/PartitionStatsUtil.java#L45] But this information is not relevant as a Partition metadata for any other purpose. Also, these parameters have considerably large-size. On running some preliminary tests, we observe that there is a performance hit on increasing message size of HMS Notifications, so we can use this configuration to filter parameters like above mentioned. > Ability to exclude potentially large parameters in HMS Notifications > > > Key: HIVE-20545 > URL: https://issues.apache.org/jira/browse/HIVE-20545 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.1.0, 4.0.0 >Reporter: Bharathkrishna Guruvayoor Murali >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Major > Attachments: HIVE-20545.1.patch, HIVE-20545.2.patch, > HIVE-20545.3.branch-3.patch, HIVE-20545.3.patch, HIVE-20545.4.patch, > HIVE-20545.6.patch, HIVE-20545.7.patch > > > Clients can add large-sized parameters in Table/Partition objects. So we need > to enable adding regex patterns through HiveConf to match parameters to be > filtered from table and partition objects before serialization in HMS > notifications. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
