[jira] [Created] (IMPALA-13069) Consider not throwing errors during deleting staging files after insertion
Yida Wu created IMPALA-13069: Summary: Consider not throwing errors during deleting staging files after insertion Key: IMPALA-13069 URL: https://issues.apache.org/jira/browse/IMPALA-13069 Project: IMPALA Issue Type: Improvement Components: Backend Reporter: Yida Wu When inserting data, we often use staging files in the cloud storage as temporary files. Once the insertion is done, we typically clear out these staging files. Currently, our approach involves deleting these files individually and raising errors if any deletion fails. However, at this stage, the insertion process is already complete, and there might be occasional instability with the cloud provider. It may be not optimal to fail a query for such errors during clearing temporary files. Perhaps worth considering a softer way to handle this case. https://github.com/apache/impala/blob/7ad94006563b88d9221b4ac978dbf5b4fc0a3ca1/be/src/runtime/dml-exec-state.cc#L343 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-12792) Wrong indentation in DataSourceScanNode.getNodeExplainString
[ https://issues.apache.org/jira/browse/IMPALA-12792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Riza Suminto resolved IMPALA-12792. --- Fix Version/s: Impala 4.5.0 Target Version: Impala 4.5.0 (was: Impala 4.4.0) Resolution: Fixed This is already resolved along with IMPALA-13018 > Wrong indentation in DataSourceScanNode.getNodeExplainString > > > Key: IMPALA-12792 > URL: https://issues.apache.org/jira/browse/IMPALA-12792 > Project: IMPALA > Issue Type: Bug > Components: Frontend >Reporter: Riza Suminto >Priority: Minor > Labels: ramp-up > Fix For: Impala 4.5.0 > > > DataSourceScanNode.getNodeExplainString use prefix instead of detailPrefix > that has two more spaces. > [https://github.com/apache/impala/blob/0d0a410cf65951d634f81ec14b474d663f9cf587/fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java#L372-L385] > > That makes the explain plan not indented properly > [https://github.com/apache/impala/blob/0d0a410cf65951d634f81ec14b474d663f9cf587/testdata/workloads/functional-planner/queries/PlannerTest/data-source-tables.test#L11-L12] > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Reopened] (IMPALA-11858) admissiond incorrectly caps memory limit to its process memory
[ https://issues.apache.org/jira/browse/IMPALA-11858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Quanlong Huang reopened IMPALA-11858: - > admissiond incorrectly caps memory limit to its process memory > -- > > Key: IMPALA-11858 > URL: https://issues.apache.org/jira/browse/IMPALA-11858 > Project: IMPALA > Issue Type: Bug >Reporter: Abhishek Rawat >Assignee: Abhishek Rawat >Priority: Critical > > When admission controller is running as a separate daemon it incorrectly caps > memory limit for the query to its process limit. This is also incorrect > behavior when admission controller is running in coordinator as executors > could have different memory limit compared to coordinator. > https://github.com/apache/impala/blob/master/be/src/scheduling/schedule-state.cc#L312#L313 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-11858) admissiond incorrectly caps memory limit to its process memory
[ https://issues.apache.org/jira/browse/IMPALA-11858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Quanlong Huang resolved IMPALA-11858. - Fix Version/s: Impala 4.3.0 Resolution: Fixed > admissiond incorrectly caps memory limit to its process memory > -- > > Key: IMPALA-11858 > URL: https://issues.apache.org/jira/browse/IMPALA-11858 > Project: IMPALA > Issue Type: Bug >Reporter: Abhishek Rawat >Assignee: Abhishek Rawat >Priority: Critical > Fix For: Impala 4.3.0 > > > When admission controller is running as a separate daemon it incorrectly caps > memory limit for the query to its process limit. This is also incorrect > behavior when admission controller is running in coordinator as executors > could have different memory limit compared to coordinator. > https://github.com/apache/impala/blob/master/be/src/scheduling/schedule-state.cc#L312#L313 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-13068) Add tests for integration with dbt
Joe McDonnell created IMPALA-13068: -- Summary: Add tests for integration with dbt Key: IMPALA-13068 URL: https://issues.apache.org/jira/browse/IMPALA-13068 Project: IMPALA Issue Type: Improvement Components: Infrastructure Affects Versions: Impala 4.5.0 Reporter: Joe McDonnell Some Impala users rely on dbt and dbt's integration with Impala for their workloads. It would be useful to have some basic tests / scripts for running dbt against Impala. This provides a smoke test for functionality. It also makes it easier for developers to debug dbt issues locally, as the development environment would already have dbt set up. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-13038) Support profile tab for imported query profiles
[ https://issues.apache.org/jira/browse/IMPALA-13038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845519#comment-17845519 ] ASF subversion and git services commented on IMPALA-13038: -- Commit 0d215da8d4e3f93ad3c1cd72aa801fbcb9464fb0 in impala's branch refs/heads/master from Surya Hebbar [ https://gitbox.apache.org/repos/asf?p=impala.git;h=0d215da8d ] IMPALA-13038: Support profile tab for imported query profiles For query profile imports currently the following tabs are supported. - Query Statement - Query Timeline - Query Text Plan With the current patch "Query Profile" tab will also be supported. In the "QueryProfileHandler", "query_id" is now added before verifying its existence in the query log as in "QuerySummaryHandler" and others. "getQueryID" function has been added to "util.js", as it is helpful across multiple query pages for retrieving the query ID into JS scripts, before the page loads up. On loading the imported "Query Profile" page, query profile download section and server's non-existing query ID alerts are removed. All unsupported navbar tabs are removed and current tab is set to active. The query profile is retrieved from the indexedDB's "imported_queries" database. Then query profile is passed onto "profileToString" function, which converts the profile into indented text for displaying on the profile page. Each profile and its child profiles are printed in the following order with the right indentation(fields are skipped, if they do not exist). Profile name: - Info strings: - Event sequences: - Offset: - Events: - Child profile(recursive): - Counters: Change-Id: Iddcf2e285abbf42f97bde19014be076ccd6374bc Reviewed-on: http://gerrit.cloudera.org:8080/21400 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Support profile tab for imported query profiles > --- > > Key: IMPALA-13038 > URL: https://issues.apache.org/jira/browse/IMPALA-13038 > Project: IMPALA > Issue Type: New Feature >Reporter: Surya Hebbar >Assignee: Surya Hebbar >Priority: Major > Attachments: json_profile_a34485359bfdfe1f_3ca8177b.json, > json_profile_a34485359bfdfe1f_3ca8177b.txt > > > Query profile imports currently support the following tabs. > - Query Statement > - Query Timeline > - Query Text Plan > It would be helpful to support "Query Profile" tab for these imports. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-11328) Mistmatch on max_errors documentation
[ https://issues.apache.org/jira/browse/IMPALA-11328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjana Malhotra resolved IMPALA-11328. --- Resolution: Fixed > Mistmatch on max_errors documentation > - > > Key: IMPALA-11328 > URL: https://issues.apache.org/jira/browse/IMPALA-11328 > Project: IMPALA > Issue Type: Documentation > Components: Docs >Affects Versions: Impala 4.0.0 >Reporter: Riza Suminto >Assignee: Sanjana Malhotra >Priority: Minor > > The doc mention that max_errors default to 1000. > [https://github.com/apache/impala/blob/62683e0ebb78902e142975971c93b8fa011fb632/docs/topics/impala_max_errors.xml#L55] > > But the code actually default to 100. > [https://github.com/apache/impala/blob/62683e0ebb78902e142975971c93b8fa011fb632/be/src/runtime/query-state.cc#L125] > > [https://github.com/apache/impala/blob/62683e0ebb78902e142975971c93b8fa011fb632/common/thrift/Query.thrift#L134] > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-13018) Fix test_tpcds_queries.py/TestTpcdsQueryForJdbcTables.test_tpcds-decimal_v2-q80a failure
[ https://issues.apache.org/jira/browse/IMPALA-13018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenzhe Zhou resolved IMPALA-13018. -- Fix Version/s: Impala 4.5.0 Resolution: Fixed > Fix > test_tpcds_queries.py/TestTpcdsQueryForJdbcTables.test_tpcds-decimal_v2-q80a > failure > > > Key: IMPALA-13018 > URL: https://issues.apache.org/jira/browse/IMPALA-13018 > Project: IMPALA > Issue Type: Sub-task > Components: Backend, Frontend >Reporter: Wenzhe Zhou >Assignee: Wenzhe Zhou >Priority: Major > Fix For: Impala 4.5.0 > > > The returned rows are not matching expected results for some decimal type of > columns. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-13036) Document Iceberg metadata tables
[ https://issues.apache.org/jira/browse/IMPALA-13036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845335#comment-17845335 ] ASF subversion and git services commented on IMPALA-13036: -- Commit aba27edc3338765a6b5133be095989f83cce4747 in impala's branch refs/heads/master from Daniel Becker [ https://gitbox.apache.org/repos/asf?p=impala.git;h=aba27edc3 ] IMPALA-13036: Document Iceberg metadata tables This change adds documentation on how Iceberg metadata tables can be used. Testing: - built docs locally Change-Id: Ic453f567b814cb4363a155e2008029e94efb6ed1 Reviewed-on: http://gerrit.cloudera.org:8080/21387 Tested-by: Impala Public Jenkins Reviewed-by: Peter Rozsa > Document Iceberg metadata tables > > > Key: IMPALA-13036 > URL: https://issues.apache.org/jira/browse/IMPALA-13036 > Project: IMPALA > Issue Type: Documentation >Reporter: Daniel Becker >Assignee: Daniel Becker >Priority: Major > Labels: impala-iceberg > > Impala now supports displaying Iceberg metadata tables, we should document > this feature. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-13036) Document Iceberg metadata tables
[ https://issues.apache.org/jira/browse/IMPALA-13036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Becker resolved IMPALA-13036. Resolution: Fixed > Document Iceberg metadata tables > > > Key: IMPALA-13036 > URL: https://issues.apache.org/jira/browse/IMPALA-13036 > Project: IMPALA > Issue Type: Documentation >Reporter: Daniel Becker >Assignee: Daniel Becker >Priority: Major > Labels: impala-iceberg > > Impala now supports displaying Iceberg metadata tables, we should document > this feature. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-11328) Mistmatch on max_errors documentation
[ https://issues.apache.org/jira/browse/IMPALA-11328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845319#comment-17845319 ] ASF subversion and git services commented on IMPALA-11328: -- Commit aac7f527da1953fcc304bda9e7e5214585fdbf18 in impala's branch refs/heads/master from m-sanjana19 [ https://gitbox.apache.org/repos/asf?p=impala.git;h=aac7f527d ] IMPALA-11328: [DOCS] Fix incorrect default value for max_errors Change-Id: I442cd3ff51520c12376a13d7c78565542793d908 Reviewed-on: http://gerrit.cloudera.org:8080/21419 Reviewed-by: Quanlong Huang Tested-by: Impala Public Jenkins > Mistmatch on max_errors documentation > - > > Key: IMPALA-11328 > URL: https://issues.apache.org/jira/browse/IMPALA-11328 > Project: IMPALA > Issue Type: Documentation > Components: Docs >Affects Versions: Impala 4.0.0 >Reporter: Riza Suminto >Assignee: Sanjana Malhotra >Priority: Minor > > The doc mention that max_errors default to 1000. > [https://github.com/apache/impala/blob/62683e0ebb78902e142975971c93b8fa011fb632/docs/topics/impala_max_errors.xml#L55] > > But the code actually default to 100. > [https://github.com/apache/impala/blob/62683e0ebb78902e142975971c93b8fa011fb632/be/src/runtime/query-state.cc#L125] > > [https://github.com/apache/impala/blob/62683e0ebb78902e142975971c93b8fa011fb632/common/thrift/Query.thrift#L134] > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-11328) Mistmatch on max_errors documentation
[ https://issues.apache.org/jira/browse/IMPALA-11328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjana Malhotra reassigned IMPALA-11328: - Assignee: Sanjana Malhotra > Mistmatch on max_errors documentation > - > > Key: IMPALA-11328 > URL: https://issues.apache.org/jira/browse/IMPALA-11328 > Project: IMPALA > Issue Type: Documentation > Components: Docs >Affects Versions: Impala 4.0.0 >Reporter: Riza Suminto >Assignee: Sanjana Malhotra >Priority: Minor > > The doc mention that max_errors default to 1000. > [https://github.com/apache/impala/blob/62683e0ebb78902e142975971c93b8fa011fb632/docs/topics/impala_max_errors.xml#L55] > > But the code actually default to 100. > [https://github.com/apache/impala/blob/62683e0ebb78902e142975971c93b8fa011fb632/be/src/runtime/query-state.cc#L125] > > [https://github.com/apache/impala/blob/62683e0ebb78902e142975971c93b8fa011fb632/common/thrift/Query.thrift#L134] > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-11499) Refactor UrlEncode function to handle special characters
[
https://issues.apache.org/jira/browse/IMPALA-11499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanlong Huang resolved IMPALA-11499.
-
Fix Version/s: Impala 4.5.0
Resolution: Fixed
Resolving this. Thank [~pranav.lodha] !
> Refactor UrlEncode function to handle special characters
>
>
> Key: IMPALA-11499
> URL: https://issues.apache.org/jira/browse/IMPALA-11499
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Reporter: Quanlong Huang
>Assignee: Pranav Yogi Lodha
>Priority: Critical
> Fix For: Impala 4.5.0
>
>
> Partition values are incorrectly URL-encoded in backend for unicode
> characters, e.g. '运营业务数据' is encoded to '�%FFBF�营业务数据' which is wrong.
> To reproduce the issue, first create a partition table:
> {code:sql}
> create table my_part_tbl (id int) partitioned by (p string) stored as parquet;
> {code}
> Then insert data into it using partition values containing '运'. They will
> fail:
> {noformat}
> [localhost:21050] default> insert into my_part_tbl partition(p='运营业务数据')
> values (0);
> Query: insert into my_part_tbl partition(p='运营业务数据') values (0)
> Query submitted at: 2022-08-16 10:03:56 (Coordinator:
> http://quanlong-OptiPlex-BJ:25000)
> Query progress can be monitored at:
> http://quanlong-OptiPlex-BJ:25000/query_plan?query_id=404ac3027c4b7169:39d16a2d
> ERROR: Error(s) moving partition files. First error (of 1) was: Hdfs op
> (RENAME
> hdfs://localhost:20500/test-warehouse/my_part_tbl/_impala_insert_staging/404ac3027c4b7169_39d16a2d/.404ac3027c4b7169-39d16a2d_1475855322_dir/p=�%FFBF�营业务数据/404ac3027c4b7169-39d16a2d_1585092794_data.0.parq
> TO
> hdfs://localhost:20500/test-warehouse/my_part_tbl/p=�%FFBF�营业务数据/404ac3027c4b7169-39d16a2d_1585092794_data.0.parq)
> failed, error was:
> hdfs://localhost:20500/test-warehouse/my_part_tbl/_impala_insert_staging/404ac3027c4b7169_39d16a2d/.404ac3027c4b7169-39d16a2d_1475855322_dir/p=�%FFBF�营业务数据/404ac3027c4b7169-39d16a2d_1585092794_data.0.parq
> Error(5): Input/output error
> [localhost:21050] default> insert into my_part_tbl partition(p='运') values
> (0);
> Query: insert into my_part_tbl partition(p='运') values (0)
> Query submitted at: 2022-08-16 10:04:22 (Coordinator:
> http://quanlong-OptiPlex-BJ:25000)
> Query progress can be monitored at:
> http://quanlong-OptiPlex-BJ:25000/query_plan?query_id=a64e5883473ec28d:86e7e335
> ERROR: Error(s) moving partition files. First error (of 1) was: Hdfs op
> (RENAME
> hdfs://localhost:20500/test-warehouse/my_part_tbl/_impala_insert_staging/a64e5883473ec28d_86e7e335/.a64e5883473ec28d-86e7e335_1582623091_dir/p=�%FFBF�/a64e5883473ec28d-86e7e335_163454510_data.0.parq
> TO
> hdfs://localhost:20500/test-warehouse/my_part_tbl/p=�%FFBF�/a64e5883473ec28d-86e7e335_163454510_data.0.parq)
> failed, error was:
> hdfs://localhost:20500/test-warehouse/my_part_tbl/_impala_insert_staging/a64e5883473ec28d_86e7e335/.a64e5883473ec28d-86e7e335_1582623091_dir/p=�%FFBF�/a64e5883473ec28d-86e7e335_163454510_data.0.parq
> Error(5): Input/output error
> {noformat}
> However, partition value without the character '运' is OK:
> {noformat}
> [localhost:21050] default> insert into my_part_tbl partition(p='营业务数据')
> values (0);
> Query: insert into my_part_tbl partition(p='营业务数据') values (0)
> Query submitted at: 2022-08-16 10:04:13 (Coordinator:
> http://quanlong-OptiPlex-BJ:25000)
> Query progress can be monitored at:
> http://quanlong-OptiPlex-BJ:25000/query_plan?query_id=b04894bfcfc3836a:b1ac9036
> Modified 1 row(s) in 0.21s
> {noformat}
> Hive is able to execute all these statements.
> I'm able to narrow down the issue into Backend, where we URL-encode the
> partition value in HdfsTableSink::InitOutputPartition():
> {code:cpp}
> string value_str;
> partition_key_expr_evals_[j]->PrintValue(value, &value_str);
> // Directory names containing partition-key values need to be
> UrlEncoded, in
> // particular to avoid problems when '/' is part of the key value
> (which might
> // occur, for example, with date strings). Hive will URL decode the
> value
> // transparently when Impala's frontend asks the metastore for
> partition key values,
> // which makes it particularly important that we use the same encoding
> as Hive. It's
> // also not necessary to encode the values when writing partition
> metadata. You can
> // check this with 'show partitions ' in Hive, followed by a
> select from a
> // decoded partition key value.
> string encoded_str;
> UrlEncode(value_str, &encoded_str, true);
> string part_key_value = (encoded_str.empty()
[jira] [Commented] (IMPALA-13018) Fix test_tpcds_queries.py/TestTpcdsQueryForJdbcTables.test_tpcds-decimal_v2-q80a failure
[
https://issues.apache.org/jira/browse/IMPALA-13018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845222#comment-17845222
]
ASF subversion and git services commented on IMPALA-13018:
--
Commit 3cbb3be5f72dbb889744675fa109dbd1659a7a84 in impala's branch
refs/heads/master from wzhou-code
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=3cbb3be5f ]
IMPALA-13018: Block push down of conjuncts with implicit casting on base
columns for jdbc tables
The query of q80a consists BETWEEN with casting to timestamp in where
clause like:
d_date between cast('2000-08-23' as timestamp)
and (cast('2000-08-23' as timestamp) + interval 30 days)
Between predicate does cast all exprs to compatible types. Planner
generates predicates for DataSourceScanNode as:
CAST(d_date AS TIMESTAMP) >= TIMESTAMP '2000-08-23 00:00:00',
CAST(d_date AS TIMESTAMP) <= TIMESTAMP '2000-09-22 00:00:00'
But casting to Date/Timestamp for a column cannot be pushed down to JDBC
table now. This patch fixes the issue by blocking such conjuncts with
implicit unsafe casting or casting to date/timestamp to be added into
offered predicate list for JDBC table.
Note that explicit casting on base columns are not allowed to
pushdown.
Testing:
- Add new planner unit-tests, including explicit casting, implicit
casting to date/timestamp, built-in functions, arithmetic
expressions.
The predicates which are accepted for JDBC are shown in plan under
"data source predicates" of DataSourceScanNode, predicates which
are not accepted for JDBC are shown in plan under "predicates" of
DataSourceScanNodes.
- Passed all tpcds queries for JDBC tables, including q80a.
- Passed core test
Change-Id: Iabd7e28b8d5f11f25a000dc4c9ab65895056b572
Reviewed-on: http://gerrit.cloudera.org:8080/21409
Reviewed-by: Riza Suminto
Tested-by: Impala Public Jenkins
> Fix
> test_tpcds_queries.py/TestTpcdsQueryForJdbcTables.test_tpcds-decimal_v2-q80a
> failure
>
>
> Key: IMPALA-13018
> URL: https://issues.apache.org/jira/browse/IMPALA-13018
> Project: IMPALA
> Issue Type: Sub-task
> Components: Backend, Frontend
>Reporter: Wenzhe Zhou
>Assignee: Wenzhe Zhou
>Priority: Major
>
> The returned rows are not matching expected results for some decimal type of
> columns.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12607) Bump GBN to get HMS thift API change HIVE-27499
[ https://issues.apache.org/jira/browse/IMPALA-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845221#comment-17845221 ] ASF subversion and git services commented on IMPALA-12607: -- Commit 68f8a6a1df0d2da91baa87b8b6699ddbc495b88e in impala's branch refs/heads/master from Sai Hemanth Gantasala [ https://gitbox.apache.org/repos/asf?p=impala.git;h=68f8a6a1d ] IMPALA-12607: Bump the GBN and fetch events specific to the db/table from the metastore Bump the GBN to 49623641 to leverage HIVE-27499, so that Impala can directly fetch the latest events specific to the db/table from the metastore, instead of fetching the events from metastore and then filtering in the cache matching the DbName/TableName. Implementation Details: Currently when a DDL/DML is performed in Impala, we fetch all the events from metastore based on current eventId and then filter them in Impala which can be a bottleneck if the events count is huge. This can be optimized by including db name and/or table name in the notification event request object and then filter by event type in impala. This can provide performance boost on tables that generate a lot of events. Note: Also included ShowUtils class in hive-minimal-exec jar as it is required in the current build version Testing: 1) Did some tests in local cluster 2) Added a test case in MetaStoreEventsProcessorTest Change-Id: I6aecd5108b31c24e6e2c6f9fba6d4d44a3b00729 Reviewed-on: http://gerrit.cloudera.org:8080/20979 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Bump GBN to get HMS thift API change HIVE-27499 > --- > > Key: IMPALA-12607 > URL: https://issues.apache.org/jira/browse/IMPALA-12607 > Project: IMPALA > Issue Type: Improvement > Components: Catalog >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > Labels: catalog-2024 > > Leverage HIVE-27499, so that Impala can directly fetch the latest events > specific to the database/table from the metastore, instead of fetching the > events from metastore and then filtering in the cache matching the > DbName/TableName. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
