[jira] [Commented] (IMPALA-13040) SIGSEGV in QueryState::UpdateFilterFromRemote
[
https://issues.apache.org/jira/browse/IMPALA-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845175#comment-17845175
]
ASF subversion and git services commented on IMPALA-13040:
--
Commit 09d2f10f4ddf3499b6255a6d14653e7738c2928b in impala's branch
refs/heads/master from Riza Suminto
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=09d2f10f4 ]
IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote
It is possible to have UpdateFilterFromRemote RPC arrive to an impalad
executor before QueryState of the destination query is created or
complete initialization. This patch add wait mechanism in
UpdateFilterFromRemote RPC endpoint to wait for few miliseconds until
QueryState exist and complete initialization.
The wait time is fixed at 500ms, with exponential sleep period in
between. If wait time passed and QueryState still not found or
initialized, UpdateFilterFromRemote RPC is deemed fail and query
execution move on without complete filter.
Testing:
- Add BE tests in network-util-test.cc
- Add test_runtime_filter_aggregation.py::TestLateQueryStateInit
- Pass exhastive runs of test_runtime_filter_aggregation.py,
test_query_live.py, and test_query_log.py
Change-Id: I156d1f0c694b91ba34be70bc53ae9bacf924b3b9
Reviewed-on: http://gerrit.cloudera.org:8080/21383
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> SIGSEGV in QueryState::UpdateFilterFromRemote
> --
>
> Key: IMPALA-13040
> URL: https://issues.apache.org/jira/browse/IMPALA-13040
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Reporter: Csaba Ringhofer
>Priority: Critical
>
> {code}
> Crash reason: SIGSEGV /SEGV_MAPERR
> Crash address: 0x48
> Process uptime: not available
> Thread 114 (crashed)
> 0 libpthread.so.0 + 0x9d00
> rax = 0x00019e57ad00 rdx = 0x2a656720
> rcx = 0x059a9860 rbx = 0x
> rsi = 0x00019e57ad00 rdi = 0x0038
> rbp = 0x7f6233d544e0 rsp = 0x7f6233d544a8
> r8 = 0x06a53540r9 = 0x0039
> r10 = 0x r11 = 0x000a
> r12 = 0x00019e57ad00 r13 = 0x7f62a2f997d0
> r14 = 0x7f6233d544f8 r15 = 0x1632c0f0
> rip = 0x7f62a2f96d00
> Found by: given as instruction pointer in context
> 1
> impalad!impala::QueryState::UpdateFilterFromRemote(impala::UpdateFilterParamsPB
> const&, kudu::rpc::RpcContext*) [query-state.cc : 1033 + 0x5]
> rbp = 0x7f6233d54520 rsp = 0x7f6233d544f0
> rip = 0x015c0837
> Found by: previous frame's frame pointer
> 2
> impalad!impala::DataStreamService::UpdateFilterFromRemote(impala::UpdateFilterParamsPB
> const*, impala::UpdateFilterResultPB*, kudu::rpc::RpcContext*)
> [data-stream-service.cc : 134 + 0xb]
> rbp = 0x7f6233d54640 rsp = 0x7f6233d54530
> rip = 0x017c05de
> Found by: previous frame's frame pointer
> {code}
> The line that crashes is
> https://github.com/apache/impala/blob/b39cd79ae84c415e0aebec2c2b4d7690d2a0cc7a/be/src/runtime/query-state.cc#L1033
> My guess is that inside the actual segfault is within WaitForPrepare() but it
> was inlined. Not sure if a remote filter can arrive even before
> QueryState::Init is finished - that would explain the issue, as
> instances_prepared_barrier_ is not yet created at that point.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-12688) Support JSON profile imports in webUI
[ https://issues.apache.org/jira/browse/IMPALA-12688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Quanlong Huang updated IMPALA-12688: Fix Version/s: Impala 4.4.0 > Support JSON profile imports in webUI > - > > Key: IMPALA-12688 > URL: https://issues.apache.org/jira/browse/IMPALA-12688 > Project: IMPALA > Issue Type: New Feature >Reporter: Surya Hebbar >Assignee: Surya Hebbar >Priority: Major > Fix For: Impala 4.4.0 > > Attachments: clear_all_button.png, descending_order_start_time.png, > imported_profiles_section.png, imported_queries_button.png, > imported_queries_list.png, imported_queries_page.png, > imported_query_statement.png, imported_query_text_plan.png, > imported_query_timeline.png, multiple_query_profile_import.png > > > It would be helpful for users to visualize the query timeline by selecting a > local JSON query profile. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-10451) TestAvroSchemaResolution.test_avro_schema_resolution fails when bumping Hive to have HIVE-24157
[
https://issues.apache.org/jira/browse/IMPALA-10451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell resolved IMPALA-10451.
Fix Version/s: Impala 4.5.0
Resolution: Fixed
> TestAvroSchemaResolution.test_avro_schema_resolution fails when bumping Hive
> to have HIVE-24157
> ---
>
> Key: IMPALA-10451
> URL: https://issues.apache.org/jira/browse/IMPALA-10451
> Project: IMPALA
> Issue Type: Bug
>Reporter: Quanlong Huang
>Assignee: Joe McDonnell
>Priority: Major
> Fix For: Impala 4.5.0
>
>
> TestAvroSchemaResolution.test_avro_schema_resolution recently fails when
> building against a Hive version with HIVE-24157.
> {code:java}
> query_test.test_avro_schema_resolution.TestAvroSchemaResolution.test_avro_schema_resolution[protocol:
> beeswax | exec_option: \{'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> avro/snap/block] (from pytest)
> query_test/test_avro_schema_resolution.py:36: in test_avro_schema_resolution
> self.run_test_case('QueryTest/avro-schema-resolution', vector,
> unique_database)
> common/impala_test_suite.py:690: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:523: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:456: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:278: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E 10 != 0
> {code}
> The failed query is
> {code:sql}
> select count(*) from functional_avro_snap.avro_coldef {code}
> The cause is that data loading for avro_coldef failed. The DML is
> {code:sql}
> INSERT OVERWRITE TABLE avro_coldef PARTITION(year=2014, month=1)
> SELECT bool_col, tinyint_col, smallint_col, int_col, bigint_col,
> float_col, double_col, date_string_col, string_col, timestamp_col
> FROM (select * from functional.alltypes order by id limit 5) a;
> {code}
> The failure (found in HS2) is:
> {code}
> 2021-01-24T01:52:16,340 ERROR [9433ee64-d706-4fa4-a146-18d71bf17013
> HiveServer2-Handler-Pool: Thread-4946] parse.CalcitePlanner: CBO failed,
> skipping CBO.
> org.apache.hadoop.hive.ql.exec.UDFArgumentException: Casting DATE/TIMESTAMP
> types to NUMERIC is prohibited (hive.strict.timestamp.conversion)
> at
> org.apache.hadoop.hive.ql.udf.TimestampCastRestrictorResolver.getEvalMethod(TimestampCastRestrictorResolver.java:62)
> ~[hive-exec-3.1.3000.7.1.6.0-169.jar:3.1.3000.7.1.6.0-169]
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.initialize(GenericUDFBridge.java:168)
> ~[hive-exec-3.1.3000.7.1.6.0-169.jar:3.1.3000.7.1.6.0-169]
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDF.initializeAndFoldConstants(GenericUDF.java:149)
> ~[hive-exec-3.1.3000.7.1.6.0-169.jar:3.1.3000.7.1.6.0-169]
> at
> org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.newInstance(ExprNodeGenericFuncDesc.java:260)
> ~[hive-exec-3.1.3000.7.1.6.0-169.jar:3.1.3000.7.1.6.0-169]
> at
> org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.newInstance(ExprNodeGenericFuncDesc.java:292)
> ~[hive-exec-3.1.3000.7.1.6.0-169.jar:3.1.3000.7.1.6.0-169]
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getFuncExprNodeDescWithUdfData(TypeCheckProcFactory.java:987)
> ~[hive-exec-3.1.3000.7.1.6.0-169.jar:3.1.3000.7.1.6.0-169]
> at
> org.apache.hadoop.hive.ql.parse.ParseUtils.createConversionCast(ParseUtils.java:163)
> ~[hive-exec-3.1.3000.7.1.6.0-169.jar:3.1.3000.7.1.6.0-169]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genConversionSelectOperator(SemanticAnalyzer.java:8551)
> ~[hive-exec-3.1.3000.7.1.6.0-169.jar:3.1.3000.7.1.6.0-169]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFileSinkPlan(SemanticAnalyzer.java:7908)
> ~[hive-exec-3.1.3000.7.1.6.0-169.jar:3.1.3000.7.1.6.0-169]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:11100)
> ~[hive-exec-3.1.3000.7.1.6.0-169.jar:3.1.3000.7.1.6.0-169]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:10972)
> ~[hive-exec-3.1.3000.7.1.6.0-169.jar:3.1.3000.7.1.6.0-169]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11901)
> ~[hive-exec-3.1.3000.7.1.6.0-169.jar:3.1.3000.7.1.6.0-169]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11771)
> ~[hive-exec-3.1.3000.7.1.6.0-169.jar:3.1.3000.7.1.6.0-169]
> at
>
[jira] [Commented] (IMPALA-10451) TestAvroSchemaResolution.test_avro_schema_resolution fails when bumping Hive to have HIVE-24157
[
https://issues.apache.org/jira/browse/IMPALA-10451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845053#comment-17845053
]
ASF subversion and git services commented on IMPALA-10451:
--
Commit d1d28c0eec52c212b4b8cd09080327c542f24bfa in impala's branch
refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=d1d28c0ee ]
IMPALA-10451: Fix avro table loading failures caused by HIVE-24157
HIVE-24157 introduces a restriction to prohibit casting DATE/TIMESTAMP
types to and from NUMERIC. It's enabled by default and can be turned off
by set hive.strict.timestamp.conversion=false.
This restriction breaks the data loading on avro_coldef and
avro_extra_coldef tables, which results in empty data set and finally
fails TestAvroSchemaResolution.test_avro_schema_resolution.
This patch explicitly disables the restriction in loading these two avro
tables.
The Hive version currently used for development does not have HIVE-24157,
but upstream Hive does have it. Adding hive.strict.timestamp.conversion
does not cause problems for Hive versions that don't have HIVE-24157.
Tests:
- Run the data loading and test_avro_schema_resolution locally using
a Hive that has HIVE-24157.
- Run CORE tests
- Run data loading with a Hive that doesn't have HIVE-24157.
Change-Id: I3e2a47d60d4079fece9c04091258215f3d6a7b52
Reviewed-on: http://gerrit.cloudera.org:8080/21413
Reviewed-by: Impala Public Jenkins
Tested-by: Joe McDonnell
> TestAvroSchemaResolution.test_avro_schema_resolution fails when bumping Hive
> to have HIVE-24157
> ---
>
> Key: IMPALA-10451
> URL: https://issues.apache.org/jira/browse/IMPALA-10451
> Project: IMPALA
> Issue Type: Bug
>Reporter: Quanlong Huang
>Assignee: Joe McDonnell
>Priority: Major
>
> TestAvroSchemaResolution.test_avro_schema_resolution recently fails when
> building against a Hive version with HIVE-24157.
> {code:java}
> query_test.test_avro_schema_resolution.TestAvroSchemaResolution.test_avro_schema_resolution[protocol:
> beeswax | exec_option: \{'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> avro/snap/block] (from pytest)
> query_test/test_avro_schema_resolution.py:36: in test_avro_schema_resolution
> self.run_test_case('QueryTest/avro-schema-resolution', vector,
> unique_database)
> common/impala_test_suite.py:690: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:523: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:456: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:278: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E 10 != 0
> {code}
> The failed query is
> {code:sql}
> select count(*) from functional_avro_snap.avro_coldef {code}
> The cause is that data loading for avro_coldef failed. The DML is
> {code:sql}
> INSERT OVERWRITE TABLE avro_coldef PARTITION(year=2014, month=1)
> SELECT bool_col, tinyint_col, smallint_col, int_col, bigint_col,
> float_col, double_col, date_string_col, string_col, timestamp_col
> FROM (select * from functional.alltypes order by id limit 5) a;
> {code}
> The failure (found in HS2) is:
> {code}
> 2021-01-24T01:52:16,340 ERROR [9433ee64-d706-4fa4-a146-18d71bf17013
> HiveServer2-Handler-Pool: Thread-4946] parse.CalcitePlanner: CBO failed,
> skipping CBO.
> org.apache.hadoop.hive.ql.exec.UDFArgumentException: Casting DATE/TIMESTAMP
> types to NUMERIC is prohibited (hive.strict.timestamp.conversion)
> at
> org.apache.hadoop.hive.ql.udf.TimestampCastRestrictorResolver.getEvalMethod(TimestampCastRestrictorResolver.java:62)
> ~[hive-exec-3.1.3000.7.1.6.0-169.jar:3.1.3000.7.1.6.0-169]
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.initialize(GenericUDFBridge.java:168)
> ~[hive-exec-3.1.3000.7.1.6.0-169.jar:3.1.3000.7.1.6.0-169]
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDF.initializeAndFoldConstants(GenericUDF.java:149)
> ~[hive-exec-3.1.3000.7.1.6.0-169.jar:3.1.3000.7.1.6.0-169]
> at
> org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.newInstance(ExprNodeGenericFuncDesc.java:260)
> ~[hive-exec-3.1.3000.7.1.6.0-169.jar:3.1.3000.7.1.6.0-169]
> at
> org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.newInstance(ExprNodeGenericFuncDesc.java:292)
> ~[hive-exec-3.1.3000.7.1.6.0-169.jar:3.1.3000.7.1.6.0-169]
> at
>
[jira] [Commented] (IMPALA-11499) Refactor UrlEncode function to handle special characters
[
https://issues.apache.org/jira/browse/IMPALA-11499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845052#comment-17845052
]
ASF subversion and git services commented on IMPALA-11499:
--
Commit 85cd07a11e876f3d8773f2638f699c61a6b0dd4c in impala's branch
refs/heads/master from pranavyl
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=85cd07a11 ]
IMPALA-11499: Refactor UrlEncode function to handle special characters
An error came from an issue with URL encoding, where certain Unicode
characters were being incorrectly encoded due to their UTF-8
representation matching characters in the set of characters to escape.
For example, the string '运', which consists of three bytes
0xe8 0xbf 0x90 was wrongly getting encoded into '\E8%FFBF\90',
because the middle byte matched one of the two bytes that
represented the "\u00FF" literal. Inclusion of "\u00FF" was likely
a mistake from the beginning and it should have been '\x7F'.
The patch makes three key changes:
1. Before the change, the set of characters that need to be escaped
was stored as a string. The current patch uses an unordered_set
instead.
2. '\xFF', which is an invalid UTF-8 byte and whose inclusion was
erroneous from the beginning, is replaced with '\x7F', which is a
control character for DELETE, ensuring consistency and correctness in
URL encoding.
3. The list of characters to be escaped is extended to match the
current list in Hive.
Testing: Tests on both traditional Hive tables and Iceberg tables
are included in unicode-column-name.test, insert.test,
coding-util-test.cc and test_insert.py.
Change-Id: I88c4aba5d811dfcec809583d0c16fcbc0ca730fb
Reviewed-on: http://gerrit.cloudera.org:8080/21131
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> Refactor UrlEncode function to handle special characters
>
>
> Key: IMPALA-11499
> URL: https://issues.apache.org/jira/browse/IMPALA-11499
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Reporter: Quanlong Huang
>Assignee: Pranav Yogi Lodha
>Priority: Critical
>
> Partition values are incorrectly URL-encoded in backend for unicode
> characters, e.g. '运营业务数据' is encoded to '�%FFBF�营业务数据' which is wrong.
> To reproduce the issue, first create a partition table:
> {code:sql}
> create table my_part_tbl (id int) partitioned by (p string) stored as parquet;
> {code}
> Then insert data into it using partition values containing '运'. They will
> fail:
> {noformat}
> [localhost:21050] default> insert into my_part_tbl partition(p='运营业务数据')
> values (0);
> Query: insert into my_part_tbl partition(p='运营业务数据') values (0)
> Query submitted at: 2022-08-16 10:03:56 (Coordinator:
> http://quanlong-OptiPlex-BJ:25000)
> Query progress can be monitored at:
> http://quanlong-OptiPlex-BJ:25000/query_plan?query_id=404ac3027c4b7169:39d16a2d
> ERROR: Error(s) moving partition files. First error (of 1) was: Hdfs op
> (RENAME
> hdfs://localhost:20500/test-warehouse/my_part_tbl/_impala_insert_staging/404ac3027c4b7169_39d16a2d/.404ac3027c4b7169-39d16a2d_1475855322_dir/p=�%FFBF�营业务数据/404ac3027c4b7169-39d16a2d_1585092794_data.0.parq
> TO
> hdfs://localhost:20500/test-warehouse/my_part_tbl/p=�%FFBF�营业务数据/404ac3027c4b7169-39d16a2d_1585092794_data.0.parq)
> failed, error was:
> hdfs://localhost:20500/test-warehouse/my_part_tbl/_impala_insert_staging/404ac3027c4b7169_39d16a2d/.404ac3027c4b7169-39d16a2d_1475855322_dir/p=�%FFBF�营业务数据/404ac3027c4b7169-39d16a2d_1585092794_data.0.parq
> Error(5): Input/output error
> [localhost:21050] default> insert into my_part_tbl partition(p='运') values
> (0);
> Query: insert into my_part_tbl partition(p='运') values (0)
> Query submitted at: 2022-08-16 10:04:22 (Coordinator:
> http://quanlong-OptiPlex-BJ:25000)
> Query progress can be monitored at:
> http://quanlong-OptiPlex-BJ:25000/query_plan?query_id=a64e5883473ec28d:86e7e335
> ERROR: Error(s) moving partition files. First error (of 1) was: Hdfs op
> (RENAME
> hdfs://localhost:20500/test-warehouse/my_part_tbl/_impala_insert_staging/a64e5883473ec28d_86e7e335/.a64e5883473ec28d-86e7e335_1582623091_dir/p=�%FFBF�/a64e5883473ec28d-86e7e335_163454510_data.0.parq
> TO
> hdfs://localhost:20500/test-warehouse/my_part_tbl/p=�%FFBF�/a64e5883473ec28d-86e7e335_163454510_data.0.parq)
> failed, error was:
> hdfs://localhost:20500/test-warehouse/my_part_tbl/_impala_insert_staging/a64e5883473ec28d_86e7e335/.a64e5883473ec28d-86e7e335_1582623091_dir/p=�%FFBF�/a64e5883473ec28d-86e7e335_163454510_data.0.parq
> Error(5): Input/output error
> {noformat}
> However, partition value without the character '运' is OK:
> {noformat}
[jira] [Assigned] (IMPALA-10947) SQL support for querying Iceberg metadata
[ https://issues.apache.org/jira/browse/IMPALA-10947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boglarka Egyed reassigned IMPALA-10947: --- Assignee: Daniel Becker (was: Tamas Mate) > SQL support for querying Iceberg metadata > - > > Key: IMPALA-10947 > URL: https://issues.apache.org/jira/browse/IMPALA-10947 > Project: IMPALA > Issue Type: Epic > Components: Frontend >Reporter: Zoltán Borók-Nagy >Assignee: Daniel Becker >Priority: Major > Labels: impala-iceberg > > HIVE-25457 added support for querying Iceberg table metadata to Hive. > They support the following syntax: > SELECT * FROM default.iceberg_table.history; > Spark uses the same syntax: https://iceberg.apache.org/spark-queries/#history > Other than "history", the following metadata tables are available in Iceberg: > The following metadata tables are available in Iceberg: > * ENTRIES, > * FILES, > * HISTORY, > * SNAPSHOTS, > * MANIFESTS, > * PARTITIONS, > * ALL_DATA_FILES, > * ALL_MANIFESTS, > * ALL_ENTRIES > Impala currently only supports "DESCRIBE HISTORY ". The above SELECT > syntax would be more convenient for the users, also it would be more flexible > as users could easily define filters in WHERE clauses. And of course we would > be consistent with other engines. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-10451) TestAvroSchemaResolution.test_avro_schema_resolution fails when bumping Hive to have HIVE-24157
[
https://issues.apache.org/jira/browse/IMPALA-10451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanlong Huang reassigned IMPALA-10451:
---
Assignee: Joe McDonnell (was: Quanlong Huang)
> TestAvroSchemaResolution.test_avro_schema_resolution fails when bumping Hive
> to have HIVE-24157
> ---
>
> Key: IMPALA-10451
> URL: https://issues.apache.org/jira/browse/IMPALA-10451
> Project: IMPALA
> Issue Type: Bug
>Reporter: Quanlong Huang
>Assignee: Joe McDonnell
>Priority: Major
>
> TestAvroSchemaResolution.test_avro_schema_resolution recently fails when
> building against a Hive version with HIVE-24157.
> {code:java}
> query_test.test_avro_schema_resolution.TestAvroSchemaResolution.test_avro_schema_resolution[protocol:
> beeswax | exec_option: \{'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> avro/snap/block] (from pytest)
> query_test/test_avro_schema_resolution.py:36: in test_avro_schema_resolution
> self.run_test_case('QueryTest/avro-schema-resolution', vector,
> unique_database)
> common/impala_test_suite.py:690: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:523: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:456: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:278: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E 10 != 0
> {code}
> The failed query is
> {code:sql}
> select count(*) from functional_avro_snap.avro_coldef {code}
> The cause is that data loading for avro_coldef failed. The DML is
> {code:sql}
> INSERT OVERWRITE TABLE avro_coldef PARTITION(year=2014, month=1)
> SELECT bool_col, tinyint_col, smallint_col, int_col, bigint_col,
> float_col, double_col, date_string_col, string_col, timestamp_col
> FROM (select * from functional.alltypes order by id limit 5) a;
> {code}
> The failure (found in HS2) is:
> {code}
> 2021-01-24T01:52:16,340 ERROR [9433ee64-d706-4fa4-a146-18d71bf17013
> HiveServer2-Handler-Pool: Thread-4946] parse.CalcitePlanner: CBO failed,
> skipping CBO.
> org.apache.hadoop.hive.ql.exec.UDFArgumentException: Casting DATE/TIMESTAMP
> types to NUMERIC is prohibited (hive.strict.timestamp.conversion)
> at
> org.apache.hadoop.hive.ql.udf.TimestampCastRestrictorResolver.getEvalMethod(TimestampCastRestrictorResolver.java:62)
> ~[hive-exec-3.1.3000.7.1.6.0-169.jar:3.1.3000.7.1.6.0-169]
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.initialize(GenericUDFBridge.java:168)
> ~[hive-exec-3.1.3000.7.1.6.0-169.jar:3.1.3000.7.1.6.0-169]
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDF.initializeAndFoldConstants(GenericUDF.java:149)
> ~[hive-exec-3.1.3000.7.1.6.0-169.jar:3.1.3000.7.1.6.0-169]
> at
> org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.newInstance(ExprNodeGenericFuncDesc.java:260)
> ~[hive-exec-3.1.3000.7.1.6.0-169.jar:3.1.3000.7.1.6.0-169]
> at
> org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.newInstance(ExprNodeGenericFuncDesc.java:292)
> ~[hive-exec-3.1.3000.7.1.6.0-169.jar:3.1.3000.7.1.6.0-169]
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getFuncExprNodeDescWithUdfData(TypeCheckProcFactory.java:987)
> ~[hive-exec-3.1.3000.7.1.6.0-169.jar:3.1.3000.7.1.6.0-169]
> at
> org.apache.hadoop.hive.ql.parse.ParseUtils.createConversionCast(ParseUtils.java:163)
> ~[hive-exec-3.1.3000.7.1.6.0-169.jar:3.1.3000.7.1.6.0-169]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genConversionSelectOperator(SemanticAnalyzer.java:8551)
> ~[hive-exec-3.1.3000.7.1.6.0-169.jar:3.1.3000.7.1.6.0-169]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFileSinkPlan(SemanticAnalyzer.java:7908)
> ~[hive-exec-3.1.3000.7.1.6.0-169.jar:3.1.3000.7.1.6.0-169]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:11100)
> ~[hive-exec-3.1.3000.7.1.6.0-169.jar:3.1.3000.7.1.6.0-169]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:10972)
> ~[hive-exec-3.1.3000.7.1.6.0-169.jar:3.1.3000.7.1.6.0-169]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11901)
> ~[hive-exec-3.1.3000.7.1.6.0-169.jar:3.1.3000.7.1.6.0-169]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11771)
> ~[hive-exec-3.1.3000.7.1.6.0-169.jar:3.1.3000.7.1.6.0-169]
> at
>
[jira] [Created] (IMPALA-13067) Some regex make the tests unconditionally pass
Gabor Kaszab created IMPALA-13067: - Summary: Some regex make the tests unconditionally pass Key: IMPALA-13067 URL: https://issues.apache.org/jira/browse/IMPALA-13067 Project: IMPALA Issue Type: Bug Components: Infrastructure Reporter: Gabor Kaszab This issue came out in the Iceberg metadata table tests where this regex was used: [1-9]\d*|0 The "|0" part for some reason made the test framework confused and then regardless of what you provide as an expected result the tests passed. One workaround was to put the regex expression between parentheses. Or simply use "d+". https://issues.apache.org/jira/browse/IMPALA-13055 applied this second workaround on the tests. Some analysis would be great why this is the behavior of the test framework, and if it's indeed the issue of the framnework, we should fix it. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-13066) SHOW CREATE TABLE with stats and partitions
Quanlong Huang created IMPALA-13066:
---
Summary: SHOW CREATE TABLE with stats and partitions
Key: IMPALA-13066
URL: https://issues.apache.org/jira/browse/IMPALA-13066
Project: IMPALA
Issue Type: New Feature
Components: Backend, Frontend
Reporter: Quanlong Huang
SHOW CREATE TABLE produces the statement to create the table. In practise, we
also want the column stats and partitions. It'd be helpful to add an option for
also producing the ADD PARTITION and SET COLUMN STATS statements. E.g.
{code:sql}
SHOW CREATE TABLE my_tbl WITH STATS;{code}
produces
{code:sql}
CREATE TABLE my_tbl ...;
ALTER TABLE my_tbl ADD PARTITION ...;
ALTER TABLE my_tbl PARTITION (...) SET TBLPROPERTIES('numRows'='3',
'STATS_GENERATED_VIA_STATS_TASK'='true');
ALTER TABLE my_tbl SET COLUMN STATS c1
('numDVs'='19','numNulls'='0','maxSize'='8','avgSize'='8');
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
