[jira] [Created] (IMPALA-10909) rpc-mgr-kerberized-test sometimes fails with port conflict
Joe McDonnell created IMPALA-10909:
--
Summary: rpc-mgr-kerberized-test sometimes fails with port conflict
Key: IMPALA-10909
URL: https://issues.apache.org/jira/browse/IMPALA-10909
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 4.1.0
Reporter: Joe McDonnell
On a recent release exhaustive run, rpc-mgr-kerberized-test failed with a port
conflict:
{noformat}
73/128 Test #73: rpc-mgr-kerberized-test ..Child aborted***Exception:
2.63 sec
Turning perftools heap leak checking off
21/08/30 17:22:36 INFO util.JvmPauseMonitor: Starting JVM pause monitor
Loading random data
Initializing database '085f-5760-5620-ac7c/krb5kdc/principal' for realm
'KRBTEST.COM',
master key name 'K/m...@krbtest.com'
otp: Loaded
Aug 30 17:22:36 a.b.domain.com krb5kdc[20983](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
krb5kdc: Address already in use - Cannot bind server socket on ::.3
Aug 30 17:22:36 a.b.domain.com krb5kdc[20983](Error): Failed setting up a UDP
socket (for ::.3)
krb5kdc: Address already in use - Error setting up network
Wrote minidump to
Impala/logs/be_tests/minidumps/rpc-mgr-kerberized-test/9764fa11-79d6-41a7-9f0d08b7-044f392b.dmp
Wrote minidump to
Impala/logs/be_tests/minidumps/rpc-mgr-kerberized-test/9764fa11-79d6-41a7-9f0d08b7-044f392b.dmp
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x01351e7a, pid=20932, tid=0x7f8e93d31a40
#
# JRE version: Java(TM) SE Runtime Environment (8.0_144-b01) (build
1.8.0_144-b01)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.144-b01 mixed mode linux-amd64
compressed oops)
# Problematic frame:
# C [rpc-mgr-kerberized-test+0xf51e7a]
impala::MiniKdcWrapper::CreateServiceKeytab(std::__cxx11::basic_string, std::allocator > const&,
std::__cxx11::basic_string, std::allocator
>*)+0x1a
#
# Core dump written. Default location:
/data0/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/be/src/rpc/core
or core.20932
#
# An error report file with more information is saved as:
#
/data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/logs/hs_err_pid20932.log
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#{noformat}
The minidump shows a SIGSEGV:
{noformat}
Crash reason: SIGSEGV
Crash address: 0x0
Process uptime: not available
Thread 0 (crashed)
0
rpc-mgr-kerberized-test!impala::MiniKdcWrapper::CreateServiceKeytab(std::__cxx11::basic_string, std::allocator > const&,
std::__cxx11::basic_string, std::allocator
>*) [mini-kdc-wrapper.cc : 70 + 0x0]
1 rpc-mgr-kerberized-test!main [rpc-mgr-kerberized-test.cc : 306 + 0x21]
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-10905) query_test/test_iceberg.py test_time_travel fails in exhaustive builds
[
https://issues.apache.org/jira/browse/IMPALA-10905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell reassigned IMPALA-10905:
--
Assignee: Zoltán Borók-Nagy
> query_test/test_iceberg.py test_time_travel fails in exhaustive builds
> --
>
> Key: IMPALA-10905
> URL: https://issues.apache.org/jira/browse/IMPALA-10905
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 4.1.0
>Reporter: Joe McDonnell
>Assignee: Zoltán Borók-Nagy
>Priority: Blocker
> Labels: broken-build
>
> Exhaustive tests are seeing failures in
> query_test/test_iceberg.py::test_time_travel:
> {noformat}
> query_test/test_iceberg.py:166: in test_time_travel
> expect_results_t(quote(ts_1), ['1'])
> query_test/test_iceberg.py:132: in expect_results_t
> expected_results)
> query_test/test_iceberg.py:124: in expect_results
> data = self.execute_query(query)
> common/impala_test_suite.py:831: in wrapper
> return function(*args, **kwargs)
> common/impala_test_suite.py:863: in execute_query
> return self.__execute_query(self.client, query, query_options)
> common/impala_test_suite.py:940: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:205: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:189: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:365: in __execute_query
> handle = self.execute_query_async(query_string, user=user)
> beeswax/impala_beeswax.py:359: in execute_query_async
> handle = self.__do_rpc(lambda: self.imp_service.query(query,))
> beeswax/impala_beeswax.py:522: in __do_rpc
> raise ImpalaBeeswaxException(self.__build_error_message(b), b)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> EINNER EXCEPTION:
> EMESSAGE: IllegalArgumentException: Cannot find a snapshot older than
> 2021-09-03 16:18:22.295{noformat}
> This tests was introduced with IMPALA-10840. It seems pretty consistent so
> far.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-10908) Support Locale for SimpleDateFormat parser
Sergio Sainz created IMPALA-10908:
-
Summary: Support Locale for SimpleDateFormat parser
Key: IMPALA-10908
URL: https://issues.apache.org/jira/browse/IMPALA-10908
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 4.0.0
Reporter: Sergio Sainz
IDEA (enhacement request):
Java's SimpleDateFormat supports additional argument for LOCALE:
[https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/text/SimpleDateFormat.html#%3Cinit%3E(java.lang.String,java.util.Locale)]
Impala should allow providing locale to the TO_TIMESTAMP function:
Example:
Input: select to_timestamp('01-Ene-2021', 'dd-MMM-', 'es_ES');
Expected output:
01-01-21
Enero (spanish) = January (english)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-10879) Add parquet stats to iceberg manifest
[ https://issues.apache.org/jira/browse/IMPALA-10879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Jeges resolved IMPALA-10879. --- Resolution: Implemented > Add parquet stats to iceberg manifest > - > > Key: IMPALA-10879 > URL: https://issues.apache.org/jira/browse/IMPALA-10879 > Project: IMPALA > Issue Type: Improvement > Components: Backend, Frontend >Affects Versions: Impala 4.0.0 >Reporter: Attila Jeges >Assignee: Attila Jeges >Priority: Major > Labels: impala-iceberg > > Parquet stats should be written to iceberg manifest as per-datafile metrics. > This task is specifically about the following metrics: > - column_sizes : Map from column id to the total size on disk of all regions > that store the column. Does not include bytes necessary to read other > columns, like footers. Leave null for row-oriented formats > - null_value_counts : Map from column id to number of null values in the > column. > - lower_bounds : Map from column id to lower bound in the column serialized > as binary. Each value must be less than or equal to all non-null, non-NaN > values in the column for the file. > - upper_bounds : Map from column id to upper bound in the column serialized > as binary. Each value must be greater than or equal to all non-null, non-Nan > values in the column for the file. > Iceberg manifest doc: > https://iceberg.apache.org/spec/#manifests > lower_bounds and upper_bounds values should be Single-value serialized to > binary: > https://iceberg.apache.org/spec/#appendix-d-single-value-serialization -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-10907) Refactor MetastoreEvents class
Vihang Karajgaonkar created IMPALA-10907: Summary: Refactor MetastoreEvents class Key: IMPALA-10907 URL: https://issues.apache.org/jira/browse/IMPALA-10907 Project: IMPALA Issue Type: Improvement Reporter: Vihang Karajgaonkar Assignee: Vihang Karajgaonkar The MetastoreEvents.java is single class which has a bunch of inner classes (most of which are public). The file has become pretty large and it would make sense to refactor the file into separate classes to improve code readability. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-10776) Hold write lock for less time in ALTER TABLE RECOVER PARTITIONS
[ https://issues.apache.org/jira/browse/IMPALA-10776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar reassigned IMPALA-10776: Assignee: Vihang Karajgaonkar > Hold write lock for less time in ALTER TABLE RECOVER PARTITIONS > --- > > Key: IMPALA-10776 > URL: https://issues.apache.org/jira/browse/IMPALA-10776 > Project: IMPALA > Issue Type: Improvement > Components: Frontend >Reporter: Csaba Ringhofer >Assignee: Vihang Karajgaonkar >Priority: Major > > ALTER TABLE RECOVER PARTITIONS holds a write lock on the table for the whole > time while it lists the HDFS directories and creates the new partitions in > HMS. This can potentially take a long time and block catalog updates. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10776) Hold write lock for less time in ALTER TABLE RECOVER PARTITIONS
[ https://issues.apache.org/jira/browse/IMPALA-10776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17411430#comment-17411430 ] Vihang Karajgaonkar commented on IMPALA-10776: -- I can take a stab at this. > Hold write lock for less time in ALTER TABLE RECOVER PARTITIONS > --- > > Key: IMPALA-10776 > URL: https://issues.apache.org/jira/browse/IMPALA-10776 > Project: IMPALA > Issue Type: Improvement > Components: Frontend >Reporter: Csaba Ringhofer >Priority: Major > > ALTER TABLE RECOVER PARTITIONS holds a write lock on the table for the whole > time while it lists the HDFS directories and creates the new partitions in > HMS. This can potentially take a long time and block catalog updates. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-10906) Adjust dependencies after HIVE-24852
[
https://issues.apache.org/jira/browse/IMPALA-10906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fang-Yu Rao updated IMPALA-10906:
-
Description:
HIVE-24852 added the dependency on the artifact {{hadoop-hdfs}} under the
module of {{hive-service}}, which is a dependency of {{hive-jdbc}} that Impala
relies on. However, since {{hadoop-hdfs}} transitively pulls in the artifact
{{jersey-server}}, which is a banned dependency by Impala, we had to explicitly
exclude {{jersey-server}} when adding {{hive-jdbc}} as a dependency so that
Impala frontend could be compiled next time when we bump up
{{CDP_BUILD_NUMBER}} that includes this Hive patch.
Moreover, due to the fact that after HIVE-24852, the creation of a partitioned
iceberg table requires the existence of the class
{{org.apache.hadoop.hdfs.protocol.SnapshotException}} on Impala's classpath at
runtime, we had to explicitly add the dependency on the artifact
{{hadoop-hdfs}} so that such an operation will not result in a
{{NoClassDefFoundError}}. We also need to explicitly excluded some banned
artifacts that were transitively pulled in by {{hadoop-hdfs}} so that Impala
frontend could be compiled.
For easy reference, a query to reproduce the {{NoClassDefFoundError}} issue is
provided in the following. A very similar query will be executed during the
loading of Impala's test data so that once the dependencies are properly
revised, we expect Impala to be able to load test data correctly.
{code:sql}
[localhost:21050] default> CREATE TABLE IF NOT EXISTS
default.iceberg_int_partitioned_tbl (i INT, j INT, k INT)
> PARTITION BY SPEC (i identity, j identity)
> STORED AS ICEBERG;
Query: CREATE TABLE IF NOT EXISTS default.iceberg_int_partitioned_tbl (i INT, j
INT, k INT)
PARTITION BY SPEC (i identity, j identity)
STORED AS ICEBERG
ERROR: NoClassDefFoundError: org/apache/hadoop/hdfs/protocol/SnapshotException
CAUSED BY: ClassNotFoundException:
org.apache.hadoop.hdfs.protocol.SnapshotException
{code}
was:
HIVE-24852 added the dependency on the artifact {{hadoop-hdfs}} under the
module of {{hive-service}}, which is a dependency of {{hive-jdbc}} that Impala
relies on. However, since {{hadoop-hdfs}} transitively pulls in the artifact
{{jersey-server}}, which is a banned dependency by Impala, we had to explicitly
exclude {{jersey-server}} when adding {{hive-jdbc}} as a dependency so that
Impala frontend could be compiled next time when we bump up
{{CDP_BUILD_NUMBER}} that includes this Hive patch.
Moreover, due to the fact that after HIVE-24852, the creation of a partitioned
iceberg table requires the existence of the class
{{org.apache.hadoop.hdfs.protocol.SnapshotException}} on Impala's classpath at
runtime, we had to explicitly add the dependency on the artifact
{{hadoop-hdfs}} so that such an operation will not result in a
{{NoClassDefFoundError}}. We also need to explicitly excluded some banned
artifacts that were transitively pulled in by {{hadoop-hdfs}} so that Impala
frontend could be compiled.
For easy reference, a query to reproduce the {{NoClassDefFoundError}} issue is
provided in the following. A very similar query has been executed during the
loading of Impala's test data so that once the dependencies are properly
revised, we expect Impala to be able to load test data correctly.
{code:sql}
[localhost:21050] default> CREATE TABLE IF NOT EXISTS
default.iceberg_int_partitioned_tbl (i INT, j INT, k INT)
> PARTITION BY SPEC (i identity, j identity)
> STORED AS ICEBERG;
Query: CREATE TABLE IF NOT EXISTS default.iceberg_int_partitioned_tbl (i INT, j
INT, k INT)
PARTITION BY SPEC (i identity, j identity)
STORED AS ICEBERG
ERROR: NoClassDefFoundError: org/apache/hadoop/hdfs/protocol/SnapshotException
CAUSED BY: ClassNotFoundException:
org.apache.hadoop.hdfs.protocol.SnapshotException
{code}
> Adjust dependencies after HIVE-24852
>
>
> Key: IMPALA-10906
> URL: https://issues.apache.org/jira/browse/IMPALA-10906
> Project: IMPALA
> Issue Type: Improvement
>Reporter: Fang-Yu Rao
>Assignee: Fang-Yu Rao
>Priority: Major
>
> HIVE-24852 added the dependency on the artifact {{hadoop-hdfs}} under the
> module of {{hive-service}}, which is a dependency of {{hive-jdbc}} that
> Impala relies on. However, since {{hadoop-hdfs}} transitively pulls in the
> artifact {{jersey-server}}, which is a banned dependency by Impala, we had to
> explicitly exclude {{jersey-server}} when adding {{hive-jdbc}} as a
> dependency so that Impala frontend could be compiled next time when we bump
> up {{CDP_BUILD_NUMBER}} that includes this Hive patch.
> Moreover, due to the fact that after HIVE-24852, the creation of a
> partitioned iceberg table requires the
[jira] [Created] (IMPALA-10906) Adjust dependencies after HIVE-24852
Fang-Yu Rao created IMPALA-10906:
Summary: Adjust dependencies after HIVE-24852
Key: IMPALA-10906
URL: https://issues.apache.org/jira/browse/IMPALA-10906
Project: IMPALA
Issue Type: Improvement
Reporter: Fang-Yu Rao
Assignee: Fang-Yu Rao
HIVE-24852 added the dependency on the artifact {{hadoop-hdfs}} under the
module of {{hive-service}}, which is a dependency of {{hive-jdbc}} that Impala
relies on. However, since {{hadoop-hdfs}} transitively pulls in the artifact
{{jersey-server}}, which is a banned dependency by Impala, we had to explicitly
exclude {{jersey-server}} when adding {{hive-jdbc}} as a dependency so that
Impala frontend could be compiled next time when we bump up
{{CDP_BUILD_NUMBER}} that includes this Hive patch.
Moreover, due to the fact that after HIVE-24852, the creation of a partitioned
iceberg table requires the existence of the class
{{org.apache.hadoop.hdfs.protocol.SnapshotException}} on Impala's classpath at
runtime, we had to explicitly add the dependency on the artifact
{{hadoop-hdfs}} so that such an operation will not result in a
{{NoClassDefFoundError}}. We also need to explicitly excluded some banned
artifacts that were transitively pulled in by {{hadoop-hdfs}} so that Impala
frontend could be compiled.
For easy reference, a query to reproduce the {{NoClassDefFoundError}} issue is
provided in the following. A very similar query has been executed during the
loading of Impala's test data so that once the dependencies are properly
revised, we expect Impala to be able to load test data correctly.
{code:sql}
[localhost:21050] default> CREATE TABLE IF NOT EXISTS
default.iceberg_int_partitioned_tbl (i INT, j INT, k INT)
> PARTITION BY SPEC (i identity, j identity)
> STORED AS ICEBERG;
Query: CREATE TABLE IF NOT EXISTS default.iceberg_int_partitioned_tbl (i INT, j
INT, k INT)
PARTITION BY SPEC (i identity, j identity)
STORED AS ICEBERG
ERROR: NoClassDefFoundError: org/apache/hadoop/hdfs/protocol/SnapshotException
CAUSED BY: ClassNotFoundException:
org.apache.hadoop.hdfs.protocol.SnapshotException
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-10905) query_test/test_iceberg.py test_time_travel fails in exhaustive builds
Joe McDonnell created IMPALA-10905:
--
Summary: query_test/test_iceberg.py test_time_travel fails in
exhaustive builds
Key: IMPALA-10905
URL: https://issues.apache.org/jira/browse/IMPALA-10905
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 4.1.0
Reporter: Joe McDonnell
Exhaustive tests are seeing failures in
query_test/test_iceberg.py::test_time_travel:
{noformat}
query_test/test_iceberg.py:166: in test_time_travel
expect_results_t(quote(ts_1), ['1'])
query_test/test_iceberg.py:132: in expect_results_t
expected_results)
query_test/test_iceberg.py:124: in expect_results
data = self.execute_query(query)
common/impala_test_suite.py:831: in wrapper
return function(*args, **kwargs)
common/impala_test_suite.py:863: in execute_query
return self.__execute_query(self.client, query, query_options)
common/impala_test_suite.py:940: in __execute_query
return impalad_client.execute(query, user=user)
common/impala_connection.py:205: in execute
return self.__beeswax_client.execute(sql_stmt, user=user)
beeswax/impala_beeswax.py:189: in execute
handle = self.__execute_query(query_string.strip(), user=user)
beeswax/impala_beeswax.py:365: in __execute_query
handle = self.execute_query_async(query_string, user=user)
beeswax/impala_beeswax.py:359: in execute_query_async
handle = self.__do_rpc(lambda: self.imp_service.query(query,))
beeswax/impala_beeswax.py:522: in __do_rpc
raise ImpalaBeeswaxException(self.__build_error_message(b), b)
E ImpalaBeeswaxException: ImpalaBeeswaxException:
EINNER EXCEPTION:
EMESSAGE: IllegalArgumentException: Cannot find a snapshot older than
2021-09-03 16:18:22.295{noformat}
This tests was introduced with IMPALA-10840. It seems pretty consistent so far.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10896) Tests in TestImpalaShellInteractive failed in S3 build when strict_hs2_protocol=True
[
https://issues.apache.org/jira/browse/IMPALA-10896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17411387#comment-17411387
]
Steve Carlin commented on IMPALA-10896:
---
I have something that might fix this?
I do have something posted on gerrit: [https://gerrit.cloudera.org/#/c/17820/]
But I have yet to test this (need to figure out how to test this on the S3 run).
> Tests in TestImpalaShellInteractive failed in S3 build when
> strict_hs2_protocol=True
>
>
> Key: IMPALA-10896
> URL: https://issues.apache.org/jira/browse/IMPALA-10896
> Project: IMPALA
> Issue Type: Bug
>Reporter: Quanlong Huang
>Assignee: Steve Carlin
>Priority: Blocker
> Labels: broken-build
>
> See test failures and timeouts in a S3 build:
> {code:java}
> 23:38:38
> shell/test_shell_interactive.py::TestImpalaShellInteractive::test_write_delimited[table_format_and_file_extension:
> ('parquet', '.parq') | strict_hs2_protocol: False | protocol: beeswax] <-
> ../../../../../../../data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/shell/test_shell_interactive.py
> PASSED
> 23:38:38
> shell/test_shell_interactive.py::TestImpalaShellInteractive::test_write_delimited[table_format_and_file_extension:
> ('textfile', '.txt') | strict_hs2_protocol: True | protocol: hs2] <-
> ../../../../../../../data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/shell/test_shell_interactive.py
> FAILED
> 23:38:43
> shell/test_shell_interactive.py::TestImpalaShellInteractive::test_write_delimited[table_format_and_file_extension:
> ('textfile', '.txt') | strict_hs2_protocol: False | protocol: hs2-http] <-
> ../../../../../../../data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/shell/test_shell_interactive.py
> PASSED
> 01:38:43
> shell/test_shell_interactive.py::TestImpalaShellInteractive::test_write_delimited[table_format_and_file_extension:
> ('parquet', '.parq') | strict_hs2_protocol: True | protocol: hs2-http] <-
> ../../../../../../../data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/shell/test_shell_interactive.py
> FAILED
> 01:38:43
> shell/test_shell_interactive.py::TestImpalaShellInteractive::test_write_delimited[table_format_and_file_extension:
> ('parquet', '.parq') | strict_hs2_protocol: True | protocol: hs2] <-
> ../../../../../../../data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/shell/test_shell_interactive.py
> FAILED
> 03:38:43
> shell/test_shell_interactive.py::TestImpalaShellInteractive::test_change_delimiter[table_format_and_file_extension:
> ('parquet', '.parq') | strict_hs2_protocol: False | protocol: beeswax] <-
> ../../../../../../../data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/shell/test_shell_interactive.py
> PASSED
> 05:38:43
> shell/test_shell_interactive.py::TestImpalaShellInteractive::test_change_delimiter[table_format_and_file_extension:
> ('textfile', '.txt') | strict_hs2_protocol: True | protocol: hs2] <-
> ../../../../../../../data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/shell/test_shell_interactive.py
> FAILED
> 05:38:44
> shell/test_shell_interactive.py::TestImpalaShellInteractive::test_change_delimiter[table_format_and_file_extension:
> ('textfile', '.txt') | strict_hs2_protocol: False | protocol: hs2-http] <-
> ../../../../../../../data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/shell/test_shell_interactive.py
> PASSED
> 07:38:44
> shell/test_shell_interactive.py::TestImpalaShellInteractive::test_change_delimiter[table_format_and_file_extension:
> ('parquet', '.parq') | strict_hs2_protocol: True | protocol: hs2-http] <-
> ../../../../../../../data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/shell/test_shell_interactive.py
> FAILED
> 07:38:44
> shell/test_shell_interactive.py::TestImpalaShellInteractive::test_change_delimiter[table_format_and_file_extension:
> ('parquet', '.parq') | strict_hs2_protocol: True | protocol: hs2] <-
> ../../../../../../../data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/shell/test_shell_interactive.py
> FAILED
> 09:38:44
> shell/test_shell_interactive.py::TestImpalaShellInteractive::test_print_to_file[table_format_and_file_extension:
> ('parquet', '.parq') | strict_hs2_protocol: False | protocol: beeswax] <-
> ../../../../../../../data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/shell/test_shell_interactive.py
> PASSED
> 11:38:44
> shell/test_shell_interactive.py::TestImpalaShellInteractive::test_print_to_file[table_format_and_file_extension:
> ('textfile', '.txt') | strict_hs2_protocol: True | protocol: hs2] <-
>
[jira] [Commented] (IMPALA-10896) Tests in TestImpalaShellInteractive failed in S3 build when strict_hs2_protocol=True
[
https://issues.apache.org/jira/browse/IMPALA-10896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17411383#comment-17411383
]
Joe McDonnell commented on IMPALA-10896:
The Impala tests don't run Hive in the S3 configuration, so if the
strict_hs2_protocol=True case is trying to talk to Hive, it will fail. If that
is the case, we can just skip these tests on S3.
> Tests in TestImpalaShellInteractive failed in S3 build when
> strict_hs2_protocol=True
>
>
> Key: IMPALA-10896
> URL: https://issues.apache.org/jira/browse/IMPALA-10896
> Project: IMPALA
> Issue Type: Bug
>Reporter: Quanlong Huang
>Assignee: Steve Carlin
>Priority: Blocker
> Labels: broken-build
>
> See test failures and timeouts in a S3 build:
> {code:java}
> 23:38:38
> shell/test_shell_interactive.py::TestImpalaShellInteractive::test_write_delimited[table_format_and_file_extension:
> ('parquet', '.parq') | strict_hs2_protocol: False | protocol: beeswax] <-
> ../../../../../../../data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/shell/test_shell_interactive.py
> PASSED
> 23:38:38
> shell/test_shell_interactive.py::TestImpalaShellInteractive::test_write_delimited[table_format_and_file_extension:
> ('textfile', '.txt') | strict_hs2_protocol: True | protocol: hs2] <-
> ../../../../../../../data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/shell/test_shell_interactive.py
> FAILED
> 23:38:43
> shell/test_shell_interactive.py::TestImpalaShellInteractive::test_write_delimited[table_format_and_file_extension:
> ('textfile', '.txt') | strict_hs2_protocol: False | protocol: hs2-http] <-
> ../../../../../../../data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/shell/test_shell_interactive.py
> PASSED
> 01:38:43
> shell/test_shell_interactive.py::TestImpalaShellInteractive::test_write_delimited[table_format_and_file_extension:
> ('parquet', '.parq') | strict_hs2_protocol: True | protocol: hs2-http] <-
> ../../../../../../../data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/shell/test_shell_interactive.py
> FAILED
> 01:38:43
> shell/test_shell_interactive.py::TestImpalaShellInteractive::test_write_delimited[table_format_and_file_extension:
> ('parquet', '.parq') | strict_hs2_protocol: True | protocol: hs2] <-
> ../../../../../../../data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/shell/test_shell_interactive.py
> FAILED
> 03:38:43
> shell/test_shell_interactive.py::TestImpalaShellInteractive::test_change_delimiter[table_format_and_file_extension:
> ('parquet', '.parq') | strict_hs2_protocol: False | protocol: beeswax] <-
> ../../../../../../../data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/shell/test_shell_interactive.py
> PASSED
> 05:38:43
> shell/test_shell_interactive.py::TestImpalaShellInteractive::test_change_delimiter[table_format_and_file_extension:
> ('textfile', '.txt') | strict_hs2_protocol: True | protocol: hs2] <-
> ../../../../../../../data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/shell/test_shell_interactive.py
> FAILED
> 05:38:44
> shell/test_shell_interactive.py::TestImpalaShellInteractive::test_change_delimiter[table_format_and_file_extension:
> ('textfile', '.txt') | strict_hs2_protocol: False | protocol: hs2-http] <-
> ../../../../../../../data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/shell/test_shell_interactive.py
> PASSED
> 07:38:44
> shell/test_shell_interactive.py::TestImpalaShellInteractive::test_change_delimiter[table_format_and_file_extension:
> ('parquet', '.parq') | strict_hs2_protocol: True | protocol: hs2-http] <-
> ../../../../../../../data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/shell/test_shell_interactive.py
> FAILED
> 07:38:44
> shell/test_shell_interactive.py::TestImpalaShellInteractive::test_change_delimiter[table_format_and_file_extension:
> ('parquet', '.parq') | strict_hs2_protocol: True | protocol: hs2] <-
> ../../../../../../../data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/shell/test_shell_interactive.py
> FAILED
> 09:38:44
> shell/test_shell_interactive.py::TestImpalaShellInteractive::test_print_to_file[table_format_and_file_extension:
> ('parquet', '.parq') | strict_hs2_protocol: False | protocol: beeswax] <-
> ../../../../../../../data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/shell/test_shell_interactive.py
> PASSED
> 11:38:44
> shell/test_shell_interactive.py::TestImpalaShellInteractive::test_print_to_file[table_format_and_file_extension:
> ('textfile', '.txt') | strict_hs2_protocol: True | protocol: hs2] <-
>
[jira] [Updated] (IMPALA-10896) Tests in TestImpalaShellInteractive failed in S3 build when strict_hs2_protocol=True
[
https://issues.apache.org/jira/browse/IMPALA-10896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell updated IMPALA-10896:
---
Priority: Blocker (was: Critical)
> Tests in TestImpalaShellInteractive failed in S3 build when
> strict_hs2_protocol=True
>
>
> Key: IMPALA-10896
> URL: https://issues.apache.org/jira/browse/IMPALA-10896
> Project: IMPALA
> Issue Type: Bug
>Reporter: Quanlong Huang
>Assignee: Steve Carlin
>Priority: Blocker
> Labels: broken-build
>
> See test failures and timeouts in a S3 build:
> {code:java}
> 23:38:38
> shell/test_shell_interactive.py::TestImpalaShellInteractive::test_write_delimited[table_format_and_file_extension:
> ('parquet', '.parq') | strict_hs2_protocol: False | protocol: beeswax] <-
> ../../../../../../../data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/shell/test_shell_interactive.py
> PASSED
> 23:38:38
> shell/test_shell_interactive.py::TestImpalaShellInteractive::test_write_delimited[table_format_and_file_extension:
> ('textfile', '.txt') | strict_hs2_protocol: True | protocol: hs2] <-
> ../../../../../../../data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/shell/test_shell_interactive.py
> FAILED
> 23:38:43
> shell/test_shell_interactive.py::TestImpalaShellInteractive::test_write_delimited[table_format_and_file_extension:
> ('textfile', '.txt') | strict_hs2_protocol: False | protocol: hs2-http] <-
> ../../../../../../../data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/shell/test_shell_interactive.py
> PASSED
> 01:38:43
> shell/test_shell_interactive.py::TestImpalaShellInteractive::test_write_delimited[table_format_and_file_extension:
> ('parquet', '.parq') | strict_hs2_protocol: True | protocol: hs2-http] <-
> ../../../../../../../data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/shell/test_shell_interactive.py
> FAILED
> 01:38:43
> shell/test_shell_interactive.py::TestImpalaShellInteractive::test_write_delimited[table_format_and_file_extension:
> ('parquet', '.parq') | strict_hs2_protocol: True | protocol: hs2] <-
> ../../../../../../../data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/shell/test_shell_interactive.py
> FAILED
> 03:38:43
> shell/test_shell_interactive.py::TestImpalaShellInteractive::test_change_delimiter[table_format_and_file_extension:
> ('parquet', '.parq') | strict_hs2_protocol: False | protocol: beeswax] <-
> ../../../../../../../data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/shell/test_shell_interactive.py
> PASSED
> 05:38:43
> shell/test_shell_interactive.py::TestImpalaShellInteractive::test_change_delimiter[table_format_and_file_extension:
> ('textfile', '.txt') | strict_hs2_protocol: True | protocol: hs2] <-
> ../../../../../../../data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/shell/test_shell_interactive.py
> FAILED
> 05:38:44
> shell/test_shell_interactive.py::TestImpalaShellInteractive::test_change_delimiter[table_format_and_file_extension:
> ('textfile', '.txt') | strict_hs2_protocol: False | protocol: hs2-http] <-
> ../../../../../../../data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/shell/test_shell_interactive.py
> PASSED
> 07:38:44
> shell/test_shell_interactive.py::TestImpalaShellInteractive::test_change_delimiter[table_format_and_file_extension:
> ('parquet', '.parq') | strict_hs2_protocol: True | protocol: hs2-http] <-
> ../../../../../../../data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/shell/test_shell_interactive.py
> FAILED
> 07:38:44
> shell/test_shell_interactive.py::TestImpalaShellInteractive::test_change_delimiter[table_format_and_file_extension:
> ('parquet', '.parq') | strict_hs2_protocol: True | protocol: hs2] <-
> ../../../../../../../data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/shell/test_shell_interactive.py
> FAILED
> 09:38:44
> shell/test_shell_interactive.py::TestImpalaShellInteractive::test_print_to_file[table_format_and_file_extension:
> ('parquet', '.parq') | strict_hs2_protocol: False | protocol: beeswax] <-
> ../../../../../../../data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/shell/test_shell_interactive.py
> PASSED
> 11:38:44
> shell/test_shell_interactive.py::TestImpalaShellInteractive::test_print_to_file[table_format_and_file_extension:
> ('textfile', '.txt') | strict_hs2_protocol: True | protocol: hs2] <-
> ../../../../../../../data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/shell/test_shell_interactive.py
> FAILED
> 11:38:45
> shell/test_shell_interactive.py::TestImpalaShellInteractive::test_print_to_file[table_format_and_file_extension:
> ('textfile',
[jira] [Commented] (IMPALA-10723) Allow basic querying and computing stats on a materialized view
[
https://issues.apache.org/jira/browse/IMPALA-10723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17411297#comment-17411297
]
Aman Sinha commented on IMPALA-10723:
-
I had posted a patch sometime ago: https://gerrit.cloudera.org/c/17595/
It needs a bit of follow-up which is delayed due to other priorities. I will
try and get back to it sometime soon.
> Allow basic querying and computing stats on a materialized view
> ---
>
> Key: IMPALA-10723
> URL: https://issues.apache.org/jira/browse/IMPALA-10723
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
>Reporter: Aman Sinha
>Assignee: Aman Sinha
>Priority: Major
>
> Currently, in Impala, a Materialized View (MV) created via Hive is visible
> through the metadata catalog and can be queried but the query is expanded to
> its corresponding view definition. This is incorrect because a materialized
> view is a regular (physical) table, not a view. Even though Impala does not
> support either creating MV or automatic rewriting to use MV, querying an MV
> directly should be allowed.
> Here's the current behavior:
> {noformat}
> [localhost:21050] functional> explain select * from materialized_view;
> Query: explain select * from materialized_view
> ++
> | Explain String
>|
> ++
> | Max Per-Host Resource Reservation: Memory=4.00MB Threads=2
>|
> | Per-Host Resource Estimates: Memory=10MB
>|
> | WARNING: The following tables are missing relevant table and/or column
> statistics. |
> | functional.insert_only_transactional_table
>|
> |
>|
> | PLAN-ROOT SINK
>|
> | |
>|
> | 01:EXCHANGE [UNPARTITIONED]
>|
> | |
>|
> | 00:SCAN HDFS [functional.insert_only_transactional_table]
>|
> |HDFS partitions=1/1 files=0 size=0B
>|
> |row-size=4B cardinality=0
>|
> ++
> {noformat}
> Note that the plan shows the scan of the underlying table instead of the
> materialized_view table. We should only be scanning the MV (including
> applying partition pruning, predicate pushdown etc.) and not treating this as
> a view.
> This JIRA is to enhance the frontend to recognize a materialized view as a
> table rather than a view. This will further allow commands such as COMPUTE
> STATS, DROP STATS, SHOW [TABLE | COLUMN] STATS to be run on the MV. One
> motivation for doing this is to allow an external frontend that supports
> automatic query rewrites using MVs to leverage the statistics on MVs.
> Since Impala is not creating the MV, we will need to block DML operations on
> the MV. Further, special handling needs to be done for Ranger authorization
> policies such that any column masking/row filtering policies defined on the
> source tables of the MV are taken into consideration.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-10904) Add some perf tools to the repository
[ https://issues.apache.org/jira/browse/IMPALA-10904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltán Borók-Nagy reassigned IMPALA-10904: -- Assignee: Zoltán Borók-Nagy > Add some perf tools to the repository > - > > Key: IMPALA-10904 > URL: https://issues.apache.org/jira/browse/IMPALA-10904 > Project: IMPALA > Issue Type: Improvement > Components: Infrastructure >Reporter: Zoltán Borók-Nagy >Assignee: Zoltán Borók-Nagy >Priority: Major > > Flame graphs are efficient tools to measure performance. > We should make it easy to generate flame graphs for Impala queries, i.e. we > should have some scripts in the repo that developers can easily use. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-10904) Add some perf tools to the repository
Zoltán Borók-Nagy created IMPALA-10904: -- Summary: Add some perf tools to the repository Key: IMPALA-10904 URL: https://issues.apache.org/jira/browse/IMPALA-10904 Project: IMPALA Issue Type: Improvement Components: Infrastructure Reporter: Zoltán Borók-Nagy Flame graphs are efficient tools to measure performance. We should make it easy to generate flame graphs for Impala queries, i.e. we should have some scripts in the repo that developers can easily use. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
