[jira] [Commented] (IMPALA-7864) TestLocalCatalogRetries::test_replan_limit is flaky
[ https://issues.apache.org/jira/browse/IMPALA-7864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987584#comment-16987584 ] Quanlong Huang commented on IMPALA-7864: Saw this again in a private build. Can't reproduce it in my local env. > TestLocalCatalogRetries::test_replan_limit is flaky > --- > > Key: IMPALA-7864 > URL: https://issues.apache.org/jira/browse/IMPALA-7864 > Project: IMPALA > Issue Type: Bug > Components: Catalog >Affects Versions: Impala 3.0, Impala 2.12.0 > Environment: Ubuntu 16.04 >Reporter: Jim Apple >Assignee: Vihang Karajgaonkar >Priority: Critical > Labels: broken-build, catalog-v2, flaky > Fix For: Impala 3.2.0 > > > In https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/3605/, > TestLocalCatalogRetries::test_replan_limit failed on an unrelated patch. On > my development machine, the test passed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9151) Number of executors during planning needs to account for suspended executor groupa
[
https://issues.apache.org/jira/browse/IMPALA-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Volker resolved IMPALA-9151.
-
Fix Version/s: Impala 3.4.0
Resolution: Fixed
> Number of executors during planning needs to account for suspended executor
> groupa
> --
>
> Key: IMPALA-9151
> URL: https://issues.apache.org/jira/browse/IMPALA-9151
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Reporter: Lars Volker
>Assignee: Lars Volker
>Priority: Major
> Fix For: Impala 3.4.0
>
>
> When configuring Impala with executor groups, the planner might see a
> {{ExecutorMembershipSnapshot}} that has no executors in it. This can happen
> if the first executor group has not started up yet, or if all executor groups
> have been shutdown. If this happens, the planner will make sub-optimal
> decisions, e.g. decide on a broadcast join vs a PHJ. In the former case, we
> should have a configurable fallback cluster size to use during planning. In
> the latter case, we should hang on to the last executor group size that we
> had observed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9181) Serialize TQueryCtx once per query
[ https://issues.apache.org/jira/browse/IMPALA-9181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Tauber-Marshall resolved IMPALA-9181. Fix Version/s: Impala 3.4.0 Resolution: Fixed > Serialize TQueryCtx once per query > -- > > Key: IMPALA-9181 > URL: https://issues.apache.org/jira/browse/IMPALA-9181 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 3.4.0 >Reporter: Thomas Tauber-Marshall >Assignee: Thomas Tauber-Marshall >Priority: Major > Fix For: Impala 3.4.0 > > > When issuing Exec() rpcs to backends, we currently serialize the TQueryCtx > once per backend. This is inefficient as the TQueryCtx is the same for all > backends and really only needs to be serialized once. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9181) Serialize TQueryCtx once per query
[ https://issues.apache.org/jira/browse/IMPALA-9181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Tauber-Marshall resolved IMPALA-9181. Fix Version/s: Impala 3.4.0 Resolution: Fixed > Serialize TQueryCtx once per query > -- > > Key: IMPALA-9181 > URL: https://issues.apache.org/jira/browse/IMPALA-9181 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 3.4.0 >Reporter: Thomas Tauber-Marshall >Assignee: Thomas Tauber-Marshall >Priority: Major > Fix For: Impala 3.4.0 > > > When issuing Exec() rpcs to backends, we currently serialize the TQueryCtx > once per backend. This is inefficient as the TQueryCtx is the same for all > backends and really only needs to be serialized once. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IMPALA-9181) Serialize TQueryCtx once per query
[ https://issues.apache.org/jira/browse/IMPALA-9181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987356#comment-16987356 ] ASF subversion and git services commented on IMPALA-9181: - Commit a1588e44980c648cb7f9263cbd0409abfbaeacf7 in impala's branch refs/heads/master from Thomas Tauber-Marshall [ https://gitbox.apache.org/repos/asf?p=impala.git;h=a1588e4 ] IMPALA-9181: Serialize TQueryCtx once per query When issuing Exec() rpcs to backends, we currently serialize the TQueryCtx once per backend. This is inefficient as the TQueryCtx is the same for all backends and really only needs to be serialized once. Serializing the TQueryCtx can be expensive as it contains both the full text of the original query and the descriptor table, which can be quite large. In a synthetic dataset I tested with, scanning a table with 100k partitions leads to a descriptor table size of ~20MB. This patch serializes the TQueryCtx in the coordinator and then passes it to each BackendState when calling Exec(). Followup work might consider if we really need all of the info in the TQueryCtx to be distributed to all backends. Testing: - Passed full run of existing tests. - Single node perf run showed no significant change. Change-Id: I6a4dd302fd5602ec2775492a041ddd51e7d7a6c6 Reviewed-on: http://gerrit.cloudera.org:8080/14777 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Serialize TQueryCtx once per query > -- > > Key: IMPALA-9181 > URL: https://issues.apache.org/jira/browse/IMPALA-9181 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 3.4.0 >Reporter: Thomas Tauber-Marshall >Assignee: Thomas Tauber-Marshall >Priority: Major > > When issuing Exec() rpcs to backends, we currently serialize the TQueryCtx > once per backend. This is inefficient as the TQueryCtx is the same for all > backends and really only needs to be serialized once. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9131) Use single quotes when printing out FORMAT clause within CAST.
[
https://issues.apache.org/jira/browse/IMPALA-9131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987275#comment-16987275
]
ASF subversion and git services commented on IMPALA-9131:
-
Commit 1dc729f2b27da46671cd33903b2ef52a0ae29ebe in impala's branch
refs/heads/master from Gabor Kaszab
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=1dc729f ]
IMPALA-9131: Use single quotes around FORMAT clause in CAST
When running a CAST(..FORMAT..) query then the header of the output
shows the value of the FORMAT clause surrounded by double quotes.
However, the SQL way is to use single quotes for strings so this
patch changes the printout from using double quotes to use single
quotes instead.
Additionally, this fixes a bug where it wasn't possible to have a
single quote separator in a format string that was itself surrounded
by single quotes. As a fix the single quote separator can be escaped
by a backslash in this case.
Another additional fix to make the FX modifier case-insensitive.
Change-Id: I3310abfa6f3ccbbe4c437846c6dd05791153e6f7
Reviewed-on: http://gerrit.cloudera.org:8080/14665
Reviewed-by: Attila Jeges
Tested-by: Impala Public Jenkins
> Use single quotes when printing out FORMAT clause within CAST.
> --
>
> Key: IMPALA-9131
> URL: https://issues.apache.org/jira/browse/IMPALA-9131
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 3.3.0
>Reporter: Gabor Kaszab
>Assignee: Gabor Kaszab
>Priority: Major
>
> Here the content of the FORMAT clause is surrounded by double quotes.
> {code:java}
> select cast('2016/10/10' as date format '/MM/DD');
> ++
> | cast('2016/10/10' as date format "/mm/dd") |
> ++
> | 2016-10-10 |
> ++
> {code}
> In order to follow SQL standards this should be surrounded by single quotes
> regardless of how the user gave the FORMAT clause.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-9207) Impala Doc: Document adding #Inst in exec summary
[ https://issues.apache.org/jira/browse/IMPALA-9207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexandra Rodoni updated IMPALA-9207: - Labels: future_release_doc in_34 (was: ) > Impala Doc: Document adding #Inst in exec summary > - > > Key: IMPALA-9207 > URL: https://issues.apache.org/jira/browse/IMPALA-9207 > Project: IMPALA > Issue Type: Sub-task >Reporter: Norbert Luksa >Assignee: Alexandra Rodoni >Priority: Major > Labels: future_release_doc, in_34 > > https://gerrit.cloudera.org/#/c/14715/ -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Closed] (IMPALA-9208) impala still failed to read orc format table using "enable_orc_scanner=true"
[ https://issues.apache.org/jira/browse/IMPALA-9208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Volker closed IMPALA-9208. --- Resolution: Not A Problem > impala still failed to read orc format table using "enable_orc_scanner=true" > > > Key: IMPALA-9208 > URL: https://issues.apache.org/jira/browse/IMPALA-9208 > Project: IMPALA > Issue Type: Bug > Components: Clients >Affects Versions: Impala 3.3.0 >Reporter: lv haiyang >Priority: Major > > [bigdata@sdw1 ~]$ impala-shell --var=enable_orc_scanner=true > Starting Impala Shell without Kerberos authentication > Opened TCP connection to sdw1:21000 > Connected to sdw1:21000 > Server version: impalad version 3.4.0-SNAPSHOT RELEASE (build > 190ad89ebd46161f63a520c2a6d75ad56701e145) > *** > Welcome to the Impala shell. > (Impala Shell v3.4.0-SNAPSHOT (190ad89) built on Tue Nov 19 08:52:31 UTC 2019) > The SET command shows the current value of all shell and query options. > *** > [sdw1:21000] default> use company; > Query: use company > [sdw1:21000] company> select * from orc_table; > Query: select * from orc_table > Query submitted at: 2019-12-02 11:53:41 (Coordinator: http://sdw1:25000) > ERROR: NotImplementedException: ORC scans are disabled by > --enable_orc_scanner flag -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IMPALA-9208) impala still failed to read orc format table using "enable_orc_scanner=true"
[ https://issues.apache.org/jira/browse/IMPALA-9208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987231#comment-16987231 ] Lars Volker commented on IMPALA-9208: - You need to pass the flag as a parameter to the impalad when starting your cluster. Please use u...@impala.apache.org for any issues with using Impala. > impala still failed to read orc format table using "enable_orc_scanner=true" > > > Key: IMPALA-9208 > URL: https://issues.apache.org/jira/browse/IMPALA-9208 > Project: IMPALA > Issue Type: Bug > Components: Clients >Affects Versions: Impala 3.3.0 >Reporter: lv haiyang >Priority: Major > > [bigdata@sdw1 ~]$ impala-shell --var=enable_orc_scanner=true > Starting Impala Shell without Kerberos authentication > Opened TCP connection to sdw1:21000 > Connected to sdw1:21000 > Server version: impalad version 3.4.0-SNAPSHOT RELEASE (build > 190ad89ebd46161f63a520c2a6d75ad56701e145) > *** > Welcome to the Impala shell. > (Impala Shell v3.4.0-SNAPSHOT (190ad89) built on Tue Nov 19 08:52:31 UTC 2019) > The SET command shows the current value of all shell and query options. > *** > [sdw1:21000] default> use company; > Query: use company > [sdw1:21000] company> select * from orc_table; > Query: select * from orc_table > Query submitted at: 2019-12-02 11:53:41 (Coordinator: http://sdw1:25000) > ERROR: NotImplementedException: ORC scans are disabled by > --enable_orc_scanner flag -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-9213) Client logs should indicate if a query has been retried
Sahil Takiar created IMPALA-9213: Summary: Client logs should indicate if a query has been retried Key: IMPALA-9213 URL: https://issues.apache.org/jira/browse/IMPALA-9213 Project: IMPALA Issue Type: Sub-task Reporter: Sahil Takiar The client logs should give some indication that a query has been retried and should print out information such as the new query id and the link to the retried query on the debug web UI. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (IMPALA-9209) TestRPCException.test_end_data_stream_error is flaky
[
https://issues.apache.org/jira/browse/IMPALA-9209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Tauber-Marshall reassigned IMPALA-9209:
--
Assignee: Thomas Tauber-Marshall
> TestRPCException.test_end_data_stream_error is flaky
>
>
> Key: IMPALA-9209
> URL: https://issues.apache.org/jira/browse/IMPALA-9209
> Project: IMPALA
> Issue Type: Bug
>Reporter: Sahil Takiar
>Assignee: Thomas Tauber-Marshall
>Priority: Major
>
> custom_cluster.test_rpc_exception.TestRPCException.test_end_data_stream_error
> (from pytest)
> Error Message
> {code}
> assert (not 'Debug Action: IMPALA_SERVICE_POOL:FAIL@0.5' or 8 == 0) + where
> 8 = >( PID: 5846
> (/data/jenkins/workspace/impala-cdh6.x-exhaustive/re..._SERVICE_POOL:127.0.0.1:27002:EndDataStream:FAIL@0.5
> --default_query_options=)>) +where TestRPCException._get_num_fails of object at 0x698c790>> = 0x698c790>._get_num_fails
> {code}
> Stacktrace
> {code}
> custom_cluster/test_rpc_exception.py:123: in test_end_data_stream_error
> self.execute_test_query("Debug Action: IMPALA_SERVICE_POOL:FAIL@0.5")
> custom_cluster/test_rpc_exception.py:75: in execute_test_query
> assert not exception_string or self._get_num_fails(impalad) == 0
> E assert (not 'Debug Action: IMPALA_SERVICE_POOL:FAIL@0.5' or 8 == 0)
> E+ where 8 = >( PID: 5846
> (/data/jenkins/workspace/impala-cdh6.x-exhaustive/re..._SERVICE_POOL:127.0.0.1:27002:EndDataStream:FAIL@0.5
> --default_query_options=)>)
> E+where > =
> ._get_num_fails
> {code}
> Standard Error
> {code}
> 23:14:02 MainThread: Found 0 impalad/0 statestored/0 catalogd process(es)
> 23:14:02 MainThread: Starting State Store logging to
> /data/jenkins/workspace/impala-cdh6.x-exhaustive/repos/Impala/logs/custom_cluster_tests/statestored.INFO
> 23:14:02 MainThread: Starting Catalog Service logging to
> /data/jenkins/workspace/impala-cdh6.x-exhaustive/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
> 23:14:02 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-cdh6.x-exhaustive/repos/Impala/logs/custom_cluster_tests/impalad.INFO
> 23:14:02 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-cdh6.x-exhaustive/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
> 23:14:02 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-cdh6.x-exhaustive/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
> 23:14:05 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 23:14:05 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 23:14:05 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-116e.vpc.cloudera.com:25000
> 23:14:05 MainThread: Waiting for num_known_live_backends=3. Current value: 0
> 23:14:06 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 23:14:06 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-116e.vpc.cloudera.com:25000
> 23:14:06 MainThread: Waiting for num_known_live_backends=3. Current value: 0
> 23:14:07 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 23:14:07 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-116e.vpc.cloudera.com:25000
> 23:14:07 MainThread: num_known_live_backends has reached value: 3
> 23:14:07 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 23:14:07 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-116e.vpc.cloudera.com:25001
> 23:14:07 MainThread: num_known_live_backends has reached value: 3
> 23:14:08 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 23:14:08 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-116e.vpc.cloudera.com:25002
> 23:14:08 MainThread: num_known_live_backends has reached value: 3
> 23:14:09 MainThread: Impala Cluster Running with 3 nodes (3 coordinators, 3
> executors).
> DEBUG:impala_cluster:Found 3 impalad/1 statestored/1 catalogd process(es)
> INFO:impala_service:Getting metric: statestore.live-backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-116e.vpc.cloudera.com:25010
> INFO:impala_service:Metric 'statestore.live-backends' has reached desired
> value: 4
> DEBUG:impala_service:Getting num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-116e.vpc.cloudera.com:25000
> INFO:impala_service:num_known_live_backends has reached value: 3
> DEBUG:impala_service:Getting num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-116e.vpc.cloudera.com:25001
> INFO:impala_service:num_known_live_backends has reached value: 3
> DEBUG:impala_service:Getting num_known_live_backends from
>
[jira] [Created] (IMPALA-9212) Impala authentication setup
Salman Isha created IMPALA-9212: --- Summary: Impala authentication setup Key: IMPALA-9212 URL: https://issues.apache.org/jira/browse/IMPALA-9212 Project: IMPALA Issue Type: Question Components: Security Environment: Centos 7.6 docker Reporter: Salman Isha Hi Guys, I was trying to setup authentication for impala shell using kerberos authentication. Note that I am using a docker container to host the impala. I also install and configure the KDC server on the same docker container. While using the kerberos setting in /etc/impala/conf all impala, statestore and catalog service gone down. i got below error from catalog.ERROR Log file created at: 2019/12/03 12:16:59 Running on machine: impala.test.com Log line format: [IWEF]mmdd hh:mm:ss.uu threadid file:line] msg E1203 12:16:59.708680 5640 logging.cc:121] stderr will be logged to this file. F1203 12:17:02.315798 5640 catalogd-main.cc:88] RPC Error: Client for localhost:24000 hit an unexpected exception: No more data to read., type: N6apache6thrift9transport19TTransportExceptionE, rpc: N6impala27TRegisterSubscriberResponseE, send: done . Impalad exiting. *** Check failure stack trace: *** @ 0x20eec9d google::LogMessage::Fail() @ 0x20f0542 google::LogMessage::SendToLog() @ 0x20ee677 google::LogMessage::Flush() @ 0x20f1c3e google::LogMessageFatal::~LogMessageFatal() @ 0x933358 CatalogdMain() @ 0x8ea6b7 main @ 0x7fb5617a23d5 __libc_start_main @ 0x932031 (unknown) Can anyone help me to solve this or setting up the authentication of impala shell ? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IMPALA-9212) Impala authentication setup
Salman Isha created IMPALA-9212: --- Summary: Impala authentication setup Key: IMPALA-9212 URL: https://issues.apache.org/jira/browse/IMPALA-9212 Project: IMPALA Issue Type: Question Components: Security Environment: Centos 7.6 docker Reporter: Salman Isha Hi Guys, I was trying to setup authentication for impala shell using kerberos authentication. Note that I am using a docker container to host the impala. I also install and configure the KDC server on the same docker container. While using the kerberos setting in /etc/impala/conf all impala, statestore and catalog service gone down. i got below error from catalog.ERROR Log file created at: 2019/12/03 12:16:59 Running on machine: impala.test.com Log line format: [IWEF]mmdd hh:mm:ss.uu threadid file:line] msg E1203 12:16:59.708680 5640 logging.cc:121] stderr will be logged to this file. F1203 12:17:02.315798 5640 catalogd-main.cc:88] RPC Error: Client for localhost:24000 hit an unexpected exception: No more data to read., type: N6apache6thrift9transport19TTransportExceptionE, rpc: N6impala27TRegisterSubscriberResponseE, send: done . Impalad exiting. *** Check failure stack trace: *** @ 0x20eec9d google::LogMessage::Fail() @ 0x20f0542 google::LogMessage::SendToLog() @ 0x20ee677 google::LogMessage::Flush() @ 0x20f1c3e google::LogMessageFatal::~LogMessageFatal() @ 0x933358 CatalogdMain() @ 0x8ea6b7 main @ 0x7fb5617a23d5 __libc_start_main @ 0x932031 (unknown) Can anyone help me to solve this or setting up the authentication of impala shell ? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
