[jira] [Resolved] (IMPALA-8073) SentryProxy.testAddCatalog() failed in private build because of socket error
[
https://issues.apache.org/jira/browse/IMPALA-8073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fredy Wijaya resolved IMPALA-8073.
--
Resolution: Fixed
> SentryProxy.testAddCatalog() failed in private build because of socket error
>
>
> Key: IMPALA-8073
> URL: https://issues.apache.org/jira/browse/IMPALA-8073
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Reporter: Tim Armstrong
>Assignee: Fredy Wijaya
>Priority: Critical
> Labels: flaky
> Attachments:
> FeSupport.impala-ec2-centos74-m5-4xlarge-ondemand-0eae.vpc.cloudera.com.jenkins.log.INFO.20190110-184543.10852
>
>
> {noformat}
> org.apache.impala.util.SentryProxyTest.testAddCatalog
> Failing for the past 1 build (Since Failed#4229 )
> Took 3 min 40 sec.
> add description
> Error Message
> Error initializing Catalog. Catalog may be empty.
> Stacktrace
> java.lang.IllegalStateException: Error initializing Catalog. Catalog may be
> empty.
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:1444)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getAllDatabases(HiveMetaStoreClient.java:1351)
> at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:150)
> at com.sun.proxy.$Proxy16.getAllDatabases(Unknown Source)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.reset(CatalogServiceCatalog.java:1181)
> at
> org.apache.impala.testutil.CatalogServiceTestCatalog.createWithAuth(CatalogServiceTestCatalog.java:59)
> at
> org.apache.impala.util.SentryProxyTest.withAllPrincipalTypes(SentryProxyTest.java:603)
> at
> org.apache.impala.util.SentryProxyTest.testAddCatalog(SentryProxyTest.java:140)
> {noformat}
> The error in the log (attached) appears to be a connection to the HMS error.
> {noformat}
> 0110 18:48:08.139935 10853 HiveMetaStoreClient.java:461] Trying to connect to
> metastore with URI thrift://localhost:9083
> I0110 18:48:08.140183 10853 HiveMetaStoreClient.java:535] Opened a connection
> to metastore, current connections: 797
> W0110 18:48:28.143384 10853 HiveMetaStoreClient.java:560] set_ugi() not
> successful, Likely cause: new client talking to old server. Continuing
> without it.
> Java exception follows:
> org.apache.thrift.transport.TTransportException: java.net.SocketException:
> Connection reset
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(ThriftHiveMetastore.java:4129)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_ugi(ThriftHiveMetastore.java:4115)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:552)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:299)
> at sun.reflect.GeneratedConstructorAccessor3.newInstance(Unknown
> Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1750)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:80)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:130)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:101)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:94)
> at
> org.apache.impala.catalog.MetaStoreClientPool$MetaStoreClient.(MetaStoreClientPool.java:93)
> at
> org.apache.impala.catalog.MetaStoreClientPool$MetaStoreClient.(MetaStoreClientPool.java:72)
> at
>
[jira] [Commented] (IMPALA-7928) Investigate consistent placement of remote scan ranges
[ https://issues.apache.org/jira/browse/IMPALA-7928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749342#comment-16749342 ] Alex Rodoni commented on IMPALA-7928: - [~joemcdonnell] Doc impact? > Investigate consistent placement of remote scan ranges > -- > > Key: IMPALA-7928 > URL: https://issues.apache.org/jira/browse/IMPALA-7928 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 3.2.0 >Reporter: Joe McDonnell >Assignee: Joe McDonnell >Priority: Critical > > With the file handle cache, it is useful for repeated scans of the same file > to go to the same node, as that node will already have a file handle cached. > When scheduling remote ranges, the scheduler introduces randomness that can > spread reads across all of the nodes. Repeated executions of queries on the > same set of files will not schedule the remote reads on the same nodes. This > causes a large amount of duplication across file handle caches on different > nodes. This reduces the efficiency of the cache significantly. > It may be useful for the scheduler to introduce some determinism in > scheduling remote reads to take advantage of the file handle cache. This is a > variation on the well-known tradeoff between skew and locality. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7265) Cache remote file handles
[ https://issues.apache.org/jira/browse/IMPALA-7265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749340#comment-16749340 ] Alex Rodoni commented on IMPALA-7265: - [~joemcdonnell] Is there a doc impact for this feature? > Cache remote file handles > - > > Key: IMPALA-7265 > URL: https://issues.apache.org/jira/browse/IMPALA-7265 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 3.1.0 >Reporter: Joe McDonnell >Assignee: Joe McDonnell >Priority: Critical > > The file handle cache currently does not allow caching remote file handles. > This means that clusters that have a lot of remote reads can suffer from > overloading the NameNode. Impala should be able to cache remote file handles. > There are some open questions about remote file handles and whether they > behave differently from local file handles. In particular: > # Is there any resource constraint on the number of remote file handles > open? (e.g. do they maintain a network connection?) > # Are there any semantic differences in how remote file handles behave when > files are deleted, overwritten, or appended? > # Are there any extra failure cases for remote file handles? (i.e. if a > machine goes down or a remote file handle is left open for an extended period > of time) > The form of caching will depend on the answers, but at the very least, it > should be possible to cache a remote file handle at the level of a query so > that a Parquet file with multiple columns can share file handles. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6169) Implement DATE type
[ https://issues.apache.org/jira/browse/IMPALA-6169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749337#comment-16749337 ] Alex Rodoni commented on IMPALA-6169: - [~attilaj] [~csringhofer] [~zi] What part of this epic will be in 3.2? > Implement DATE type > --- > > Key: IMPALA-6169 > URL: https://issues.apache.org/jira/browse/IMPALA-6169 > Project: IMPALA > Issue Type: New Feature > Components: Backend >Reporter: Quanlong Huang >Assignee: Attila Jeges >Priority: Major > > In Hive, the Date type describes a particular year/month/day, in the form > -MM-DD. > Hive has supported Date type in Parquet two years ago in Hive-1.2.0. (See > https://issues.apache.org/jira/browse/HIVE-8119 and > https://cwiki.apache.org/confluence/display/Hive/Parquet#Parquet-VersionsandLimitations.) > We should add support for Date type too. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8098) [Docs] Document incompatible changes to :shutdown command
[
https://issues.apache.org/jira/browse/IMPALA-8098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Rodoni updated IMPALA-8098:
Target Version: Impala 3.2.0
> [Docs] Document incompatible changes to :shutdown command
> -
>
> Key: IMPALA-8098
> URL: https://issues.apache.org/jira/browse/IMPALA-8098
> Project: IMPALA
> Issue Type: Task
>Reporter: Andrew Sherman
>Assignee: Alex Rodoni
>Priority: Major
> Labels: future_release_doc, in_32
>
> The :shutdown command is used to shutdown a remote server. The common case is
> that a user specifies the impalad to shutdown by specifying a host e.g.
> :shutdown('host100'). If a user has more than one impalad on a remote host
> then the form :shutdown(':') can be used to specify the port by
> which the imapald can be contacted. Prior to IMPALA-7985 this port was the
> backend port, e.g. :shutdown('host100:22000'). With IMPALA-7985 the port to
> use is the KRPC port , e.g. :shutdown('host100:27000').
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-3323) impala-shell --ldap_password_cmd has no config file equivalent
[
https://issues.apache.org/jira/browse/IMPALA-3323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749331#comment-16749331
]
Alex Rodoni commented on IMPALA-3323:
-
[~fredyw] Is this a user-facing feature / fix with a doc impact?
> impala-shell --ldap_password_cmd has no config file equivalent
> --
>
> Key: IMPALA-3323
> URL: https://issues.apache.org/jira/browse/IMPALA-3323
> Project: IMPALA
> Issue Type: Bug
> Components: Clients
>Affects Versions: Impala 2.5.0
>Reporter: John Russell
>Assignee: Fredy Wijaya
>Priority: Minor
> Labels: newbie
> Fix For: Impala 3.2.0
>
>
> Putting ldap_password_cmd in the .impalarc file results in a parse error when
> impala-shell starts up. Judging from the impala-shell source, it looks like a
> dest= parameter is missing so there's no config file equivalent for the
> --ldap_password_cmd command-line option. E.g. here's one with a config file
> equivalent and one without:
> {code}
> parser.add_option("--auth_creds_ok_in_clear", dest="creds_ok_in_clear",
> action="store_true", help="If set, LDAP authentication " +
> "may be used with an insecure connection to Impala. " +
> "WARNING: Authentication credentials will therefore be
> sent " +
> "unencrypted, and may be vulnerable to attack.")
> parser.add_option("--ldap_password_cmd",
> help="Shell command to run to retrieve the LDAP password")
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-3323) impala-shell --ldap_password_cmd has no config file equivalent
[
https://issues.apache.org/jira/browse/IMPALA-3323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fredy Wijaya resolved IMPALA-3323.
--
Resolution: Fixed
Fix Version/s: Impala 3.2.0
> impala-shell --ldap_password_cmd has no config file equivalent
> --
>
> Key: IMPALA-3323
> URL: https://issues.apache.org/jira/browse/IMPALA-3323
> Project: IMPALA
> Issue Type: Bug
> Components: Clients
>Affects Versions: Impala 2.5.0
>Reporter: John Russell
>Assignee: Fredy Wijaya
>Priority: Minor
> Labels: newbie
> Fix For: Impala 3.2.0
>
>
> Putting ldap_password_cmd in the .impalarc file results in a parse error when
> impala-shell starts up. Judging from the impala-shell source, it looks like a
> dest= parameter is missing so there's no config file equivalent for the
> --ldap_password_cmd command-line option. E.g. here's one with a config file
> equivalent and one without:
> {code}
> parser.add_option("--auth_creds_ok_in_clear", dest="creds_ok_in_clear",
> action="store_true", help="If set, LDAP authentication " +
> "may be used with an insecure connection to Impala. " +
> "WARNING: Authentication credentials will therefore be
> sent " +
> "unencrypted, and may be vulnerable to attack.")
> parser.add_option("--ldap_password_cmd",
> help="Shell command to run to retrieve the LDAP password")
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8098) [Docs] Document incompatible changes to :shutdown command
[
https://issues.apache.org/jira/browse/IMPALA-8098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Rodoni updated IMPALA-8098:
Labels: future_release_doc in_32 (was: future_release_doc)
> [Docs] Document incompatible changes to :shutdown command
> -
>
> Key: IMPALA-8098
> URL: https://issues.apache.org/jira/browse/IMPALA-8098
> Project: IMPALA
> Issue Type: Task
>Reporter: Andrew Sherman
>Assignee: Alex Rodoni
>Priority: Major
> Labels: future_release_doc, in_32
>
> The :shutdown command is used to shutdown a remote server. The common case is
> that a user specifies the impalad to shutdown by specifying a host e.g.
> :shutdown('host100'). If a user has more than one impalad on a remote host
> then the form :shutdown(':') can be used to specify the port by
> which the imapald can be contacted. Prior to IMPALA-7985 this port was the
> backend port, e.g. :shutdown('host100:22000'). With IMPALA-7985 the port to
> use is the KRPC port , e.g. :shutdown('host100:27000').
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8098) [Docs] Document incompatible changes to :shutdown command
Andrew Sherman created IMPALA-8098:
--
Summary: [Docs] Document incompatible changes to :shutdown command
Key: IMPALA-8098
URL: https://issues.apache.org/jira/browse/IMPALA-8098
Project: IMPALA
Issue Type: Task
Reporter: Andrew Sherman
Assignee: Alex Rodoni
The :shutdown command is used to shutdown a remote server. The common case is
that a user specifies the impalad to shutdown by specifying a host e.g.
:shutdown('host100'). If a user has more than one impalad on a remote host then
the form :shutdown(':') can be used to specify the port by which
the imapald can be contacted. Prior to IMPALA-7985 this port was the backend
port, e.g. :shutdown('host100:22000'). With IMPALA-7985 the port to use is the
KRPC port , e.g. :shutdown('host100:27000').
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8027) KRPC datastream timing out on both the receiver and sender side even in a minicluster
[
https://issues.apache.org/jira/browse/IMPALA-8027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749308#comment-16749308
]
Michael Ho commented on IMPALA-8027:
The profile didn't suggest much either. The last report time for the
coordinator fragment was 05:16:55, which was when the profile was initially
created. So, the coordinator itself didn't receive any update from the
coordinator fragment itself for two minutes. This suggests the Prepare() didn't
complete within the time frame.
{noformat}
Instance 194f5b70907ac97c:84a116d6
(host=impala-ec2-centos74-m5-4xlarge-ondemand-07b3.vpc.cloudera.com:22000)
Last report received time: 2018-12-28 05:16:52.214
{noformat}
FWIW, other fragment instances running on other Impalad peers were able to send
report so it seems the status reporting handling was working on coordinator
({{impala-ec2-centos74-m5-4xlarge-ondemand-07b3.vpc.cloudera.com:22000}}):
{noformat}
Instance 194f5b70907ac97c:84a116d60003
(host=impala-ec2-centos74-m5-4xlarge-ondemand-07b3.vpc.cloudera.com:22001)
Last report received time: 2018-12-28 05:18:56.204
Hdfs split stats (:<# splits>/): -1:4/152.76 KB
{noformat}
The fragment instances running on
{{impala-ec2-centos74-m5-4xlarge-ondemand-07b3.vpc.cloudera.com:22000}}
eventually completed.
{noformat}
I1228 05:16:52.217275 26685 query-state.cc:568] Executing instance.
instance_id=194f5b70907ac97c:84a116d6 fragment_idx=0
per_fragment_instance_idx=0 coord_state_idx=0 #in-flight=7
I1228 05:20:08.981983 26685 krpc-data-stream-mgr.cc:294] DeregisterRecvr():
fragment_instance_id=194f5b70907ac97c:84a116d6, node=2
I1228 05:20:08.982043 26685 query-state.cc:576] Instance completed.
instance_id=194f5b70907ac97c:84a116d6 #in-flight=5 status=CANCELLED:
Cancelled
{noformat}
{noformat}
I1228 05:16:52.217344 26686 query-state.cc:568] Executing instance.
instance_id=194f5b70907ac97c:84a116d60001 fragment_idx=1
per_fragment_instance_idx=0 coord_state_idx=0 #in-flight=8
I1228 05:20:08.992434 26686 llvm-codegen.cc:1104] For query
194f5b70907ac97c:84a116d6 the following functions were not finalized
and have been removed from the module:
I1228 05:20:09.049794 26686 query-state.cc:576] Instance completed.
instance_id=194f5b70907ac97c:84a116d60001 #in-flight=5 status=CANCELLED:
Cancelled
I1228 05:20:09.056677 26686 query-exec-mgr.cc:182] ReleaseQueryState(): deleted
query_id=194f5b70907ac97c:84a116d6
{noformat}
With the data available so far, one could only conclude that the test failed
due to some flaky scheduling issue which slowed down some query fragments
running on
{{impala-ec2-centos74-m5-4xlarge-ondemand-07b3.vpc.cloudera.com:22000}}. Given
this hasn't happened again since the initial reporting, I don't think there is
much we need to do at this point.
Longer term, fixing IMPALA-6618 will remove this class of flaky timeout issues.
> KRPC datastream timing out on both the receiver and sender side even in a
> minicluster
> -
>
> Key: IMPALA-8027
> URL: https://issues.apache.org/jira/browse/IMPALA-8027
> Project: IMPALA
> Issue Type: Bug
> Components: Distributed Exec
>Affects Versions: Impala 3.2.0
>Reporter: Bikramjeet Vig
>Assignee: Michael Ho
>Priority: Critical
> Labels: broken-build
>
> krpc datastreams seem to time out at the same time at both sender and
> receiver causing two running queries to fail. This happened while running
> core tests on s3.
> Logs from coordinator:
> {noformat}
> I1228 05:18:56.202587 13396 krpc-data-stream-mgr.cc:353] Sender 127.0.0.1
> timed out waiting for receiver fragment instance:
> 8f46b2518734bef1:6ef2d404, dest node: 2
> I1228 05:18:56.203061 13396 rpcz_store.cc:265] Call
> impala.DataStreamService.TransmitData from 127.0.0.1:53118 (request call id
> 11274) took 120782ms. Request Metrics: {}
> I1228 05:18:56.203114 13396 krpc-data-stream-mgr.cc:353] Sender 127.0.0.1
> timed out waiting for receiver fragment instance:
> 194f5b70907ac97c:84a116d6, dest node: 2
> I1228 05:18:56.203136 13396 rpcz_store.cc:265] Call
> impala.DataStreamService.TransmitData from 127.0.0.1:53110 (request call id
> 8637) took 123811ms. Request Metrics: {}
> I1228 05:18:56.203155 13396 krpc-data-stream-mgr.cc:353] Sender 127.0.0.1
> timed out waiting for receiver fragment instance:
> 194f5b70907ac97c:84a116d6, dest node: 2
> I1228 05:18:56.203167 13396 rpcz_store.cc:265] Call
> impala.DataStreamService.TransmitData from 127.0.0.1:53118 (request call id
> 11273) took 123776ms. Request Metrics: {}
> I1228 05:18:56.203181 13396 krpc-data-stream-mgr.cc:408] Reduced stream ID
> cache from 413 items, to 410, eviction took: 1ms
> I1228 05:18:56.204746 13377
[jira] [Work started] (IMPALA-8091) minicluster kudu failure: Cannot initialize clock: failed to wait for clock sync
[
https://issues.apache.org/jira/browse/IMPALA-8091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-8091 started by Michael Brown.
-
> minicluster kudu failure: Cannot initialize clock: failed to wait for clock
> sync
>
>
> Key: IMPALA-8091
> URL: https://issues.apache.org/jira/browse/IMPALA-8091
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.2.0
>Reporter: David Knupp
>Assignee: Michael Brown
>Priority: Blocker
> Labels: broken-build
>
> *Console log*:
> {noformat}
> 19:10:04 19:10:04 Error executing impala SQL:
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/logs/data_loading/sql/functional/create-functional-query-exhaustive-impala-generated-kudu-none-none.sql
> See:
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/logs/data_loading/sql/functional/create-functional-query-exhaustive-impala-generated-kudu-none-none.sql.log
> 19:10:04 Background task Loading functional-query data (pid 4831) failed.
> 19:10:04 Background task Loading TPC-H data (pid 4832) failed.
> 19:10:04 ERROR in
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/testdata/bin/create-load-data.sh
> at line 85: CM_HOST=${2-}
> [...]
> 19:10:05 2019-01-16 19:10:05,461 - archive_core_dumps - INFO - Found core
> files: ['./core.1547693849.30799.kudu-master',
> './core.1547693849.30783.kudu-tserver',
> './core.1547693849.30767.kudu-tserver',
> './core.1547693849.30808.kudu-tserver']
> 19:10:05 2019-01-16 19:10:05,649 - archive_core_dumps - INFO - [New LWP 30799]
> 19:10:05 [New LWP 30834]
> 19:10:05 [New LWP 30835]
> 19:10:05 [New LWP 30838]
> 19:10:05 [New LWP 30837]
> 19:10:05 [New LWP 30836]
> 19:10:05 Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> 19:10:05 Program terminated with signal SIGABRT, Aborted.
> 19:10:05 #0 0x7f9a2cc611f7 in ?? ()
> 19:10:05
> 19:10:05 2019-01-16 19:10:05,650 - archive_core_dumps - INFO - Found binary
> path through GDB:
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c
> 19:10:05 2019-01-16 19:10:05,893 - archive_core_dumps - WARNING - Failed to
> determine binary because multiple candidate binaries were found and none of
> their paths contained 'latest' to disambiguate:
> 19:10:05 Core:./core.1547693849.30799.kudu-master
> 19:10:05
> Binaries:['./testdata/cluster/node_templates/common/etc/init.d/kudu-master',
> './testdata/cluster/cdh6/node-1/etc/init.d/kudu-master']
> 19:10:05
> 19:10:05 2019-01-16 19:10:05,917 - archive_core_dumps - INFO - [New LWP 30783]
> 19:10:05 [New LWP 30810]
> 19:10:05 [New LWP 30812]
> 19:10:05 [New LWP 30811]
> 19:10:05 [New LWP 30824]
> 19:10:05 [New LWP 30820]
> 19:10:05 Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> 19:10:05 Program terminated with signal SIGABRT, Aborted.
> 19:10:05 #0 0x7f0b81fb11f7 in ?? ()
> {noformat}
> *Backtraces*:
> {noformat}
> CORE: ./core.1547693849.30799.kudu-master
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x7f9a2cc611f7 in ?? ()
> #0 0x7f9a2cc611f7 in ?? ()
> #1 0x7f9a2cc628e8 in ?? ()
> #2 0x0020 in ?? ()
> #3 0x in ?? ()
> CORE: ./core.1547693849.30783.kudu-tserver
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x7f0b81fb11f7 in ?? ()
> #0 0x7f0b81fb11f7 in ?? ()
> #1 0x7f0b81fb28e8 in ?? ()
> #2 0x0020 in ?? ()
> #3 0x in ?? ()
> CORE: ./core.1547693849.30767.kudu-tserver
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x7f60b1e2f1f7 in ?? ()
> #0 0x7f60b1e2f1f7 in ?? ()
> #1 0x7f60b1e308e8 in ?? ()
> #2 0x0020 in ?? ()
> #3 0x in ?? ()
> CORE: ./core.1547693849.30808.kudu-tserver
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x7fa3cb5591f7 in ?? ()
> #0 0x7fa3cb5591f7 in ?? ()
> #1 0x7fa3cb55a8e8 in ?? ()
> #2 0x0020 in ?? ()
> #3 0x in ??
[jira] [Assigned] (IMPALA-8091) minicluster kudu failure: Cannot initialize clock: failed to wait for clock sync
[
https://issues.apache.org/jira/browse/IMPALA-8091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Brown reassigned IMPALA-8091:
-
Assignee: Michael Brown (was: Thomas Tauber-Marshall)
> minicluster kudu failure: Cannot initialize clock: failed to wait for clock
> sync
>
>
> Key: IMPALA-8091
> URL: https://issues.apache.org/jira/browse/IMPALA-8091
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.2.0
>Reporter: David Knupp
>Assignee: Michael Brown
>Priority: Blocker
> Labels: broken-build
>
> *Console log*:
> {noformat}
> 19:10:04 19:10:04 Error executing impala SQL:
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/logs/data_loading/sql/functional/create-functional-query-exhaustive-impala-generated-kudu-none-none.sql
> See:
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/logs/data_loading/sql/functional/create-functional-query-exhaustive-impala-generated-kudu-none-none.sql.log
> 19:10:04 Background task Loading functional-query data (pid 4831) failed.
> 19:10:04 Background task Loading TPC-H data (pid 4832) failed.
> 19:10:04 ERROR in
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/testdata/bin/create-load-data.sh
> at line 85: CM_HOST=${2-}
> [...]
> 19:10:05 2019-01-16 19:10:05,461 - archive_core_dumps - INFO - Found core
> files: ['./core.1547693849.30799.kudu-master',
> './core.1547693849.30783.kudu-tserver',
> './core.1547693849.30767.kudu-tserver',
> './core.1547693849.30808.kudu-tserver']
> 19:10:05 2019-01-16 19:10:05,649 - archive_core_dumps - INFO - [New LWP 30799]
> 19:10:05 [New LWP 30834]
> 19:10:05 [New LWP 30835]
> 19:10:05 [New LWP 30838]
> 19:10:05 [New LWP 30837]
> 19:10:05 [New LWP 30836]
> 19:10:05 Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> 19:10:05 Program terminated with signal SIGABRT, Aborted.
> 19:10:05 #0 0x7f9a2cc611f7 in ?? ()
> 19:10:05
> 19:10:05 2019-01-16 19:10:05,650 - archive_core_dumps - INFO - Found binary
> path through GDB:
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c
> 19:10:05 2019-01-16 19:10:05,893 - archive_core_dumps - WARNING - Failed to
> determine binary because multiple candidate binaries were found and none of
> their paths contained 'latest' to disambiguate:
> 19:10:05 Core:./core.1547693849.30799.kudu-master
> 19:10:05
> Binaries:['./testdata/cluster/node_templates/common/etc/init.d/kudu-master',
> './testdata/cluster/cdh6/node-1/etc/init.d/kudu-master']
> 19:10:05
> 19:10:05 2019-01-16 19:10:05,917 - archive_core_dumps - INFO - [New LWP 30783]
> 19:10:05 [New LWP 30810]
> 19:10:05 [New LWP 30812]
> 19:10:05 [New LWP 30811]
> 19:10:05 [New LWP 30824]
> 19:10:05 [New LWP 30820]
> 19:10:05 Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> 19:10:05 Program terminated with signal SIGABRT, Aborted.
> 19:10:05 #0 0x7f0b81fb11f7 in ?? ()
> {noformat}
> *Backtraces*:
> {noformat}
> CORE: ./core.1547693849.30799.kudu-master
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x7f9a2cc611f7 in ?? ()
> #0 0x7f9a2cc611f7 in ?? ()
> #1 0x7f9a2cc628e8 in ?? ()
> #2 0x0020 in ?? ()
> #3 0x in ?? ()
> CORE: ./core.1547693849.30783.kudu-tserver
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x7f0b81fb11f7 in ?? ()
> #0 0x7f0b81fb11f7 in ?? ()
> #1 0x7f0b81fb28e8 in ?? ()
> #2 0x0020 in ?? ()
> #3 0x in ?? ()
> CORE: ./core.1547693849.30767.kudu-tserver
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x7f60b1e2f1f7 in ?? ()
> #0 0x7f60b1e2f1f7 in ?? ()
> #1 0x7f60b1e308e8 in ?? ()
> #2 0x0020 in ?? ()
> #3 0x in ?? ()
> CORE: ./core.1547693849.30808.kudu-tserver
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x7fa3cb5591f7 in ?? ()
> #0 0x7fa3cb5591f7 in ?? ()
> #1 0x7fa3cb55a8e8 in ?? ()
> #2
[jira] [Work started] (IMPALA-7928) Investigate consistent placement of remote scan ranges
[ https://issues.apache.org/jira/browse/IMPALA-7928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-7928 started by Joe McDonnell. - > Investigate consistent placement of remote scan ranges > -- > > Key: IMPALA-7928 > URL: https://issues.apache.org/jira/browse/IMPALA-7928 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 3.2.0 >Reporter: Joe McDonnell >Assignee: Joe McDonnell >Priority: Critical > > With the file handle cache, it is useful for repeated scans of the same file > to go to the same node, as that node will already have a file handle cached. > When scheduling remote ranges, the scheduler introduces randomness that can > spread reads across all of the nodes. Repeated executions of queries on the > same set of files will not schedule the remote reads on the same nodes. This > causes a large amount of duplication across file handle caches on different > nodes. This reduces the efficiency of the cache significantly. > It may be useful for the scheduler to introduce some determinism in > scheduling remote reads to take advantage of the file handle cache. This is a > variation on the well-known tradeoff between skew and locality. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8097) Experimental flag for running all queries with mt_dop
Tim Armstrong created IMPALA-8097: - Summary: Experimental flag for running all queries with mt_dop Key: IMPALA-8097 URL: https://issues.apache.org/jira/browse/IMPALA-8097 Project: IMPALA Issue Type: Sub-task Reporter: Tim Armstrong Assignee: Tim Armstrong It's possible to execute queries with joins and inserts with mt_dop with some small modifications to the Impala code (without the separate join build work). This isn't production-ready because of the other subtasks of IMPALA-3902 but it would be useful to have an experimental flag for people to play around with to test the functionality. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8091) minicluster kudu failure: Cannot initialize clock: failed to wait for clock sync
[
https://issues.apache.org/jira/browse/IMPALA-8091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749232#comment-16749232
]
Michael Brown commented on IMPALA-8091:
---
Relevant code in {{testdata/cluster/admin}}:
{noformat}
# line 333
if [[ "${SERVICE}" == "kudu" ]]; then
NTP_CANARY=pool.ntp.org
if ! ping -c 1 -w 5 "${NTP_CANARY}" >/dev/null 2>/dev/null; then
echo "WARNING: cannot reach ${NTP_CANARY}; ntp sync recommended for
Kudu"
elif ntp-wait --help >/dev/null 2>/dev/null && ! ntp-wait -v; then
echo "ntp-wait failed; cannot start kudu"
return 1
fi
fi
{noformat}
We silently move on if ntp-wait isn't present. We should warn. We don't try to
install ntp-wait, which I get, because it's rude to do that to somebody's
system. We should install ntp-perl for CentOS 7 in bootstrap_system.sh.
> minicluster kudu failure: Cannot initialize clock: failed to wait for clock
> sync
>
>
> Key: IMPALA-8091
> URL: https://issues.apache.org/jira/browse/IMPALA-8091
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.2.0
>Reporter: David Knupp
>Assignee: Thomas Tauber-Marshall
>Priority: Blocker
> Labels: broken-build
>
> *Console log*:
> {noformat}
> 19:10:04 19:10:04 Error executing impala SQL:
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/logs/data_loading/sql/functional/create-functional-query-exhaustive-impala-generated-kudu-none-none.sql
> See:
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/logs/data_loading/sql/functional/create-functional-query-exhaustive-impala-generated-kudu-none-none.sql.log
> 19:10:04 Background task Loading functional-query data (pid 4831) failed.
> 19:10:04 Background task Loading TPC-H data (pid 4832) failed.
> 19:10:04 ERROR in
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/testdata/bin/create-load-data.sh
> at line 85: CM_HOST=${2-}
> [...]
> 19:10:05 2019-01-16 19:10:05,461 - archive_core_dumps - INFO - Found core
> files: ['./core.1547693849.30799.kudu-master',
> './core.1547693849.30783.kudu-tserver',
> './core.1547693849.30767.kudu-tserver',
> './core.1547693849.30808.kudu-tserver']
> 19:10:05 2019-01-16 19:10:05,649 - archive_core_dumps - INFO - [New LWP 30799]
> 19:10:05 [New LWP 30834]
> 19:10:05 [New LWP 30835]
> 19:10:05 [New LWP 30838]
> 19:10:05 [New LWP 30837]
> 19:10:05 [New LWP 30836]
> 19:10:05 Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> 19:10:05 Program terminated with signal SIGABRT, Aborted.
> 19:10:05 #0 0x7f9a2cc611f7 in ?? ()
> 19:10:05
> 19:10:05 2019-01-16 19:10:05,650 - archive_core_dumps - INFO - Found binary
> path through GDB:
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c
> 19:10:05 2019-01-16 19:10:05,893 - archive_core_dumps - WARNING - Failed to
> determine binary because multiple candidate binaries were found and none of
> their paths contained 'latest' to disambiguate:
> 19:10:05 Core:./core.1547693849.30799.kudu-master
> 19:10:05
> Binaries:['./testdata/cluster/node_templates/common/etc/init.d/kudu-master',
> './testdata/cluster/cdh6/node-1/etc/init.d/kudu-master']
> 19:10:05
> 19:10:05 2019-01-16 19:10:05,917 - archive_core_dumps - INFO - [New LWP 30783]
> 19:10:05 [New LWP 30810]
> 19:10:05 [New LWP 30812]
> 19:10:05 [New LWP 30811]
> 19:10:05 [New LWP 30824]
> 19:10:05 [New LWP 30820]
> 19:10:05 Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> 19:10:05 Program terminated with signal SIGABRT, Aborted.
> 19:10:05 #0 0x7f0b81fb11f7 in ?? ()
> {noformat}
> *Backtraces*:
> {noformat}
> CORE: ./core.1547693849.30799.kudu-master
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x7f9a2cc611f7 in ?? ()
> #0 0x7f9a2cc611f7 in ?? ()
> #1 0x7f9a2cc628e8 in ?? ()
> #2 0x0020 in ?? ()
> #3 0x in ?? ()
> CORE: ./core.1547693849.30783.kudu-tserver
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x7f0b81fb11f7 in ?? ()
> #0 0x7f0b81fb11f7 in ?? ()
> #1 0x7f0b81fb28e8 in ?? ()
> #2 0x0020 in ?? ()
> #3 0x in ?? ()
> CORE: ./core.1547693849.30767.kudu-tserver
> BINARY: ./be/build/latest/service/impalad
> Core
[jira] [Commented] (IMPALA-8091) minicluster kudu failure: Cannot initialize clock: failed to wait for clock sync
[
https://issues.apache.org/jira/browse/IMPALA-8091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749190#comment-16749190
]
Michael Brown commented on IMPALA-8091:
---
On Ubuntu 16, the {{ntp}} package provides {{ntp-wait}}. On CentOS 7, it's
provided by {{ntp-perl}}. This is happening in a downstream environment where
{{ntp-perl}} is not installed. On the same downstream environment, {{chrony}}
seems to be installed but isn't running a daemon (ok). I'm looking to see if
there is an upstream improvement that can be made here.
> minicluster kudu failure: Cannot initialize clock: failed to wait for clock
> sync
>
>
> Key: IMPALA-8091
> URL: https://issues.apache.org/jira/browse/IMPALA-8091
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.2.0
>Reporter: David Knupp
>Assignee: Thomas Tauber-Marshall
>Priority: Blocker
> Labels: broken-build
>
> *Console log*:
> {noformat}
> 19:10:04 19:10:04 Error executing impala SQL:
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/logs/data_loading/sql/functional/create-functional-query-exhaustive-impala-generated-kudu-none-none.sql
> See:
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/logs/data_loading/sql/functional/create-functional-query-exhaustive-impala-generated-kudu-none-none.sql.log
> 19:10:04 Background task Loading functional-query data (pid 4831) failed.
> 19:10:04 Background task Loading TPC-H data (pid 4832) failed.
> 19:10:04 ERROR in
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/testdata/bin/create-load-data.sh
> at line 85: CM_HOST=${2-}
> [...]
> 19:10:05 2019-01-16 19:10:05,461 - archive_core_dumps - INFO - Found core
> files: ['./core.1547693849.30799.kudu-master',
> './core.1547693849.30783.kudu-tserver',
> './core.1547693849.30767.kudu-tserver',
> './core.1547693849.30808.kudu-tserver']
> 19:10:05 2019-01-16 19:10:05,649 - archive_core_dumps - INFO - [New LWP 30799]
> 19:10:05 [New LWP 30834]
> 19:10:05 [New LWP 30835]
> 19:10:05 [New LWP 30838]
> 19:10:05 [New LWP 30837]
> 19:10:05 [New LWP 30836]
> 19:10:05 Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> 19:10:05 Program terminated with signal SIGABRT, Aborted.
> 19:10:05 #0 0x7f9a2cc611f7 in ?? ()
> 19:10:05
> 19:10:05 2019-01-16 19:10:05,650 - archive_core_dumps - INFO - Found binary
> path through GDB:
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c
> 19:10:05 2019-01-16 19:10:05,893 - archive_core_dumps - WARNING - Failed to
> determine binary because multiple candidate binaries were found and none of
> their paths contained 'latest' to disambiguate:
> 19:10:05 Core:./core.1547693849.30799.kudu-master
> 19:10:05
> Binaries:['./testdata/cluster/node_templates/common/etc/init.d/kudu-master',
> './testdata/cluster/cdh6/node-1/etc/init.d/kudu-master']
> 19:10:05
> 19:10:05 2019-01-16 19:10:05,917 - archive_core_dumps - INFO - [New LWP 30783]
> 19:10:05 [New LWP 30810]
> 19:10:05 [New LWP 30812]
> 19:10:05 [New LWP 30811]
> 19:10:05 [New LWP 30824]
> 19:10:05 [New LWP 30820]
> 19:10:05 Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> 19:10:05 Program terminated with signal SIGABRT, Aborted.
> 19:10:05 #0 0x7f0b81fb11f7 in ?? ()
> {noformat}
> *Backtraces*:
> {noformat}
> CORE: ./core.1547693849.30799.kudu-master
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x7f9a2cc611f7 in ?? ()
> #0 0x7f9a2cc611f7 in ?? ()
> #1 0x7f9a2cc628e8 in ?? ()
> #2 0x0020 in ?? ()
> #3 0x in ?? ()
> CORE: ./core.1547693849.30783.kudu-tserver
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x7f0b81fb11f7 in ?? ()
> #0 0x7f0b81fb11f7 in ?? ()
> #1 0x7f0b81fb28e8 in ?? ()
> #2 0x0020 in ?? ()
> #3 0x in ?? ()
> CORE: ./core.1547693849.30767.kudu-tserver
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x7f60b1e2f1f7 in ?? ()
> #0 0x7f60b1e2f1f7 in ?? ()
> #1 0x7f60b1e308e8 in ?? ()
> #2 0x0020 in ?? ()
> #3 0x in ?? ()
> CORE:
[jira] [Work started] (IMPALA-7934) Switch to using Java 8's Base64 impl for incremental stats encoding
[ https://issues.apache.org/jira/browse/IMPALA-7934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-7934 started by Fredy Wijaya. > Switch to using Java 8's Base64 impl for incremental stats encoding > --- > > Key: IMPALA-7934 > URL: https://issues.apache.org/jira/browse/IMPALA-7934 > Project: IMPALA > Issue Type: Bug > Components: Catalog >Affects Versions: Impala 3.1.0 >Reporter: bharath v >Assignee: Fredy Wijaya >Priority: Major > Labels: ramp-up > Attachments: base64.png > > > Incremental stats are compressed and Base64 encoded before they are chunked > and written to the HMS' partition parameters map. When they are read back, we > need to Base64 decode and decompress. > For certain incremental stats heavy tables, we noticed that a significant > amount of time is spent in these base64 classes (see the attached image for > the stack. Unfortunately, I don't have the text version of it). > Java 8 comes with its own Base64 implementation and that has shown much > better perf results [1] compared to apache codec's impl. So consider > switching to Java 8's base64 impl. > [1] http://java-performance.info/base64-encoding-and-decoding-performance/ > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8091) minicluster kudu failure: Cannot initialize clock: failed to wait for clock sync
[
https://issues.apache.org/jira/browse/IMPALA-8091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749176#comment-16749176
]
Michael Brown commented on IMPALA-8091:
---
This affected numerous runs over a quiet weekend without much commit churn.
> minicluster kudu failure: Cannot initialize clock: failed to wait for clock
> sync
>
>
> Key: IMPALA-8091
> URL: https://issues.apache.org/jira/browse/IMPALA-8091
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.2.0
>Reporter: David Knupp
>Assignee: Thomas Tauber-Marshall
>Priority: Blocker
> Labels: broken-build
>
> *Console log*:
> {noformat}
> 19:10:04 19:10:04 Error executing impala SQL:
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/logs/data_loading/sql/functional/create-functional-query-exhaustive-impala-generated-kudu-none-none.sql
> See:
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/logs/data_loading/sql/functional/create-functional-query-exhaustive-impala-generated-kudu-none-none.sql.log
> 19:10:04 Background task Loading functional-query data (pid 4831) failed.
> 19:10:04 Background task Loading TPC-H data (pid 4832) failed.
> 19:10:04 ERROR in
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/testdata/bin/create-load-data.sh
> at line 85: CM_HOST=${2-}
> [...]
> 19:10:05 2019-01-16 19:10:05,461 - archive_core_dumps - INFO - Found core
> files: ['./core.1547693849.30799.kudu-master',
> './core.1547693849.30783.kudu-tserver',
> './core.1547693849.30767.kudu-tserver',
> './core.1547693849.30808.kudu-tserver']
> 19:10:05 2019-01-16 19:10:05,649 - archive_core_dumps - INFO - [New LWP 30799]
> 19:10:05 [New LWP 30834]
> 19:10:05 [New LWP 30835]
> 19:10:05 [New LWP 30838]
> 19:10:05 [New LWP 30837]
> 19:10:05 [New LWP 30836]
> 19:10:05 Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> 19:10:05 Program terminated with signal SIGABRT, Aborted.
> 19:10:05 #0 0x7f9a2cc611f7 in ?? ()
> 19:10:05
> 19:10:05 2019-01-16 19:10:05,650 - archive_core_dumps - INFO - Found binary
> path through GDB:
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c
> 19:10:05 2019-01-16 19:10:05,893 - archive_core_dumps - WARNING - Failed to
> determine binary because multiple candidate binaries were found and none of
> their paths contained 'latest' to disambiguate:
> 19:10:05 Core:./core.1547693849.30799.kudu-master
> 19:10:05
> Binaries:['./testdata/cluster/node_templates/common/etc/init.d/kudu-master',
> './testdata/cluster/cdh6/node-1/etc/init.d/kudu-master']
> 19:10:05
> 19:10:05 2019-01-16 19:10:05,917 - archive_core_dumps - INFO - [New LWP 30783]
> 19:10:05 [New LWP 30810]
> 19:10:05 [New LWP 30812]
> 19:10:05 [New LWP 30811]
> 19:10:05 [New LWP 30824]
> 19:10:05 [New LWP 30820]
> 19:10:05 Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> 19:10:05 Program terminated with signal SIGABRT, Aborted.
> 19:10:05 #0 0x7f0b81fb11f7 in ?? ()
> {noformat}
> *Backtraces*:
> {noformat}
> CORE: ./core.1547693849.30799.kudu-master
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x7f9a2cc611f7 in ?? ()
> #0 0x7f9a2cc611f7 in ?? ()
> #1 0x7f9a2cc628e8 in ?? ()
> #2 0x0020 in ?? ()
> #3 0x in ?? ()
> CORE: ./core.1547693849.30783.kudu-tserver
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x7f0b81fb11f7 in ?? ()
> #0 0x7f0b81fb11f7 in ?? ()
> #1 0x7f0b81fb28e8 in ?? ()
> #2 0x0020 in ?? ()
> #3 0x in ?? ()
> CORE: ./core.1547693849.30767.kudu-tserver
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x7f60b1e2f1f7 in ?? ()
> #0 0x7f60b1e2f1f7 in ?? ()
> #1 0x7f60b1e308e8 in ?? ()
> #2 0x0020 in ?? ()
> #3 0x in ?? ()
> CORE: ./core.1547693849.30808.kudu-tserver
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x7fa3cb5591f7 in ?? ()
> #0 0x7fa3cb5591f7 in
[jira] [Updated] (IMPALA-8091) minicluster kudu failure: Cannot initialize clock: failed to wait for clock sync
[
https://issues.apache.org/jira/browse/IMPALA-8091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Brown updated IMPALA-8091:
--
Priority: Blocker (was: Critical)
> minicluster kudu failure: Cannot initialize clock: failed to wait for clock
> sync
>
>
> Key: IMPALA-8091
> URL: https://issues.apache.org/jira/browse/IMPALA-8091
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.2.0
>Reporter: David Knupp
>Assignee: Thomas Tauber-Marshall
>Priority: Blocker
> Labels: broken-build
>
> *Console log*:
> {noformat}
> 19:10:04 19:10:04 Error executing impala SQL:
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/logs/data_loading/sql/functional/create-functional-query-exhaustive-impala-generated-kudu-none-none.sql
> See:
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/logs/data_loading/sql/functional/create-functional-query-exhaustive-impala-generated-kudu-none-none.sql.log
> 19:10:04 Background task Loading functional-query data (pid 4831) failed.
> 19:10:04 Background task Loading TPC-H data (pid 4832) failed.
> 19:10:04 ERROR in
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/testdata/bin/create-load-data.sh
> at line 85: CM_HOST=${2-}
> [...]
> 19:10:05 2019-01-16 19:10:05,461 - archive_core_dumps - INFO - Found core
> files: ['./core.1547693849.30799.kudu-master',
> './core.1547693849.30783.kudu-tserver',
> './core.1547693849.30767.kudu-tserver',
> './core.1547693849.30808.kudu-tserver']
> 19:10:05 2019-01-16 19:10:05,649 - archive_core_dumps - INFO - [New LWP 30799]
> 19:10:05 [New LWP 30834]
> 19:10:05 [New LWP 30835]
> 19:10:05 [New LWP 30838]
> 19:10:05 [New LWP 30837]
> 19:10:05 [New LWP 30836]
> 19:10:05 Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> 19:10:05 Program terminated with signal SIGABRT, Aborted.
> 19:10:05 #0 0x7f9a2cc611f7 in ?? ()
> 19:10:05
> 19:10:05 2019-01-16 19:10:05,650 - archive_core_dumps - INFO - Found binary
> path through GDB:
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c
> 19:10:05 2019-01-16 19:10:05,893 - archive_core_dumps - WARNING - Failed to
> determine binary because multiple candidate binaries were found and none of
> their paths contained 'latest' to disambiguate:
> 19:10:05 Core:./core.1547693849.30799.kudu-master
> 19:10:05
> Binaries:['./testdata/cluster/node_templates/common/etc/init.d/kudu-master',
> './testdata/cluster/cdh6/node-1/etc/init.d/kudu-master']
> 19:10:05
> 19:10:05 2019-01-16 19:10:05,917 - archive_core_dumps - INFO - [New LWP 30783]
> 19:10:05 [New LWP 30810]
> 19:10:05 [New LWP 30812]
> 19:10:05 [New LWP 30811]
> 19:10:05 [New LWP 30824]
> 19:10:05 [New LWP 30820]
> 19:10:05 Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> 19:10:05 Program terminated with signal SIGABRT, Aborted.
> 19:10:05 #0 0x7f0b81fb11f7 in ?? ()
> {noformat}
> *Backtraces*:
> {noformat}
> CORE: ./core.1547693849.30799.kudu-master
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x7f9a2cc611f7 in ?? ()
> #0 0x7f9a2cc611f7 in ?? ()
> #1 0x7f9a2cc628e8 in ?? ()
> #2 0x0020 in ?? ()
> #3 0x in ?? ()
> CORE: ./core.1547693849.30783.kudu-tserver
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x7f0b81fb11f7 in ?? ()
> #0 0x7f0b81fb11f7 in ?? ()
> #1 0x7f0b81fb28e8 in ?? ()
> #2 0x0020 in ?? ()
> #3 0x in ?? ()
> CORE: ./core.1547693849.30767.kudu-tserver
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x7f60b1e2f1f7 in ?? ()
> #0 0x7f60b1e2f1f7 in ?? ()
> #1 0x7f60b1e308e8 in ?? ()
> #2 0x0020 in ?? ()
> #3 0x in ?? ()
> CORE: ./core.1547693849.30808.kudu-tserver
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x7fa3cb5591f7 in ?? ()
> #0 0x7fa3cb5591f7 in ?? ()
> #1 0x7fa3cb55a8e8 in ?? ()
> #2 0x0020 in ?? ()
>
[jira] [Assigned] (IMPALA-8096) Limit on #rows returned from query
[
https://issues.apache.org/jira/browse/IMPALA-8096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pooja Nilangekar reassigned IMPALA-8096:
Assignee: Pooja Nilangekar
> Limit on #rows returned from query
> --
>
> Key: IMPALA-8096
> URL: https://issues.apache.org/jira/browse/IMPALA-8096
> Project: IMPALA
> Issue Type: Sub-task
> Components: Backend
>Reporter: Tim Armstrong
>Assignee: Pooja Nilangekar
>Priority: Major
> Labels: resource-management
>
> Sometimes users accidentally run queries that return a large number of rows,
> e.g.
> {code}
> SELECT * FROM table
> {code}
> When they really only need to look at a subset of the rows. It would be
> useful to have a guardrail to fail queries the return more rows than a
> particular limit. Maybe it would make sense to integrate with IMPALA-4268 so
> that the query is failed when the buffer fills up, but it may also be useful
> to have an easier-to-understand option based on #rows.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8091) minicluster kudu failure: Cannot initialize clock: failed to wait for clock sync
[
https://issues.apache.org/jira/browse/IMPALA-8091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749157#comment-16749157
]
Michael Brown commented on IMPALA-8091:
---
This is a lot of diagnostic info...
{noformat}
I0116 18:56:30.146215 30799 master_main.cc:80] Initializing master server...
I0116 18:56:30.146333 30799 system_ntp.cc:122] Waiting up to
--ntp_initial_sync_wait_secs=60 seconds for the clock to synchronize
W0116 18:56:30.147728 30799 system_ntp.cc:132] Could not find ntp-wait; trying
chrony waitsync instead: Not found: Unable to find binary: ntp-wait
E0116 18:57:29.300457 30799 system_ntp.cc:157] Dumping NTP diagnostics
E0116 18:57:29.302417 30799 system_ntp.cc:102] /sbin/ntptime
--
stdout:
ntp_gettime() returns code 5 (ERROR)
time dfea6d99.4d59a2ec Wed, Jan 16 2019 18:57:29.302, (.302149739),
maximum error 1600 us, estimated error 1600 us, TAI offset 0
ntp_adjtime() returns code 5 (ERROR)
modes 0x0 (),
offset 0.000 us, frequency 129.100 ppm, interval 1 s,
maximum error 1600 us, estimated error 1600 us,
status 0x2041 (PLL,UNSYNC,NANO),
time constant 6, precision 0.001 us, tolerance 500 ppm,
E0116 18:57:29.330603 30799 system_ntp.cc:102] /sbin/ntpq -n -c timeout 1000 -c
readvar -c sysinfo -c lpeers -c opeers -c version
--
stdout:
associd=0 status=c618 leap_alarm, sync_ntp, 1 event, no_sys_peer,
version="ntpd 4.2.6p5@1.2349-o Wed Apr 12 21:24:06 UTC 2017 (1)",
processor="x86_64", system="Linux/3.10.0-693.5.2.el7.x86_64", leap=11,
stratum=2, precision=-24, rootdelay=65.438, rootdisp=43.467,
refid=96.126.122.39,
reftime=dfea6d72.e0d7d2b9 Wed, Jan 16 2019 18:56:50.878,
clock=dfea6d99.4f1265e8 Wed, Jan 16 2019 18:57:29.308, peer=64381, tc=6,
mintc=3, offset=0.000, frequency=129.100, sys_jitter=11.220,
clk_jitter=12.567, clk_wander=0.221
remote refid st t when poll reach delay offset jitter
==
+66.228.48.38127.67.113.922 u 39 647 70.236 -35.592 28.619
+208.75.89.4 63.145.169.3 2 u 37 647 64.246 -20.776 29.110
*96.126.122.39 .GPS.1 u 38 647 63.424 -31.810 28.640
+69.89.207.199 212.215.1.1572 u 36 647 37.729 -37.207 28.643
remote local st t when poll reach delay offsetdisp
==
+66.228.48.38172.28.204.722 u 39 647 70.236 -35.592 0.913
+208.75.89.4 172.28.204.722 u 37 647 64.246 -20.776 0.921
*96.126.122.39 172.28.204.721 u 38 647 63.424 -31.810 0.906
+69.89.207.199 172.28.204.722 u 36 647 37.729 -37.207 0.914
ntpq 4.2.6p5@1.2349-o Wed Apr 12 21:24:08 UTC 2017 (1)
stderr:
***Command `sysinfo' unknown
E0116 18:57:29.333056 30799 system_ntp.cc:102] /sbin/ntpdc -n -c timeout 1000
-c peers -c sysinfo -c sysstats -c version
--
stdout:
remote local st poll reach delay offsetdisp
===
*96.126.122.39 172.28.204.721 647 0.06342 -0.031810 0.03009
=208.75.89.4 172.28.204.722 647 0.06424 -0.020776 0.03033
=69.89.207.199 172.28.204.722 647 0.03772 -0.037207 0.03023
=66.228.48.38172.28.204.722 647 0.07022 -0.035592 0.03021
system peer: 96.126.122.39
system peer mode: client
leap indicator: 11
stratum: 2
precision:-24
root distance:0.06543 s
root dispersion: 0.04346 s
reference ID: [96.126.122.39]
reference time: dfea6d72.e0d7d2b9 Wed, Jan 16 2019 18:56:50.878
system flags: auth ntp kernel stats
jitter: 0.011215 s
stability:0.000 ppm
broadcastdelay: 0.00 s
authdelay:0.00 s
time since restart: 1459
time since reset: 1459
packets received: 180
packets processed: 124
current version:124
previous version: 0
declined: 0
access denied: 0
bad length or format: 0
bad authentication: 0
rate exceeded: 0
ntpdc 4.2.6p5@1.2349-o Wed Apr 12 21:24:08 UTC 2017 (1)
E0116 18:57:29.336165 30799 system_ntp.cc:102] /usr/bin/chronyc -n tracking
--
stdout:
506 Cannot talk to daemon
E0116 18:57:29.339103 30799 system_ntp.cc:102] /usr/bin/chronyc -n sources
--
stdout:
506 Cannot talk to daemon
F0116 18:57:29.339143 30799 master_main.cc:81] Check failed: _s.ok() Bad
status: Runtime error: Cannot initialize clock: failed to wait for clock sync
using command '/usr/bin/chronyc waitsync 60 0 0 1': /usr/bin/chronyc: process
exited
[jira] [Updated] (IMPALA-8091) minicluster kudu failure: Cannot initialize clock: failed to wait for clock sync
[
https://issues.apache.org/jira/browse/IMPALA-8091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Brown updated IMPALA-8091:
--
Component/s: (was: Backend)
Infrastructure
> minicluster kudu failure: Cannot initialize clock: failed to wait for clock
> sync
>
>
> Key: IMPALA-8091
> URL: https://issues.apache.org/jira/browse/IMPALA-8091
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.2.0
>Reporter: David Knupp
>Assignee: Thomas Tauber-Marshall
>Priority: Critical
> Labels: broken-build
>
> *Console log*:
> {noformat}
> 19:10:04 19:10:04 Error executing impala SQL:
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/logs/data_loading/sql/functional/create-functional-query-exhaustive-impala-generated-kudu-none-none.sql
> See:
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/logs/data_loading/sql/functional/create-functional-query-exhaustive-impala-generated-kudu-none-none.sql.log
> 19:10:04 Background task Loading functional-query data (pid 4831) failed.
> 19:10:04 Background task Loading TPC-H data (pid 4832) failed.
> 19:10:04 ERROR in
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/testdata/bin/create-load-data.sh
> at line 85: CM_HOST=${2-}
> [...]
> 19:10:05 2019-01-16 19:10:05,461 - archive_core_dumps - INFO - Found core
> files: ['./core.1547693849.30799.kudu-master',
> './core.1547693849.30783.kudu-tserver',
> './core.1547693849.30767.kudu-tserver',
> './core.1547693849.30808.kudu-tserver']
> 19:10:05 2019-01-16 19:10:05,649 - archive_core_dumps - INFO - [New LWP 30799]
> 19:10:05 [New LWP 30834]
> 19:10:05 [New LWP 30835]
> 19:10:05 [New LWP 30838]
> 19:10:05 [New LWP 30837]
> 19:10:05 [New LWP 30836]
> 19:10:05 Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> 19:10:05 Program terminated with signal SIGABRT, Aborted.
> 19:10:05 #0 0x7f9a2cc611f7 in ?? ()
> 19:10:05
> 19:10:05 2019-01-16 19:10:05,650 - archive_core_dumps - INFO - Found binary
> path through GDB:
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c
> 19:10:05 2019-01-16 19:10:05,893 - archive_core_dumps - WARNING - Failed to
> determine binary because multiple candidate binaries were found and none of
> their paths contained 'latest' to disambiguate:
> 19:10:05 Core:./core.1547693849.30799.kudu-master
> 19:10:05
> Binaries:['./testdata/cluster/node_templates/common/etc/init.d/kudu-master',
> './testdata/cluster/cdh6/node-1/etc/init.d/kudu-master']
> 19:10:05
> 19:10:05 2019-01-16 19:10:05,917 - archive_core_dumps - INFO - [New LWP 30783]
> 19:10:05 [New LWP 30810]
> 19:10:05 [New LWP 30812]
> 19:10:05 [New LWP 30811]
> 19:10:05 [New LWP 30824]
> 19:10:05 [New LWP 30820]
> 19:10:05 Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> 19:10:05 Program terminated with signal SIGABRT, Aborted.
> 19:10:05 #0 0x7f0b81fb11f7 in ?? ()
> {noformat}
> *Backtraces*:
> {noformat}
> CORE: ./core.1547693849.30799.kudu-master
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x7f9a2cc611f7 in ?? ()
> #0 0x7f9a2cc611f7 in ?? ()
> #1 0x7f9a2cc628e8 in ?? ()
> #2 0x0020 in ?? ()
> #3 0x in ?? ()
> CORE: ./core.1547693849.30783.kudu-tserver
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x7f0b81fb11f7 in ?? ()
> #0 0x7f0b81fb11f7 in ?? ()
> #1 0x7f0b81fb28e8 in ?? ()
> #2 0x0020 in ?? ()
> #3 0x in ?? ()
> CORE: ./core.1547693849.30767.kudu-tserver
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x7f60b1e2f1f7 in ?? ()
> #0 0x7f60b1e2f1f7 in ?? ()
> #1 0x7f60b1e308e8 in ?? ()
> #2 0x0020 in ?? ()
> #3 0x in ?? ()
> CORE: ./core.1547693849.30808.kudu-tserver
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x7fa3cb5591f7 in ?? ()
> #0 0x7fa3cb5591f7 in ?? ()
> #1 0x7fa3cb55a8e8 in ?? ()
> #2
[jira] [Updated] (IMPALA-8091) minicluster kudu failure: Cannot initialize clock: failed to wait for clock sync
[
https://issues.apache.org/jira/browse/IMPALA-8091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Brown updated IMPALA-8091:
--
Summary: minicluster kudu failure: Cannot initialize clock: failed to wait
for clock sync (was: Kudu SIGABRT'ed during dataload on Impala release build)
> minicluster kudu failure: Cannot initialize clock: failed to wait for clock
> sync
>
>
> Key: IMPALA-8091
> URL: https://issues.apache.org/jira/browse/IMPALA-8091
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: David Knupp
>Assignee: Thomas Tauber-Marshall
>Priority: Critical
> Labels: broken-build
>
> *Console log*:
> {noformat}
> 19:10:04 19:10:04 Error executing impala SQL:
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/logs/data_loading/sql/functional/create-functional-query-exhaustive-impala-generated-kudu-none-none.sql
> See:
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/logs/data_loading/sql/functional/create-functional-query-exhaustive-impala-generated-kudu-none-none.sql.log
> 19:10:04 Background task Loading functional-query data (pid 4831) failed.
> 19:10:04 Background task Loading TPC-H data (pid 4832) failed.
> 19:10:04 ERROR in
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/testdata/bin/create-load-data.sh
> at line 85: CM_HOST=${2-}
> [...]
> 19:10:05 2019-01-16 19:10:05,461 - archive_core_dumps - INFO - Found core
> files: ['./core.1547693849.30799.kudu-master',
> './core.1547693849.30783.kudu-tserver',
> './core.1547693849.30767.kudu-tserver',
> './core.1547693849.30808.kudu-tserver']
> 19:10:05 2019-01-16 19:10:05,649 - archive_core_dumps - INFO - [New LWP 30799]
> 19:10:05 [New LWP 30834]
> 19:10:05 [New LWP 30835]
> 19:10:05 [New LWP 30838]
> 19:10:05 [New LWP 30837]
> 19:10:05 [New LWP 30836]
> 19:10:05 Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> 19:10:05 Program terminated with signal SIGABRT, Aborted.
> 19:10:05 #0 0x7f9a2cc611f7 in ?? ()
> 19:10:05
> 19:10:05 2019-01-16 19:10:05,650 - archive_core_dumps - INFO - Found binary
> path through GDB:
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c
> 19:10:05 2019-01-16 19:10:05,893 - archive_core_dumps - WARNING - Failed to
> determine binary because multiple candidate binaries were found and none of
> their paths contained 'latest' to disambiguate:
> 19:10:05 Core:./core.1547693849.30799.kudu-master
> 19:10:05
> Binaries:['./testdata/cluster/node_templates/common/etc/init.d/kudu-master',
> './testdata/cluster/cdh6/node-1/etc/init.d/kudu-master']
> 19:10:05
> 19:10:05 2019-01-16 19:10:05,917 - archive_core_dumps - INFO - [New LWP 30783]
> 19:10:05 [New LWP 30810]
> 19:10:05 [New LWP 30812]
> 19:10:05 [New LWP 30811]
> 19:10:05 [New LWP 30824]
> 19:10:05 [New LWP 30820]
> 19:10:05 Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> 19:10:05 Program terminated with signal SIGABRT, Aborted.
> 19:10:05 #0 0x7f0b81fb11f7 in ?? ()
> {noformat}
> *Backtraces*:
> {noformat}
> CORE: ./core.1547693849.30799.kudu-master
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x7f9a2cc611f7 in ?? ()
> #0 0x7f9a2cc611f7 in ?? ()
> #1 0x7f9a2cc628e8 in ?? ()
> #2 0x0020 in ?? ()
> #3 0x in ?? ()
> CORE: ./core.1547693849.30783.kudu-tserver
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x7f0b81fb11f7 in ?? ()
> #0 0x7f0b81fb11f7 in ?? ()
> #1 0x7f0b81fb28e8 in ?? ()
> #2 0x0020 in ?? ()
> #3 0x in ?? ()
> CORE: ./core.1547693849.30767.kudu-tserver
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x7f60b1e2f1f7 in ?? ()
> #0 0x7f60b1e2f1f7 in ?? ()
> #1 0x7f60b1e308e8 in ?? ()
> #2 0x0020 in ?? ()
> #3 0x in ?? ()
> CORE: ./core.1547693849.30808.kudu-tserver
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x7fa3cb5591f7
[jira] [Commented] (IMPALA-8091) Kudu SIGABRT'ed during dataload on Impala release build
[
https://issues.apache.org/jira/browse/IMPALA-8091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749143#comment-16749143
]
Michael Brown commented on IMPALA-8091:
---
The failure isn't easy to triage from a console log. There, it looks like:
{noformat}
11:40:19 11:40:19 Error executing impala SQL:
/data/jenkins/workspace/impala-asf-master-core/repos/Impala/logs/data_loading/sql/functional/create-functional-query-exhaustive-impala-generated-kudu-none-none.sql
See:
/data/jenkins/workspace/impala-asf-master-core/repos/Impala/logs/data_loading/sql/functional/create-functional-query-exhaustive-impala-generated-kudu-none-none.sql.log
{noformat}
That log looks like this:
{noformat}
Traceback (most recent call last):
File
"/data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/bin/load-data.py",
line 207, in exec_impala_query_from_file
result = impala_client.execute(query)
File
"/data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/tests/beeswax/impala_beeswax.py",
line 182, in execute
handle = self.__execute_query(query_string.strip(), user=user)
File
"/data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/tests/beeswax/impala_beeswax.py",
line 357, in __execute_query
handle = self.execute_query_async(query_string, user=user)
File
"/data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/tests/beeswax/impala_beeswax.py",
line 351, in execute_query_async
handle = self.__do_rpc(lambda: self.imp_service.query(query,))
File
"/data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/tests/beeswax/impala_beeswax.py",
line 509, in __do_rpc
raise ImpalaBeeswaxException(self.__build_error_message(b), b)
ImpalaBeeswaxException: ImpalaBeeswaxException:
INNER EXCEPTION:
MESSAGE: ImpalaRuntimeException: Error creating Kudu table
'impala::tpch_kudu.lineitem'
CAUSED BY: NonRecoverableException: too many attempts:
KuduRpc(method=ListTables, tablet=null, attempt=101,
DeadlineTracker(timeout=18, elapsed=177706), Traces: [0ms] querying master,
[45ms] Sub rpc
: ConnectToMaster sending RPC to server master-127.0.0.1:7051, [100ms] Sub rpc:
ConnectToMaster received from server master-127.0.0.1:7051 response Network
error: java.net.ConnectException: Connection r
efused: /127.0.0.1:7051, [107ms] delaying RPC due to Service unavailable:
Master config (127.0.0.1:7051) has no leader.
{noformat}
This is the kudu service in the minicluster dropping out from under Impala.
Note btw that Kudu initially started ok.
{noformat}
11:26:26 Starting kudu (Web UI - http://localhost:8051)
11:26:41 The cluster is running
{noformat}
Finally, there's evidence in the minicluster logs for Kudu that all 4 services
(1 master, 3 tablet servers) all failed the same dcheck as pasted above.
> Kudu SIGABRT'ed during dataload on Impala release build
> ---
>
> Key: IMPALA-8091
> URL: https://issues.apache.org/jira/browse/IMPALA-8091
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: David Knupp
>Assignee: Thomas Tauber-Marshall
>Priority: Critical
> Labels: broken-build
>
> *Console log*:
> {noformat}
> 19:10:04 19:10:04 Error executing impala SQL:
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/logs/data_loading/sql/functional/create-functional-query-exhaustive-impala-generated-kudu-none-none.sql
> See:
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/logs/data_loading/sql/functional/create-functional-query-exhaustive-impala-generated-kudu-none-none.sql.log
> 19:10:04 Background task Loading functional-query data (pid 4831) failed.
> 19:10:04 Background task Loading TPC-H data (pid 4832) failed.
> 19:10:04 ERROR in
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/testdata/bin/create-load-data.sh
> at line 85: CM_HOST=${2-}
> [...]
> 19:10:05 2019-01-16 19:10:05,461 - archive_core_dumps - INFO - Found core
> files: ['./core.1547693849.30799.kudu-master',
> './core.1547693849.30783.kudu-tserver',
> './core.1547693849.30767.kudu-tserver',
> './core.1547693849.30808.kudu-tserver']
> 19:10:05 2019-01-16 19:10:05,649 - archive_core_dumps - INFO - [New LWP 30799]
> 19:10:05 [New LWP 30834]
> 19:10:05 [New LWP 30835]
> 19:10:05 [New LWP 30838]
> 19:10:05 [New LWP 30837]
> 19:10:05 [New LWP 30836]
> 19:10:05 Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> 19:10:05 Program terminated with signal SIGABRT, Aborted.
> 19:10:05 #0 0x7f9a2cc611f7 in ?? ()
> 19:10:05
> 19:10:05 2019-01-16 19:10:05,650 - archive_core_dumps - INFO - Found binary
> path through GDB:
>
[jira] [Commented] (IMPALA-8091) Kudu SIGABRT'ed during dataload on Impala release build
[
https://issues.apache.org/jira/browse/IMPALA-8091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749130#comment-16749130
]
Michael Brown commented on IMPALA-8091:
---
I've seen this too, and I think this is likely an environmental problem.
{noformat}
F0116 18:57:29.339143 30799 master_main.cc:81] Check failed: _s.ok() Bad
status: Runtime error: Cannot initialize clock: failed to wait for clock sync
using command '/usr/bin/chronyc waitsync 60 0 0 1': /usr/bin/chronyc: process
exited with non-zero status 1
{noformat}
> Kudu SIGABRT'ed during dataload on Impala release build
> ---
>
> Key: IMPALA-8091
> URL: https://issues.apache.org/jira/browse/IMPALA-8091
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: David Knupp
>Assignee: Thomas Tauber-Marshall
>Priority: Critical
> Labels: broken-build
>
> *Console log*:
> {noformat}
> 19:10:04 19:10:04 Error executing impala SQL:
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/logs/data_loading/sql/functional/create-functional-query-exhaustive-impala-generated-kudu-none-none.sql
> See:
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/logs/data_loading/sql/functional/create-functional-query-exhaustive-impala-generated-kudu-none-none.sql.log
> 19:10:04 Background task Loading functional-query data (pid 4831) failed.
> 19:10:04 Background task Loading TPC-H data (pid 4832) failed.
> 19:10:04 ERROR in
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/testdata/bin/create-load-data.sh
> at line 85: CM_HOST=${2-}
> [...]
> 19:10:05 2019-01-16 19:10:05,461 - archive_core_dumps - INFO - Found core
> files: ['./core.1547693849.30799.kudu-master',
> './core.1547693849.30783.kudu-tserver',
> './core.1547693849.30767.kudu-tserver',
> './core.1547693849.30808.kudu-tserver']
> 19:10:05 2019-01-16 19:10:05,649 - archive_core_dumps - INFO - [New LWP 30799]
> 19:10:05 [New LWP 30834]
> 19:10:05 [New LWP 30835]
> 19:10:05 [New LWP 30838]
> 19:10:05 [New LWP 30837]
> 19:10:05 [New LWP 30836]
> 19:10:05 Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> 19:10:05 Program terminated with signal SIGABRT, Aborted.
> 19:10:05 #0 0x7f9a2cc611f7 in ?? ()
> 19:10:05
> 19:10:05 2019-01-16 19:10:05,650 - archive_core_dumps - INFO - Found binary
> path through GDB:
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c
> 19:10:05 2019-01-16 19:10:05,893 - archive_core_dumps - WARNING - Failed to
> determine binary because multiple candidate binaries were found and none of
> their paths contained 'latest' to disambiguate:
> 19:10:05 Core:./core.1547693849.30799.kudu-master
> 19:10:05
> Binaries:['./testdata/cluster/node_templates/common/etc/init.d/kudu-master',
> './testdata/cluster/cdh6/node-1/etc/init.d/kudu-master']
> 19:10:05
> 19:10:05 2019-01-16 19:10:05,917 - archive_core_dumps - INFO - [New LWP 30783]
> 19:10:05 [New LWP 30810]
> 19:10:05 [New LWP 30812]
> 19:10:05 [New LWP 30811]
> 19:10:05 [New LWP 30824]
> 19:10:05 [New LWP 30820]
> 19:10:05 Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> 19:10:05 Program terminated with signal SIGABRT, Aborted.
> 19:10:05 #0 0x7f0b81fb11f7 in ?? ()
> {noformat}
> *Backtraces*:
> {noformat}
> CORE: ./core.1547693849.30799.kudu-master
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x7f9a2cc611f7 in ?? ()
> #0 0x7f9a2cc611f7 in ?? ()
> #1 0x7f9a2cc628e8 in ?? ()
> #2 0x0020 in ?? ()
> #3 0x in ?? ()
> CORE: ./core.1547693849.30783.kudu-tserver
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x7f0b81fb11f7 in ?? ()
> #0 0x7f0b81fb11f7 in ?? ()
> #1 0x7f0b81fb28e8 in ?? ()
> #2 0x0020 in ?? ()
> #3 0x in ?? ()
> CORE: ./core.1547693849.30767.kudu-tserver
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x7f60b1e2f1f7 in ?? ()
> #0 0x7f60b1e2f1f7 in ?? ()
> #1 0x7f60b1e308e8 in ?? ()
> #2 0x0020 in ?? ()
> #3 0x in ?? ()
> CORE: ./core.1547693849.30808.kudu-tserver
> BINARY: ./be/build/latest/service/impalad
[jira] [Assigned] (IMPALA-8093) Profiles prefix counters inconsistently
[
https://issues.apache.org/jira/browse/IMPALA-8093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen reassigned IMPALA-8093:
Assignee: Yongzhi Chen
> Profiles prefix counters inconsistently
> ---
>
> Key: IMPALA-8093
> URL: https://issues.apache.org/jira/browse/IMPALA-8093
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Lars Volker
>Assignee: Yongzhi Chen
>Priority: Major
> Labels: newbie, observability, profile, ramp-up
>
> In pretty printed profiles, plain counters are prefixed with \{{ - }} but
> TimeSeriesCounters are not. We should make them consistent.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8096) Limit on #rows returned from query
Tim Armstrong created IMPALA-8096:
-
Summary: Limit on #rows returned from query
Key: IMPALA-8096
URL: https://issues.apache.org/jira/browse/IMPALA-8096
Project: IMPALA
Issue Type: Sub-task
Components: Backend
Reporter: Tim Armstrong
Sometimes users accidentally run queries that return a large number of rows,
e.g.
{code}
SELECT * FROM table
{code}
When they really only need to look at a subset of the rows. It would be useful
to have a guardrail to fail queries the return more rows than a particular
limit. Maybe it would make sense to integrate with IMPALA-4268 so that the
query is failed when the buffer fills up, but it may also be useful to have an
easier-to-understand option based on #rows.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8027) KRPC datastream timing out on both the receiver and sender side even in a minicluster
[
https://issues.apache.org/jira/browse/IMPALA-8027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749067#comment-16749067
]
Michael Ho commented on IMPALA-8027:
Looks like the coordinator seems to be slow somehow between 05:16:52 and
05:18:56. Both failing queries had to do with time out waiting for the
coordinator fragment not becoming ready within 2 minutes.
Log from coordinators:
{noformat}
I1228 05:16:55.374305 26714 query-state.cc:568] Executing instance.
instance_id=8f46b2518734bef1:6ef2d404 fragment_idx=0
per_fragment_instance_idx=0 coord_state_idx=0 #in-flight=9
I1228 05:16:55.374305 26710 coordinator.cc:368] started execution on 2 backends
for query_id=8f46b2518734bef1:6ef2d404
{noformat}
{noformat}
I1228 05:16:52.217275 26685 query-state.cc:568] Executing instance.
instance_id=194f5b70907ac97c:84a116d6 fragment_idx=0
per_fragment_instance_idx=0 coord_state_idx=0 #in-flight=7
I1228 05:16:52.217211 26678 coordinator.cc:368] started execution on 3 backends
for query_id=194f5b70907ac97c:84a116d6
{noformat}
The top output snapshots from that time period didn't suggest much other than
Impalad wasn't scheduled much in them. Looking into query profiles next.
> KRPC datastream timing out on both the receiver and sender side even in a
> minicluster
> -
>
> Key: IMPALA-8027
> URL: https://issues.apache.org/jira/browse/IMPALA-8027
> Project: IMPALA
> Issue Type: Bug
> Components: Distributed Exec
>Affects Versions: Impala 3.2.0
>Reporter: Bikramjeet Vig
>Assignee: Michael Ho
>Priority: Critical
> Labels: broken-build
>
> krpc datastreams seem to time out at the same time at both sender and
> receiver causing two running queries to fail. This happened while running
> core tests on s3.
> Logs from coordinator:
> {noformat}
> I1228 05:18:56.202587 13396 krpc-data-stream-mgr.cc:353] Sender 127.0.0.1
> timed out waiting for receiver fragment instance:
> 8f46b2518734bef1:6ef2d404, dest node: 2
> I1228 05:18:56.203061 13396 rpcz_store.cc:265] Call
> impala.DataStreamService.TransmitData from 127.0.0.1:53118 (request call id
> 11274) took 120782ms. Request Metrics: {}
> I1228 05:18:56.203114 13396 krpc-data-stream-mgr.cc:353] Sender 127.0.0.1
> timed out waiting for receiver fragment instance:
> 194f5b70907ac97c:84a116d6, dest node: 2
> I1228 05:18:56.203136 13396 rpcz_store.cc:265] Call
> impala.DataStreamService.TransmitData from 127.0.0.1:53110 (request call id
> 8637) took 123811ms. Request Metrics: {}
> I1228 05:18:56.203155 13396 krpc-data-stream-mgr.cc:353] Sender 127.0.0.1
> timed out waiting for receiver fragment instance:
> 194f5b70907ac97c:84a116d6, dest node: 2
> I1228 05:18:56.203167 13396 rpcz_store.cc:265] Call
> impala.DataStreamService.TransmitData from 127.0.0.1:53118 (request call id
> 11273) took 123776ms. Request Metrics: {}
> I1228 05:18:56.203181 13396 krpc-data-stream-mgr.cc:408] Reduced stream ID
> cache from 413 items, to 410, eviction took: 1ms
> I1228 05:18:56.204746 13377 coordinator.cc:707] Backend completed:
> host=impala-ec2-centos74-m5-4xlarge-ondemand-07b3.vpc.cloudera.com:22001
> remaining=2 query_id=8f46b2518734bef1:6ef2d404
> I1228 05:18:56.204756 13377 coordinator-backend-state.cc:262]
> query_id=8f46b2518734bef1:6ef2d404: first in-progress backend:
> impala-ec2-centos74-m5-4xlarge-ondemand-07b3.vpc.cloudera.com:22000
> I1228 05:18:56.204769 13377 coordinator.cc:522] ExecState: query
> id=8f46b2518734bef1:6ef2d404
> finstance=8f46b2518734bef1:6ef2d4040001 on
> host=impala-ec2-centos74-m5-4xlarge-ondemand-07b3.vpc.cloudera.com:22001
> (EXECUTING -> ERROR) status=Sender 127.0.0.1 timed out waiting for receiver
> fragment instance: 8f46b2518734bef1:6ef2d404, dest node: 2
> {noformat}
> Logs from executor:
> {noformat}
> E1228 05:18:56.203181 26715 krpc-data-stream-sender.cc:343] channel send to
> 127.0.0.1:27000 failed:
> (fragment_instance_id=8f46b2518734bef1:6ef2d404): Sender 127.0.0.1
> timed out waiting for receiver fragment instance:
> 8f46b2518734bef1:6ef2d404, dest node: 2
> E1228 05:18:56.203256 26682 krpc-data-stream-sender.cc:343] channel send to
> 127.0.0.1:27000 failed:
> (fragment_instance_id=194f5b70907ac97c:84a116d6): Sender 127.0.0.1
> timed out waiting for receiver fragment instance:
> 194f5b70907ac97c:84a116d6, dest node: 2
> I1228 05:18:56.203451 26715 query-state.cc:576] Instance completed.
> instance_id=8f46b2518734bef1:6ef2d4040001 #in-flight=3
> status=DATASTREAM_SENDER_TIMEOUT: Sender 127.0.0.1 timed out waiting for
> receiver fragment instance: 8f46b2518734bef1:6ef2d404, dest node: 2
> I1228
[jira] [Updated] (IMPALA-8093) Profiles prefix counters inconsistently
[
https://issues.apache.org/jira/browse/IMPALA-8093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Volker updated IMPALA-8093:
Labels: newbie observability profile (was: observability profile)
> Profiles prefix counters inconsistently
> ---
>
> Key: IMPALA-8093
> URL: https://issues.apache.org/jira/browse/IMPALA-8093
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Lars Volker
>Priority: Major
> Labels: newbie, observability, profile
>
> In pretty printed profiles, plain counters are prefixed with \{{ - }} but
> TimeSeriesCounters are not. We should make them consistent.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-4249) Clarify lifetime of DiskIoMgr::ScanRange objects
[
https://issues.apache.org/jira/browse/IMPALA-4249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749057#comment-16749057
]
Tim Armstrong commented on IMPALA-4249:
---
We could consider using std::shared_ptr here since it does seem to be a genuine
case of shared ownership.
> Clarify lifetime of DiskIoMgr::ScanRange objects
>
>
> Key: IMPALA-4249
> URL: https://issues.apache.org/jira/browse/IMPALA-4249
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Affects Versions: Impala 2.8.0
>Reporter: Tim Armstrong
>Priority: Minor
> Labels: resource-management
>
> When working on the DiskIoMgr code I noticed that the lifetime of the
> ScanRange objects isn't very clear. In particular, it's not clear when it is
> safe for a client to free the ScanRange object. We should at least document
> the expectation in the code.
> It seems like in principle that the client should be able to free the scan
> range after it hits eosr() or cancels the scan range. However, that's not
> true because a) BufferDescriptors hold internal pointers to the ScanRange and
> b) the DiskIoMgr touches the ScanRange in the below code after enqueueing the
> last buffer:
> {code}
> bool queue_full = scan_range->EnqueueBuffer(buffer);
> if (eosr) {
> // For cached buffers, we can't close the range until the cached buffer
> is returned.
> // Close() is called from DiskIoMgr::ReturnBuffer().
> if (!is_cached) scan_range->Close();
> } else {
> ...
> {code}
> It seems like we (mostly?) avoid issues by sticking the ScanRanges in object
> pools where they will outlive the reader contexts. This seems suboptimal - we
> shouldn't really leave lots of small unused objects lying around.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8093) Profiles prefix counters inconsistently
[
https://issues.apache.org/jira/browse/IMPALA-8093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Volker updated IMPALA-8093:
Labels: newbie observability profile ramp-up (was: newbie observability
profile)
> Profiles prefix counters inconsistently
> ---
>
> Key: IMPALA-8093
> URL: https://issues.apache.org/jira/browse/IMPALA-8093
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Lars Volker
>Priority: Major
> Labels: newbie, observability, profile, ramp-up
>
> In pretty printed profiles, plain counters are prefixed with \{{ - }} but
> TimeSeriesCounters are not. We should make them consistent.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-5847) Some query options do not work as expected in .test files
[
https://issues.apache.org/jira/browse/IMPALA-5847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Tauber-Marshall resolved IMPALA-5847.
Resolution: Fixed
Fix Version/s: Impala 3.2.0
> Some query options do not work as expected in .test files
> -
>
> Key: IMPALA-5847
> URL: https://issues.apache.org/jira/browse/IMPALA-5847
> Project: IMPALA
> Issue Type: Improvement
> Components: Infrastructure
>Reporter: Alexander Behm
>Assignee: Thomas Tauber-Marshall
>Priority: Minor
> Fix For: Impala 3.2.0
>
>
> We often use "set" in .test files to alter query options. Theoretically, a
> "set" command should change the session-level query options and in most cases
> a single .test file is executed from the same Impala session. However, for
> some options using "set" within a query section does not seem to work. For
> example, "num_nodes" does not work as expected as shown below.
> PyTest:
> {code}
> import pytest
> from tests.common.impala_test_suite import ImpalaTestSuite
> class TestStringQueries(ImpalaTestSuite):
> @classmethod
> def get_workload(cls):
> return 'functional-query'
> def test_set_bug(self, vector):
> self.run_test_case('QueryTest/set_bug', vector)
> {code}
> Corresponding .test file:
> {code}
>
> QUERY
> set num_nodes=1;
> select count(*) from functional.alltypes;
> select count(*) from functional.alltypes;
> select count(*) from functional.alltypes;
> RESULTS
> 7300
> TYPES
> BIGINT
>
> {code}
> After running the test above, I validated that the 3 queries were run from
> the same session, and that the queries run a distributed plan. The
> "num_nodes" option was definitely not picked up. I am not sure which query
> options are affected. In several .test files setting other query options does
> seem to work as expected.
> I suspect that the test framework might keep its own list of default query
> options which get submitted together with the query, so the session-level
> options are overridden on a per-request basis. For example, if I change the
> pytest to remove the "num_nodes" dictionary entry, then the test works as
> expected.
> PyTest workaround:
> {code}
> import pytest
> from tests.common.impala_test_suite import ImpalaTestSuite
> class TestStringQueries(ImpalaTestSuite):
> @classmethod
> def get_workload(cls):
> return 'functional-query'
> def test_set_bug(self, vector):
> # Workaround SET bug
> vector.get_value('exec_option').pop('num_nodes', None)
> self.run_test_case('QueryTest/set_bug', vector)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-5847) Some query options do not work as expected in .test files
[
https://issues.apache.org/jira/browse/IMPALA-5847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749012#comment-16749012
]
ASF subversion and git services commented on IMPALA-5847:
-
Commit 6fa35478d5930cc3ec6d88c6b2e3f6bedd3d8ed9 in impala's branch
refs/heads/master from Thomas Tauber-Marshall
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=6fa3547 ]
IMPALA-5847: Fix incorrect use of SET in .test files
The '.test' files are used to run queries for tests. These files are
run with a vector of default query options. They also sometimes
include SET queries that modify query options. If SET is used on a
query option that is included in the vector, the default value from
the vector will override the value from the SET, leading to tests that
don't actually run with the query options they appear to.
This patch asserts that '.test' files don't use SET for values present
in the default vector. It also fixes various tests that already had
this incorrect behavior.
Testing:
- Passed a full exhaustive run.
Change-Id: I4e4c0f31bf4850642b624acdb1f6cb8837957990
Reviewed-on: http://gerrit.cloudera.org:8080/12220
Reviewed-by: Thomas Marshall
Tested-by: Impala Public Jenkins
> Some query options do not work as expected in .test files
> -
>
> Key: IMPALA-5847
> URL: https://issues.apache.org/jira/browse/IMPALA-5847
> Project: IMPALA
> Issue Type: Improvement
> Components: Infrastructure
>Reporter: Alexander Behm
>Assignee: Thomas Tauber-Marshall
>Priority: Minor
>
> We often use "set" in .test files to alter query options. Theoretically, a
> "set" command should change the session-level query options and in most cases
> a single .test file is executed from the same Impala session. However, for
> some options using "set" within a query section does not seem to work. For
> example, "num_nodes" does not work as expected as shown below.
> PyTest:
> {code}
> import pytest
> from tests.common.impala_test_suite import ImpalaTestSuite
> class TestStringQueries(ImpalaTestSuite):
> @classmethod
> def get_workload(cls):
> return 'functional-query'
> def test_set_bug(self, vector):
> self.run_test_case('QueryTest/set_bug', vector)
> {code}
> Corresponding .test file:
> {code}
>
> QUERY
> set num_nodes=1;
> select count(*) from functional.alltypes;
> select count(*) from functional.alltypes;
> select count(*) from functional.alltypes;
> RESULTS
> 7300
> TYPES
> BIGINT
>
> {code}
> After running the test above, I validated that the 3 queries were run from
> the same session, and that the queries run a distributed plan. The
> "num_nodes" option was definitely not picked up. I am not sure which query
> options are affected. In several .test files setting other query options does
> seem to work as expected.
> I suspect that the test framework might keep its own list of default query
> options which get submitted together with the query, so the session-level
> options are overridden on a per-request basis. For example, if I change the
> pytest to remove the "num_nodes" dictionary entry, then the test works as
> expected.
> PyTest workaround:
> {code}
> import pytest
> from tests.common.impala_test_suite import ImpalaTestSuite
> class TestStringQueries(ImpalaTestSuite):
> @classmethod
> def get_workload(cls):
> return 'functional-query'
> def test_set_bug(self, vector):
> # Workaround SET bug
> vector.get_value('exec_option').pop('num_nodes', None)
> self.run_test_case('QueryTest/set_bug', vector)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
