[jira] [Commented] (IMPALA-7151) session-expiry-test failed - unable to open ThriftServer port
[
https://issues.apache.org/jira/browse/IMPALA-7151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510451#comment-16510451
]
Tim Armstrong commented on IMPALA-7151:
---
I looked at this some more. I think adding a retry loop is very tricky with the
current code because it assumes in many places that we don't attempt to restart
the servers. It's also hard to test and prevent it bit-rotting. I can see a
couple of remaining solutions, but nothing really that good:
* Use non-ephemeral ports, which eliminates the chance of other services
randomly using the ports, but could cause the tests to fail on systems where
other services happen to be using those port ranges
* Reduce the window between choosing a port and starting the server by calling
FindUnusedEphemeralPort() when we create the socket, e.g. in
ThriftServer::CreateSocket().
> session-expiry-test failed - unable to open ThriftServer port
> -
>
> Key: IMPALA-7151
> URL: https://issues.apache.org/jira/browse/IMPALA-7151
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.13.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: flaky
> Attachments: session-expiry-test.ERROR, session-expiry-test.FATAL,
> session-expiry-test.INFO, session-expiry-test.WARNING
>
>
> {noformat}
> ThriftServer 'backend' (on port: 52436) exited due to TException: Could not
> bind: Transport endpoint is not connected
> {noformat}
> This is similar to IMPALA-5499 but is on CentOS 7:
> {noformat}
> # lsb_release -a
> LSB Version:
> :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
> Distributor ID: CentOS
> Description: CentOS Linux release 7.4.1708 (Core)
> Release: 7.4.1708
> Codename: Core
> {noformat}
> It's in the ephemeral port range:
> {noformat}
> # cat /proc/sys/net/ipv4/ip_local_port_range
> 32768 60999
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-5604) Document DISABLE_CODEGEN_ROWS_THRESHOLD
[ https://issues.apache.org/jira/browse/IMPALA-5604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510450#comment-16510450 ] Alex Rodoni commented on IMPALA-5604: - That will be great! Thank you [~tarmstrong] > Document DISABLE_CODEGEN_ROWS_THRESHOLD > --- > > Key: IMPALA-5604 > URL: https://issues.apache.org/jira/browse/IMPALA-5604 > Project: IMPALA > Issue Type: Task > Components: Docs >Affects Versions: Impala 2.10.0 >Reporter: Tim Armstrong >Assignee: Alex Rodoni >Priority: Major > > We just checking in DISABLE_CODEGEN_ROWS_THRESHOLD to master with default > value 50,000. > The option uses the planner's estimates to determine whether to disable > codegen for a query based on the maximum number of rows flowing through any > part of the query plan. > It is similar to EXEC_SINGLE_NODE_ROWS_THRESHOLD in behaviour except the > threshold is per-node rows instead of total rows. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-5604) Document DISABLE_CODEGEN_ROWS_THRESHOLD
[ https://issues.apache.org/jira/browse/IMPALA-5604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510435#comment-16510435 ] Tim Armstrong commented on IMPALA-5604: --- I saw a user get confused by this: http://community.cloudera.com/t5/Interactive-Short-cycle-SQL/Very-slow-CodeGen-taking-80-of-runtime/m-p/69016/highlight/false#M4600 I think we can reuse some of the EXEC_SINGLE_NODE_ROWS_THRESHOLD text since the mechanism is very similar, the differences being: * It uses a per-backend row count instead of a total row count. * It disables codegen only (i.e. if it kicks in, it has the same effect as setting disable_codegen=true) I can have a go at writing this up if you like [~arodoni_cloudera] > Document DISABLE_CODEGEN_ROWS_THRESHOLD > --- > > Key: IMPALA-5604 > URL: https://issues.apache.org/jira/browse/IMPALA-5604 > Project: IMPALA > Issue Type: Task > Components: Docs >Affects Versions: Impala 2.10.0 >Reporter: Tim Armstrong >Assignee: Alex Rodoni >Priority: Major > > We just checking in DISABLE_CODEGEN_ROWS_THRESHOLD to master with default > value 50,000. > The option uses the planner's estimates to determine whether to disable > codegen for a query based on the maximum number of rows flowing through any > part of the query plan. > It is similar to EXEC_SINGLE_NODE_ROWS_THRESHOLD in behaviour except the > threshold is per-node rows instead of total rows. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7158) Incorrect init of HdfsScanNodeBase::progress_
[
https://issues.apache.org/jira/browse/IMPALA-7158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
bharath v resolved IMPALA-7158.
---
Resolution: Fixed
Fix Version/s: Impala 3.1.0
Impala 2.13.0
> Incorrect init of HdfsScanNodeBase::progress_
> -
>
> Key: IMPALA-7158
> URL: https://issues.apache.org/jira/browse/IMPALA-7158
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Reporter: bharath v
>Assignee: bharath v
>Priority: Minor
> Fix For: Impala 2.13.0, Impala 3.1.0
>
>
> I was digging into something else and came across this. The initialization of
> the scan split progress is incorrect
> {noformat}
> Status HdfsScanNodeBase::Open(RuntimeState* state) {
> RETURN_IF_ERROR(ScanNode::Open(state));
> .
> .
> progress_.Init(Substitute("Splits complete (node=$0)",
> total_splits),total_splits);
> return Status::OK();
> }{noformat}
> It should be {{progress_.Init(Substitute("Splits complete (node=$0)",
> id_),total_splits);}} instead
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-6942) "Cancelled due to unreachable impalad(s)" error message is misleading
[ https://issues.apache.org/jira/browse/IMPALA-6942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dan Hecht reassigned IMPALA-6942: - Assignee: Dan Hecht Thanks for the reminder. i can take care of it. > "Cancelled due to unreachable impalad(s)" error message is misleading > - > > Key: IMPALA-6942 > URL: https://issues.apache.org/jira/browse/IMPALA-6942 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 3.0 >Reporter: Dan Hecht >Assignee: Dan Hecht >Priority: Major > > The error message "Cancelled due to unreachable impalad(s)" would be better > as "Failed due to unreachable impalad(s)" since the query has failed. The > code happens to trigger cancellation via the same path as client > cancellation, but this is really a query failure. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7167) Catalogd should reject files with unsupported formats
[
https://issues.apache.org/jira/browse/IMPALA-7167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510383#comment-16510383
]
Tim Armstrong commented on IMPALA-7167:
---
I noticed this was wonky too while working on IMPALA-6941. The patch solved
this by accepting them at the impalad and then failing during query execution
http://gerrit.cloudera.org:8080/10165. The previous behaviour was very weird
because it actually allowed unrecognised suffixes (like .foobar) by
interpreting them as uncompressed text.
> Catalogd should reject files with unsupported formats
> -
>
> Key: IMPALA-7167
> URL: https://issues.apache.org/jira/browse/IMPALA-7167
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 2.13.0, Impala 3.1.0
>Reporter: Tianyi Wang
>Priority: Minor
>
> Currently when a file format is unsupported, impalad will log an error:
> {noformat}
> E0611 13:42:47.428393 22076 ImpaladCatalog.java:201] Error adding catalog
> object: Expected compressed text file with {.lzo,.gzip,.snappy,.bz2} suffix:
> 00_0.deflate
> Java exception follows:
> java.lang.RuntimeException: Expected compressed text file with
> {.lzo,.gzip,.snappy,.bz2} suffix: 00_0.deflate
> at
> org.apache.impala.catalog.HdfsPartition.(HdfsPartition.java:772)
> at
> org.apache.impala.catalog.HdfsPartition.fromThrift(HdfsPartition.java:884)
> at
> org.apache.impala.catalog.HdfsTable.loadFromThrift(HdfsTable.java:1678)
> at org.apache.impala.catalog.Table.fromThrift(Table.java:311)
> at
> org.apache.impala.catalog.ImpaladCatalog.addTable(ImpaladCatalog.java:403)
> at
> org.apache.impala.catalog.ImpaladCatalog.addCatalogObject(ImpaladCatalog.java:292)
> at
> org.apache.impala.catalog.ImpaladCatalog.updateCatalog(ImpaladCatalog.java:199)
> at
> org.apache.impala.service.Frontend.updateCatalogCache(Frontend.java:228)
> at
> org.apache.impala.service.JniFrontend.updateCatalogCache(JniFrontend.java:174)
> {noformat}
> Catalogd should filter out unsupported files instead of letting every impalad
> log the error.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7158) Incorrect init of HdfsScanNodeBase::progress_
[
https://issues.apache.org/jira/browse/IMPALA-7158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
bharath v updated IMPALA-7158:
--
Component/s: Backend
> Incorrect init of HdfsScanNodeBase::progress_
> -
>
> Key: IMPALA-7158
> URL: https://issues.apache.org/jira/browse/IMPALA-7158
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Reporter: bharath v
>Assignee: bharath v
>Priority: Minor
>
> I was digging into something else and came across this. The initialization of
> the scan split progress is incorrect
> {noformat}
> Status HdfsScanNodeBase::Open(RuntimeState* state) {
> RETURN_IF_ERROR(ScanNode::Open(state));
> .
> .
> progress_.Init(Substitute("Splits complete (node=$0)",
> total_splits),total_splits);
> return Status::OK();
> }{noformat}
> It should be {{progress_.Init(Substitute("Splits complete (node=$0)",
> id_),total_splits);}} instead
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7165) Impala Doc: Example for Dynamic Partition Pruning need to be improved
[ https://issues.apache.org/jira/browse/IMPALA-7165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510375#comment-16510375 ] Alex Rodoni commented on IMPALA-7165: - https://gerrit.cloudera.org/#/c/10703/ > Impala Doc: Example for Dynamic Partition Pruning need to be improved > - > > Key: IMPALA-7165 > URL: https://issues.apache.org/jira/browse/IMPALA-7165 > Project: IMPALA > Issue Type: Bug > Components: Docs >Affects Versions: Impala 3.0, Impala 2.12.0 >Reporter: Alex Rodoni >Assignee: Alex Rodoni >Priority: Major > > The documentation provided here: > [https://impala.apache.org/docs/build/html/topics/impala_partitioning.html#dynamic_partition_pruning] > for Impala's Dynamic Partition Pruning uses wrong example. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7167) Catalogd should reject files with unsupported formats
Tianyi Wang created IMPALA-7167:
---
Summary: Catalogd should reject files with unsupported formats
Key: IMPALA-7167
URL: https://issues.apache.org/jira/browse/IMPALA-7167
Project: IMPALA
Issue Type: Bug
Affects Versions: Impala 2.13.0, Impala 3.1.0
Reporter: Tianyi Wang
Currently when a file format is unsupported, impalad will log an error:
{noformat}
E0611 13:42:47.428393 22076 ImpaladCatalog.java:201] Error adding catalog
object: Expected compressed text file with {.lzo,.gzip,.snappy,.bz2} suffix:
00_0.deflate
Java exception follows:
java.lang.RuntimeException: Expected compressed text file with
{.lzo,.gzip,.snappy,.bz2} suffix: 00_0.deflate
at
org.apache.impala.catalog.HdfsPartition.(HdfsPartition.java:772)
at
org.apache.impala.catalog.HdfsPartition.fromThrift(HdfsPartition.java:884)
at
org.apache.impala.catalog.HdfsTable.loadFromThrift(HdfsTable.java:1678)
at org.apache.impala.catalog.Table.fromThrift(Table.java:311)
at
org.apache.impala.catalog.ImpaladCatalog.addTable(ImpaladCatalog.java:403)
at
org.apache.impala.catalog.ImpaladCatalog.addCatalogObject(ImpaladCatalog.java:292)
at
org.apache.impala.catalog.ImpaladCatalog.updateCatalog(ImpaladCatalog.java:199)
at
org.apache.impala.service.Frontend.updateCatalogCache(Frontend.java:228)
at
org.apache.impala.service.JniFrontend.updateCatalogCache(JniFrontend.java:174)
{noformat}
Catalogd should filter out unsupported files instead of letting every impalad
log the error.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7166) ExecSummary should be a first class object
[ https://issues.apache.org/jira/browse/IMPALA-7166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sandeep akinapelli updated IMPALA-7166: --- Description: Impala RuntimeProfile currently contains "ExecSummary" as a string. We should make it a first class thrift object, so that tools can extract these fields (Est rows etc..), and also label them in the UI for potential candidates. (was: Impala RuntimeProfile currently contains "ExecSummary" as a string. We should make it a first class thrift object, so that tools like workload XM can extract these fields (Est rows etc..), and also label them in the UI for potential candidates.) > ExecSummary should be a first class object > -- > > Key: IMPALA-7166 > URL: https://issues.apache.org/jira/browse/IMPALA-7166 > Project: IMPALA > Issue Type: Bug > Components: Backend >Reporter: sandeep akinapelli >Priority: Major > Labels: resource-management, usability > > Impala RuntimeProfile currently contains "ExecSummary" as a string. We should > make it a first class thrift object, so that tools can extract these fields > (Est rows etc..), and also label them in the UI for potential candidates. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7166) ExecSummary should be a first class object
[ https://issues.apache.org/jira/browse/IMPALA-7166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sandeep akinapelli updated IMPALA-7166: --- Description: Impala RuntimeProfile currently contains "ExecSummary" as a string. We should make it a first class thrift object, so that tools can extract these fields (Est rows etc..), (was: Impala RuntimeProfile currently contains "ExecSummary" as a string. We should make it a first class thrift object, so that tools can extract these fields (Est rows etc..), and also label them in the UI for potential candidates.) > ExecSummary should be a first class object > -- > > Key: IMPALA-7166 > URL: https://issues.apache.org/jira/browse/IMPALA-7166 > Project: IMPALA > Issue Type: Bug > Components: Backend >Reporter: sandeep akinapelli >Priority: Major > Labels: resource-management, usability > > Impala RuntimeProfile currently contains "ExecSummary" as a string. We should > make it a first class thrift object, so that tools can extract these fields > (Est rows etc..), -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7164) Define public API for RuntimeProfile
[ https://issues.apache.org/jira/browse/IMPALA-7164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510338#comment-16510338 ] sandeep akinapelli commented on IMPALA-7164: I agree. A good start would be to keep them together in one place so that we can track the different versions of these counters, if and when they change. > Define public API for RuntimeProfile > > > Key: IMPALA-7164 > URL: https://issues.apache.org/jira/browse/IMPALA-7164 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Reporter: Tim Armstrong >Priority: Major > Labels: resource-management, usability > > Currently the only public API for the runtime profile is the Thrift > definition. The exact layout and counters are all implementation-dependent > and subject to change. People are trying to build tools that consume runtime > profiles and process them, but that's hard since the counters can change from > version to version and sometimes the semantics change. > I think we need to figure out which things are part of the public API, > validate that they make sense and document them clearly. We should also > clearly document that things outside of this public API are subject to change > without notice. I don't think the public API necessarily needs to be > "porcelain", but we should generally try to avoid unnecessary changes and > mention any changes in release notes etc. > We could start simple and just collect "public" counter names in a module and > comment each of them. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7164) Define public API for RuntimeProfile
[ https://issues.apache.org/jira/browse/IMPALA-7164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510337#comment-16510337 ] Philip Zeyliger commented on IMPALA-7164: - Sure. > Define public API for RuntimeProfile > > > Key: IMPALA-7164 > URL: https://issues.apache.org/jira/browse/IMPALA-7164 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Reporter: Tim Armstrong >Priority: Major > Labels: resource-management, usability > > Currently the only public API for the runtime profile is the Thrift > definition. The exact layout and counters are all implementation-dependent > and subject to change. People are trying to build tools that consume runtime > profiles and process them, but that's hard since the counters can change from > version to version and sometimes the semantics change. > I think we need to figure out which things are part of the public API, > validate that they make sense and document them clearly. We should also > clearly document that things outside of this public API are subject to change > without notice. I don't think the public API necessarily needs to be > "porcelain", but we should generally try to avoid unnecessary changes and > mention any changes in release notes etc. > We could start simple and just collect "public" counter names in a module and > comment each of them. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7145) Leak of memory from OpenSSL when spill-to-disk encryption is enabled
[
https://issues.apache.org/jira/browse/IMPALA-7145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-7145.
---
Resolution: Fixed
Fix Version/s: Impala 3.1.0
Impala 2.13.0
> Leak of memory from OpenSSL when spill-to-disk encryption is enabled
>
>
> Key: IMPALA-7145
> URL: https://issues.apache.org/jira/browse/IMPALA-7145
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.0, Impala 2.3.0, Impala 2.5.0, Impala 2.4.0,
> Impala 2.6.0, Impala 2.7.0, Impala 2.8.0, Impala 2.9.0, Impala 2.10.0, Impala
> 2.11.0, Impala 2.12.0, Impala 2.13.0, Impala 3.1.0, 2.2.0, 2.1.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: resource-management
> Fix For: Impala 2.13.0, Impala 3.1.0
>
>
> I tried looping this test for a while to flush out a different issue and
> discovered a memory leak:
> {noformat}
> while impala-py.test tests/query_test/test_spilling.py -k naaj -n4 --verbose;
> do date; done
> {noformat}
> Eventually the test fails with "Memory Limit Exceeded" and there is a huge
> amount of untracked memory reported on /memz.
> I tried running the heap growth profiler and the only suspicious thing I saw
> was CRYPTO_malloc, similar to IMPALA-6792. My current hypothesis is that it's
> something related to disk spill encryption and we haven't noticed it before
> because we had disk spill encryption off.
> If I disable encryption "start-impala-cluster.py
> --impalad_args=--disk_spill_encryption=false" then I don't see the leak.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7158) Incorrect init of HdfsScanNodeBase::progress_
[
https://issues.apache.org/jira/browse/IMPALA-7158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510331#comment-16510331
]
ASF subversion and git services commented on IMPALA-7158:
-
Commit 60dbc4db56b258026dc7ce91276dcec7600a67b4 in impala's branch
refs/heads/master from Bharath Vissapragada
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=60dbc4d ]
IMPALA-7158: Fix HdfsScanNodeBase::progress_'s init
(Testing) Verified that the correct node id is being logged
with this patch and --v=2.
Change-Id: Id2a738edea80ff3fb13ff368b4093c8b4ef34df7
Reviewed-on: http://gerrit.cloudera.org:8080/10672
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> Incorrect init of HdfsScanNodeBase::progress_
> -
>
> Key: IMPALA-7158
> URL: https://issues.apache.org/jira/browse/IMPALA-7158
> Project: IMPALA
> Issue Type: Bug
>Reporter: bharath v
>Assignee: bharath v
>Priority: Minor
>
> I was digging into something else and came across this. The initialization of
> the scan split progress is incorrect
> {noformat}
> Status HdfsScanNodeBase::Open(RuntimeState* state) {
> RETURN_IF_ERROR(ScanNode::Open(state));
> .
> .
> progress_.Init(Substitute("Splits complete (node=$0)",
> total_splits),total_splits);
> return Status::OK();
> }{noformat}
> It should be {{progress_.Init(Substitute("Splits complete (node=$0)",
> id_),total_splits);}} instead
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7004) Deflake erasure coding data loading
[ https://issues.apache.org/jira/browse/IMPALA-7004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510330#comment-16510330 ] ASF subversion and git services commented on IMPALA-7004: - Commit 8d6142fd2ec00b095d9a55ca9bba76d0d43d3ab3 in impala's branch refs/heads/2.x from [~tianyiwang] [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=8d6142f ] IMPALA-6394: Disable dfs.namenode.replication.considerLoad By default namenode excludes heavily-loaded datanodes from being considered for block placement. Impala tests rely on every block to be fully-replicated so this option should be set to false. This is originally from IMPALA-7004 which was for 3.x branch only. The option has been renamed to "dfs.namenode.redundancy.considerLoad" in Hadooop 3.0. Change-Id: I00010107cbd61bffa38ed8af4395b3a0e3d13c13 Reviewed-on: http://gerrit.cloudera.org:8080/10664 Reviewed-by: Tianyi Wang Tested-by: Impala Public Jenkins > Deflake erasure coding data loading > --- > > Key: IMPALA-7004 > URL: https://issues.apache.org/jira/browse/IMPALA-7004 > Project: IMPALA > Issue Type: Sub-task >Affects Versions: Impala 3.1.0 >Reporter: Tianyi Wang >Assignee: Tianyi Wang >Priority: Major > Fix For: Impala 3.1.0 > > > Currently the data loading of erasure coding is flaky. There are 2 separate > problems: > # HBase data loading sometimes fails: HBASE-19369 > # Nested data loading sometimes fails because there are not enough "Good" > data nodes. The root cause might be same as IMPALA-6394 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-2746) Backend tests should pass with leak sanitizer enabled
[ https://issues.apache.org/jira/browse/IMPALA-2746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510333#comment-16510333 ] ASF subversion and git services commented on IMPALA-2746: - Commit 501dc6abfb360575613eb2210c0c87f3e37235ce in impala's branch refs/heads/master from [~tarmstr...@cloudera.com] [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=501dc6a ] IMPALA-7145: fix leak of OpenSSL context when spilling Add a RAII wrapper for the OpenSSL context that automatically frees on all exit paths from the function. Add a backend test wrapper that enables LeakSanitizer for an individual test. This is a step towards IMPALA-2746. Fix version check bug in asan.h. Testing: Enable LeakSanitizer for openssl-util-test. This reliably found the bug. Ran core tests under ASAN. Change-Id: I98760ed8f31b18b489a156f945c29c95c9bf3184 Reviewed-on: http://gerrit.cloudera.org:8080/10666 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Backend tests should pass with leak sanitizer enabled > - > > Key: IMPALA-2746 > URL: https://issues.apache.org/jira/browse/IMPALA-2746 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 2.3.0 >Reporter: Martin Grund >Priority: Minor > Labels: resource-management, test-infra > > Currently, when running the backend tests with ASAN, the build will fail if > memory leak detection is enabled. We should investigate where leaks occur and > fix them to make sure we can benefit from the leak detection as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7145) Leak of memory from OpenSSL when spill-to-disk encryption is enabled
[
https://issues.apache.org/jira/browse/IMPALA-7145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510332#comment-16510332
]
ASF subversion and git services commented on IMPALA-7145:
-
Commit 501dc6abfb360575613eb2210c0c87f3e37235ce in impala's branch
refs/heads/master from [~tarmstr...@cloudera.com]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=501dc6a ]
IMPALA-7145: fix leak of OpenSSL context when spilling
Add a RAII wrapper for the OpenSSL context that automatically frees
on all exit paths from the function.
Add a backend test wrapper that enables LeakSanitizer for an individual
test. This is a step towards IMPALA-2746.
Fix version check bug in asan.h.
Testing:
Enable LeakSanitizer for openssl-util-test. This reliably found the bug.
Ran core tests under ASAN.
Change-Id: I98760ed8f31b18b489a156f945c29c95c9bf3184
Reviewed-on: http://gerrit.cloudera.org:8080/10666
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> Leak of memory from OpenSSL when spill-to-disk encryption is enabled
>
>
> Key: IMPALA-7145
> URL: https://issues.apache.org/jira/browse/IMPALA-7145
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.0, Impala 2.3.0, Impala 2.5.0, Impala 2.4.0,
> Impala 2.6.0, Impala 2.7.0, Impala 2.8.0, Impala 2.9.0, Impala 2.10.0, Impala
> 2.11.0, Impala 2.12.0, Impala 2.13.0, Impala 3.1.0, 2.2.0, 2.1.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: resource-management
>
> I tried looping this test for a while to flush out a different issue and
> discovered a memory leak:
> {noformat}
> while impala-py.test tests/query_test/test_spilling.py -k naaj -n4 --verbose;
> do date; done
> {noformat}
> Eventually the test fails with "Memory Limit Exceeded" and there is a huge
> amount of untracked memory reported on /memz.
> I tried running the heap growth profiler and the only suspicious thing I saw
> was CRYPTO_malloc, similar to IMPALA-6792. My current hypothesis is that it's
> something related to disk spill encryption and we haven't noticed it before
> because we had disk spill encryption off.
> If I disable encryption "start-impala-cluster.py
> --impalad_args=--disk_spill_encryption=false" then I don't see the leak.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7165) Impala Doc: Example for Dynamic Partition Pruning need to be improved
[ https://issues.apache.org/jira/browse/IMPALA-7165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jim Apple updated IMPALA-7165: -- Description: The documentation provided here: [https://impala.apache.org/docs/build/html/topics/impala_partitioning.html#dynamic_partition_pruning] for Impala's Dynamic Partition Pruning uses wrong example. was: The documentation provided here: [http://www.cloudera.com/documentation/enterprise/latest/topics/impala_partitioning.html#dynamic_partition_pruning] for Impala's Dynamic Partition Pruning uses wrong example. > Impala Doc: Example for Dynamic Partition Pruning need to be improved > - > > Key: IMPALA-7165 > URL: https://issues.apache.org/jira/browse/IMPALA-7165 > Project: IMPALA > Issue Type: Bug > Components: Docs >Affects Versions: Impala 3.0, Impala 2.12.0 >Reporter: Alex Rodoni >Assignee: Alex Rodoni >Priority: Major > > The documentation provided here: > [https://impala.apache.org/docs/build/html/topics/impala_partitioning.html#dynamic_partition_pruning] > for Impala's Dynamic Partition Pruning uses wrong example. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7164) Define public API for RuntimeProfile
[ https://issues.apache.org/jira/browse/IMPALA-7164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510322#comment-16510322 ] Tim Armstrong commented on IMPALA-7164: --- [~philip] I'm not opposed to that, but I think we can do the work incrementally if we separate out public/non-public counters first before we do a global change that enshrines all of the existing counters in the thrift definition. I'm thinking that there are probably < 50 counters that are actually widely used so putting those in one place feels pretty manageable once we identify them. > Define public API for RuntimeProfile > > > Key: IMPALA-7164 > URL: https://issues.apache.org/jira/browse/IMPALA-7164 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Reporter: Tim Armstrong >Priority: Major > Labels: resource-management, usability > > Currently the only public API for the runtime profile is the Thrift > definition. The exact layout and counters are all implementation-dependent > and subject to change. People are trying to build tools that consume runtime > profiles and process them, but that's hard since the counters can change from > version to version and sometimes the semantics change. > I think we need to figure out which things are part of the public API, > validate that they make sense and document them clearly. We should also > clearly document that things outside of this public API are subject to change > without notice. I don't think the public API necessarily needs to be > "porcelain", but we should generally try to avoid unnecessary changes and > mention any changes in release notes etc. > We could start simple and just collect "public" counter names in a module and > comment each of them. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7165) Impala Doc: Example for Dynamic Partition Pruning need to be improved
Alex Rodoni created IMPALA-7165: --- Summary: Impala Doc: Example for Dynamic Partition Pruning need to be improved Key: IMPALA-7165 URL: https://issues.apache.org/jira/browse/IMPALA-7165 Project: IMPALA Issue Type: Bug Components: Docs Affects Versions: Impala 2.12.0, Impala 3.0 Reporter: Alex Rodoni Assignee: Alex Rodoni The documentation provided here: [http://www.cloudera.com/documentation/enterprise/latest/topics/impala_partitioning.html#dynamic_partition_pruning] for Impala's Dynamic Partition Pruning uses wrong example. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7164) Define public API for RuntimeProfile
[ https://issues.apache.org/jira/browse/IMPALA-7164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510312#comment-16510312 ] Philip Zeyliger commented on IMPALA-7164: - I think instead of a map of counters keyed by counter name, we should strive to put those names in the Thrift schema definition itself. That will allow evolution and be considerably more obvious. > Define public API for RuntimeProfile > > > Key: IMPALA-7164 > URL: https://issues.apache.org/jira/browse/IMPALA-7164 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Reporter: Tim Armstrong >Priority: Major > Labels: resource-management, usability > > Currently the only public API for the runtime profile is the Thrift > definition. The exact layout and counters are all implementation-dependent > and subject to change. People are trying to build tools that consume runtime > profiles and process them, but that's hard since the counters can change from > version to version and sometimes the semantics change. > I think we need to figure out which things are part of the public API, > validate that they make sense and document them clearly. We should also > clearly document that things outside of this public API are subject to change > without notice. I don't think the public API necessarily needs to be > "porcelain", but we should generally try to avoid unnecessary changes and > mention any changes in release notes etc. > We could start simple and just collect "public" counter names in a module and > comment each of them. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7164) Define public API for RuntimeProfile
[ https://issues.apache.org/jira/browse/IMPALA-7164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong updated IMPALA-7164: -- Labels: resource-management usability (was: usability) > Define public API for RuntimeProfile > > > Key: IMPALA-7164 > URL: https://issues.apache.org/jira/browse/IMPALA-7164 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Reporter: Tim Armstrong >Priority: Major > Labels: resource-management, usability > > Currently the only public API for the runtime profile is the Thrift > definition. The exact layout and counters are all implementation-dependent > and subject to change. People are trying to build tools that consume runtime > profiles and process them, but that's hard since the counters can change from > version to version and sometimes the semantics change. > I think we need to figure out which things are part of the public API, > validate that they make sense and document them clearly. We should also > clearly document that things outside of this public API are subject to change > without notice. I don't think the public API necessarily needs to be > "porcelain", but we should generally try to avoid unnecessary changes and > mention any changes in release notes etc. > We could start simple and just collect "public" counter names in a module and > comment each of them. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7164) Define public API for RuntimeProfile
Tim Armstrong created IMPALA-7164: - Summary: Define public API for RuntimeProfile Key: IMPALA-7164 URL: https://issues.apache.org/jira/browse/IMPALA-7164 Project: IMPALA Issue Type: Improvement Components: Backend Reporter: Tim Armstrong Currently the only public API for the runtime profile is the Thrift definition. The exact layout and counters are all implementation-dependent and subject to change. People are trying to build tools that consume runtime profiles and process them, but that's hard since the counters can change from version to version and sometimes the semantics change. I think we need to figure out which things are part of the public API, validate that they make sense and document them clearly. We should also clearly document that things outside of this public API are subject to change without notice. I don't think the public API necessarily needs to be "porcelain", but we should generally try to avoid unnecessary changes and mention any changes in release notes etc. We could start simple and just collect "public" counter names in a module and comment each of them. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7096) Confirm that IMPALA-4835 does not increase chance of OOM for scans of wide tables
[
https://issues.apache.org/jira/browse/IMPALA-7096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510283#comment-16510283
]
Mostafa Mokhtar commented on IMPALA-7096:
-
[~tarmstrong]
Attached OOM profile where most memory is consumed by the HDFS_SCAN_NODE and
very little memory is in the queues.
{code}
Query Status: Memory limit exceeded: Error occurred on backend
vb0228.foo:22000 by fragment d7448d78ac85ba6b:303b7242000c
Memory left in process limit: 98.37 GB
Memory left in query limit: -371.21 KB
Query(d7448d78ac85ba6b:303b7242): memory limit exceeded. Limit=1.00 GB
Reservation=816.00 MB ReservationLimit=819.20 MB OtherMemory=208.36 MB
Total=1.00 GB Peak=1.00 GB
Fragment d7448d78ac85ba6b:303b7242000c: Reservation=360.00 MB
OtherMemory=85.60 MB Total=445.60 MB Peak=445.60 MB
HDFS_SCAN_NODE (id=2): Reservation=360.00 MB OtherMemory=85.45 MB
Total=445.45 MB Peak=445.45 MB
Exprs: Total=4.00 KB Peak=4.00 KB
Queued Batches: Total=22.13 MB Peak=22.13 MB
KrpcDataStreamSender (dst_id=13): Total=17.72 KB Peak=17.72 KB
CodeGen: Total=1.68 KB Peak=679.50 KB
Fragment d7448d78ac85ba6b:303b72420023: Reservation=96.00 MB
OtherMemory=43.09 MB Total=139.09 MB Peak=139.09 MB
HDFS_SCAN_NODE (id=6): Reservation=96.00 MB OtherMemory=42.93 MB
Total=138.93 MB Peak=138.93 MB
Exprs: Total=4.00 KB Peak=4.00 KB
Queued Batches: Total=26.10 MB Peak=26.10 MB
KrpcDataStreamSender (dst_id=16): Total=17.72 KB Peak=17.72 KB
CodeGen: Total=1.68 KB Peak=679.50 KB
Fragment d7448d78ac85ba6b:303b7242003a: Reservation=360.00 MB
OtherMemory=79.67 MB Total=439.67 MB Peak=439.67 MB
HDFS_SCAN_NODE (id=10): Reservation=360.00 MB OtherMemory=79.52 MB
Total=439.52 MB Peak=439.52 MB
Exprs: Total=4.00 KB Peak=4.00 KB
Queued Batches: Total=22.20 MB Peak=22.20 MB
KrpcDataStreamSender (dst_id=19): Total=17.72 KB Peak=17.72 KB
CodeGen: Total=1.68 KB Peak=679.50 KB
{code}
> Confirm that IMPALA-4835 does not increase chance of OOM for scans of wide
> tables
> -
>
> Key: IMPALA-7096
> URL: https://issues.apache.org/jira/browse/IMPALA-7096
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.13.0, Impala 3.1.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: resource-management
> Attachments: ScanConsumingMostMemory.txt
>
>
> IMPALA-7078 showed some cases where non-buffer memory could accumulate in the
> row batch queue and cause memory consumption problems.
> The decision for whether to spin up a scanner thread in IMPALA-4835
> implicitly assumes that buffer memory is the bulk of memory consumed by a
> scan, but there may be cases where that is not true and the previous
> heuristic would be more conservative about starting a scanner thread.
> We should investigate this further and figure out how to avoid it if there's
> an issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6408) [DOCS] Description of "shuffle" hint does not mention changes in IMPALA-3930
[ https://issues.apache.org/jira/browse/IMPALA-6408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510179#comment-16510179 ] ASF subversion and git services commented on IMPALA-6408: - Commit 09f16232e308594a1f522d88c75db9daa5f6e038 in impala's branch refs/heads/master from [~arodoni_cloudera] [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=09f1623 ] IMPALA-6408: [DOCS] Add a missing info about SHUFFLE Change-Id: I5738557354c384aab983f64722dde5944037aed7 Reviewed-on: http://gerrit.cloudera.org:8080/10685 Reviewed-by: Csaba Ringhofer Reviewed-by: Alex Rodoni Tested-by: Impala Public Jenkins > [DOCS] Description of "shuffle" hint does not mention changes in IMPALA-3930 > > > Key: IMPALA-6408 > URL: https://issues.apache.org/jira/browse/IMPALA-6408 > Project: IMPALA > Issue Type: Bug > Components: Docs >Reporter: Csaba Ringhofer >Assignee: Alex Rodoni >Priority: Minor > Fix For: Impala 2.13.0, Impala 3.1.0 > > > The change in IMPALA-3930 states that if only one partition is written > (because all partitioning columns are constant or the target table is not > partitioned), then the "shuffle" hint leads to a plan where all rows are > merged at the coordinator where the table sink is executed. > The documentation of the "shuffle" hint does not mention this behavior. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7156) Many authorization tests failed with "User 'jenkins' does not have privileges to access"
[
https://issues.apache.org/jira/browse/IMPALA-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510178#comment-16510178
]
ASF subversion and git services commented on IMPALA-7156:
-
Commit 01c0fa5772f35bf492d19ef067cfe58cc103b37b in impala's branch
refs/heads/2.x from [~fredyw]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=01c0fa5 ]
Revert "IMPALA-7156: temporary revert to older Sentry version"
This reverts commit 908865a7e7ce38e9356057f5e0a7ab75e30fb2bc.
Change-Id: I908a91f607f0a9ce8cb972ea8673771665ad88db
Reviewed-on: http://gerrit.cloudera.org:8080/10689
Tested-by: Impala Public Jenkins
Reviewed-by: Lars Volker
> Many authorization tests failed with "User 'jenkins' does not have privileges
> to access"
>
>
> Key: IMPALA-7156
> URL: https://issues.apache.org/jira/browse/IMPALA-7156
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.13.0
>Reporter: Tim Armstrong
>Assignee: Fredy Wijaya
>Priority: Blocker
> Labels: broken-build, flaky
> Fix For: Impala 2.13.0
>
> Attachments: 0001-Fix-IMPALA-7156-breakage-in-Sentry.patch,
> jenkins-priv-logs.tar.gz
>
>
> Failed tests:
> {noformat}
>
> authorization.test_grant_revoke.TestGrantRevoke.test_role_privilege_case[exec_option:
> {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
> 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
> 'exec_single_node_rows_threshold': 0} | table_format: text/none]
> org.apache.impala.analysis.AuthorizationTest.TestServerLevelInsert[1]
> org.apache.impala.analysis.AuthorizationTest.TestFunction[1]
> org.apache.impala.analysis.AuthorizationTest.TestDropDatabase[1]
> org.apache.impala.analysis.AuthorizationTest.TestServerLevelSelect[1]
> org.apache.impala.analysis.AuthorizationTest.TestTruncate[1]
> org.apache.impala.analysis.AuthorizationTest.TestShowCreateTable[1]
> org.apache.impala.analysis.AuthorizationTest.TestShortUsernameUsed[1]
> org.apache.impala.analysis.AuthorizationTest.TestComputeStatsTable[1]
> org.apache.impala.analysis.AuthorizationTest.TestHs2GetColumns[1]
> org.apache.impala.analysis.AuthorizationTest.TestLoad[1]
> org.apache.impala.analysis.AuthorizationTest.TestCreateDatabase[1]
> org.apache.impala.analysis.AuthorizationTest.TestInsert[1]
> org.apache.impala.analysis.AuthorizationTest.TestSelect[1]
> org.apache.impala.analysis.AuthorizationTest.TestCreateView[1]
> org.apache.impala.analysis.AuthorizationTest.TestServerLevelAlter[1]
> org.apache.impala.analysis.AuthorizationTest.TestResetMetadata[1]
> org.apache.impala.analysis.AuthorizationTest.TestExplain[1]
> org.apache.impala.analysis.AuthorizationTest.TestCommentOn[1]
> org.apache.impala.analysis.AuthorizationTest.TestWithClause[1]
> org.apache.impala.analysis.AuthorizationTest.TestAlterView[1]
> org.apache.impala.analysis.AuthorizationTest.TestShowPermissions[1]
> org.apache.impala.analysis.AuthorizationTest.TestServerLevelDrop[1]
> org.apache.impala.analysis.AuthorizationTest.TestDropView[1]
> org.apache.impala.analysis.AuthorizationTest.TestShowDbResultsFiltered[1]
>
> org.apache.impala.analysis.AuthorizationTest.TestShowTableResultsFiltered[1]
> org.apache.impala.analysis.AuthorizationTest.TestUnion[1]
> org.apache.impala.analysis.AuthorizationTest.TestUseDb[1]
> org.apache.impala.analysis.AuthorizationTest.TestDropStats[1]
> org.apache.impala.analysis.AuthorizationTest.TestDropTable[1]
> org.apache.impala.analysis.AuthorizationTest.TestHs2GetSchema[1]
> org.apache.impala.analysis.AuthorizationTest.TestHs2GetTables[1]
> org.apache.impala.analysis.AuthorizationTest.TestDescribeDb[1]
> org.apache.impala.analysis.AuthorizationTest.TestAlterTable[1]
> org.apache.impala.analysis.AuthorizationTest.TestDescribe[1]
> org.apache.impala.analysis.AuthorizationTest.TestServerLevelCreate[1]
> org.apache.impala.analysis.AuthorizationTest.TestCreateTable[1]
> {noformat}
> The errors are all along the lines of:
> {noformat}
> org.apache.impala.catalog.AuthorizationException: User 'jenkins' does not
> have privileges to execute 'INSERT' on: functional_parquet.alltypes
> at
> org.apache.impala.authorization.AuthorizationChecker.checkAccess(AuthorizationChecker.java:146)
> at
> org.apache.impala.analysis.AnalysisContext.authorizePrivilegeRequest(AnalysisContext.java:567)
> at
> org.apache.impala.analysis.AnalysisContext.authorize(AnalysisContext.java:530)
> at
> org.apache.impala.analysis.AnalysisContext.analyzeAndAuthorize(AnalysisContext.java:409)
> at
>
[jira] [Commented] (IMPALA-7144) Reenable tests disabled by IMPALA-7143
[ https://issues.apache.org/jira/browse/IMPALA-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510180#comment-16510180 ] ASF subversion and git services commented on IMPALA-7144: - Commit ac25e3f2515d1bad0700888ca880e082e5d3071a in impala's branch refs/heads/master from [~fredyw] [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=ac25e3f ] IMPALA-7144: Re-enable TestDescribeTableResults This patch makes the TestDescribeTableResults more robust by only comparing the information that the authorization cares about instead of comparing all output in DESCRIBE. This change will avoid any unnecessary changes to AuthorizationTest if HMS updates the DESCRIBE output. The test is also updated to support standalone execution without relying on other tests be executed first since it can cause the test to be flaky especially if the tests in AuthorizationTest are executed in parallel. Testing: - Ran all FE tests Cherry-picks: not for 2.x Change-Id: I3aeaecf5b6d906a66d338e165a6d506e3964563f Reviewed-on: http://gerrit.cloudera.org:8080/10643 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Reenable tests disabled by IMPALA-7143 > -- > > Key: IMPALA-7144 > URL: https://issues.apache.org/jira/browse/IMPALA-7144 > Project: IMPALA > Issue Type: Bug > Components: Frontend >Affects Versions: Impala 2.13.0, Impala 3.1.0 >Reporter: Tim Armstrong >Assignee: Fredy Wijaya >Priority: Blocker > Fix For: Impala 2.13.0, Impala 3.1.0 > > > IMPALA-7143 should have unblocked tests but we need to get this test coverage > back. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7144) Reenable tests disabled by IMPALA-7143
[ https://issues.apache.org/jira/browse/IMPALA-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fredy Wijaya resolved IMPALA-7144. -- Resolution: Fixed Fix Version/s: Impala 3.1.0 Impala 2.13.0 > Reenable tests disabled by IMPALA-7143 > -- > > Key: IMPALA-7144 > URL: https://issues.apache.org/jira/browse/IMPALA-7144 > Project: IMPALA > Issue Type: Bug > Components: Frontend >Affects Versions: Impala 2.13.0, Impala 3.1.0 >Reporter: Tim Armstrong >Assignee: Fredy Wijaya >Priority: Blocker > Fix For: Impala 2.13.0, Impala 3.1.0 > > > IMPALA-7143 should have unblocked tests but we need to get this test coverage > back. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7163) Implement a state machine for the QueryState class
[ https://issues.apache.org/jira/browse/IMPALA-7163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510103#comment-16510103 ] Sailesh Mukil commented on IMPALA-7163: --- FYI [~kwho] [~dhecht] > Implement a state machine for the QueryState class > -- > > Key: IMPALA-7163 > URL: https://issues.apache.org/jira/browse/IMPALA-7163 > Project: IMPALA > Issue Type: Improvement >Reporter: Sailesh Mukil >Assignee: Sailesh Mukil >Priority: Major > > We've recently been improving our query lifecycle by adding explicit state > transitions so it's easier to reason about what should happen at a given > stage in the lifetime of a query or a fragment instance. > On the coordinator side. The coordinator's view of the query: > https://github.com/apache/impala/commit/6ca87e46736a1e591ed7d7d5fee05b4b4d2fbb50 > On the fragment instance state side. A FIS's view of its own execution: > https://github.com/apache/impala/blob/e12ee485cf4c77203b144c053ee167509cc39374/be/src/runtime/fragment-instance-state.h#L182-L203 > We don't have something like this for the QueryState class which maintains > query wide state per executor. Adding it should make the lifecycle of a query > from an executors point of view much easier to reason about. > Additional info: This was identified as part of work for > IMPALA-2990/IMPALA-4063, and is a precursor to it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7162) idle_query_timeout setting in Impala needs more clarification
[ https://issues.apache.org/jira/browse/IMPALA-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510091#comment-16510091 ] Tim Armstrong commented on IMPALA-7162: --- I just wanted to clarify that one particular point about resources, since cancelled queries now don't hold onto significant resources. I think we could clarify the difference further of what being cancelled-but-open vs closed means since it's confusing - people wonder why the queries are still hanging around in some form (the reason is that they need to remain open for the client to be able to fetch status, profiles, etc). > idle_query_timeout setting in Impala needs more clarification > - > > Key: IMPALA-7162 > URL: https://issues.apache.org/jira/browse/IMPALA-7162 > Project: IMPALA > Issue Type: Documentation > Components: Docs >Affects Versions: Impala 2.6.0 >Reporter: Laurel Hale >Priority: Minor > Labels: documentation > > 1) Please refer to the following Impala documentation: > [http://impala.apache.org/docs/build3x/html/topics/impala_timeouts.html] > 2) Please see the section where idle_query_timeout is explained. > * > Request:- > * > We've mentioned that idle_query_timeout cancels the query. > Can we please elaborate it further and mention that query still stays open > (after being cancelled) and consumes a slot in Impala admission control? > Therefore, though idle_query_timeout helps in saving some resources at Impala > end, you may still want to configure idle_Session_timeout to same value since > idle_Session_timeout even closes the query for idle sessions (instead of just > cancelling) though the requirement for it to trigger is session should stay > idle. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7111) ASAN heap-use-after-free in impala::HdfsPluginTextScanner::CheckPluginEnabled
[
https://issues.apache.org/jira/browse/IMPALA-7111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510082#comment-16510082
]
Lars Volker commented on IMPALA-7111:
-
I saw this again in an ASAN build.
> ASAN heap-use-after-free in impala::HdfsPluginTextScanner::CheckPluginEnabled
> -
>
> Key: IMPALA-7111
> URL: https://issues.apache.org/jira/browse/IMPALA-7111
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.13.0, Impala 3.1.0
>Reporter: Lars Volker
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: asan, broken-build
>
> [~tarmstr...@cloudera.com] - I'm assigning this to you since you added this
> file in IMPALA-6941.
> {noformat}
> ==4582==ERROR: AddressSanitizer: heap-use-after-free on address
> 0x603000e8aa28 at pc 0x017ab9b4 bp 0x7f67e5f6b650 sp 0x7f67e5f6b648
> READ of size 1 at 0x603000e8aa28 thread T9236
> #0 0x17ab9b3 in bool
> __gnu_cxx::__ops::_Iter_pred
> >::operator()<__gnu_cxx::__normal_iterator
> >(__gnu_cxx::__normal_iterator)
> /data/jenkins/workspace/impala-asf-2.x-core-asan/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/bits/predefined_ops.h:231:24
> #1 0x17ab745 in __gnu_cxx::__normal_iterator
> std::__find_if<__gnu_cxx::__normal_iterator,
> __gnu_cxx::__ops::_Iter_pred >
> >(__gnu_cxx::__normal_iterator,
> __gnu_cxx::__normal_iterator,
> __gnu_cxx::__ops::_Iter_pred >,
> std::random_access_iterator_tag)
> /data/jenkins/workspace/impala-asf-2.x-core-asan/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/bits/stl_algo.h:140:8
> #2 0x17ab2dc in __gnu_cxx::__normal_iterator
> std::__find_if<__gnu_cxx::__normal_iterator,
> __gnu_cxx::__ops::_Iter_pred >
> >(__gnu_cxx::__normal_iterator,
> __gnu_cxx::__normal_iterator,
> __gnu_cxx::__ops::_Iter_pred >)
> /data/jenkins/workspace/impala-asf-2.x-core-asan/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/bits/stl_algo.h:161:14
> #3 0x17aaf6c in __gnu_cxx::__normal_iterator
> std::find_if<__gnu_cxx::__normal_iterator,
> boost::algorithm::detail::is_any_ofF
> >(__gnu_cxx::__normal_iterator,
> __gnu_cxx::__normal_iterator,
> boost::algorithm::detail::is_any_ofF)
> /data/jenkins/workspace/impala-asf-2.x-core-asan/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/bits/stl_algo.h:3803:14
> #4 0x17aaba1 in boost::iterator_range<__gnu_cxx::__normal_iterator std::string> >
> boost::algorithm::detail::token_finderF
> >::operator()<__gnu_cxx::__normal_iterator
> >(__gnu_cxx::__normal_iterator,
> __gnu_cxx::__normal_iterator) const
> /data/jenkins/workspace/impala-asf-2.x-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/algorithm/string/detail/finder.hpp:565:41
> #5 0x17ac118 in
> boost::function2 std::string> >, __gnu_cxx::__normal_iterator,
> __gnu_cxx::__normal_iterator
> >::operator()(__gnu_cxx::__normal_iterator,
> __gnu_cxx::__normal_iterator) const
> /data/jenkins/workspace/impala-asf-2.x-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:766:14
> #6 0x17abf8d in
> boost::algorithm::detail::find_iterator_base<__gnu_cxx::__normal_iterator std::string> >::do_find(__gnu_cxx::__normal_iterator,
> __gnu_cxx::__normal_iterator) const
> /data/jenkins/workspace/impala-asf-2.x-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/algorithm/string/detail/find_iterator.hpp:63:32
> #7 0x17aa00c in
> boost::algorithm::split_iterator<__gnu_cxx::__normal_iterator std::string> >::increment()
> /data/jenkins/workspace/impala-asf-2.x-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/algorithm/string/find_iterator.hpp:305:44
> #8 0x17a95a5 in
> boost::algorithm::split_iterator<__gnu_cxx::__normal_iterator std::string>
> >::split_iterator
> > >(__gnu_cxx::__normal_iterator,
> __gnu_cxx::__normal_iterator,
> boost::algorithm::detail::token_finderF
> >)
> /data/jenkins/workspace/impala-asf-2.x-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/algorithm/string/find_iterator.hpp:265:21
> #9 0x17a8d5e in std::vector >&
> boost::algorithm::iter_split std::allocator >, std::string,
> boost::algorithm::detail::token_finderF
> > >(std::vector >&, std::string&,
> boost::algorithm::detail::token_finderF
> >)
> /data/jenkins/workspace/impala-asf-2.x-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/algorithm/string/iter_find.hpp:170:21
> #10 0x179754f in std::vector >&
> boost::algorithm::split
> >, std::string, boost::algorithm::detail::is_any_ofF
> >(std::vector >&, std::string&,
> boost::algorithm::detail::is_any_ofF,
>
[jira] [Commented] (IMPALA-7162) idle_query_timeout setting in Impala needs more clarification
[ https://issues.apache.org/jira/browse/IMPALA-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510081#comment-16510081 ] Laurel Hale commented on IMPALA-7162: - [~tarmstrong] Is this not a valid issue? > idle_query_timeout setting in Impala needs more clarification > - > > Key: IMPALA-7162 > URL: https://issues.apache.org/jira/browse/IMPALA-7162 > Project: IMPALA > Issue Type: Documentation > Components: Docs >Affects Versions: Impala 2.6.0 >Reporter: Laurel Hale >Priority: Minor > Labels: documentation > > 1) Please refer to the following Impala documentation: > [http://impala.apache.org/docs/build3x/html/topics/impala_timeouts.html] > 2) Please see the section where idle_query_timeout is explained. > * > Request:- > * > We've mentioned that idle_query_timeout cancels the query. > Can we please elaborate it further and mention that query still stays open > (after being cancelled) and consumes a slot in Impala admission control? > Therefore, though idle_query_timeout helps in saving some resources at Impala > end, you may still want to configure idle_Session_timeout to same value since > idle_Session_timeout even closes the query for idle sessions (instead of just > cancelling) though the requirement for it to trigger is session should stay > idle. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7163) Implement a state machine for the QueryState class
[ https://issues.apache.org/jira/browse/IMPALA-7163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sailesh Mukil updated IMPALA-7163: -- Description: We've recently been improving our query lifecycle by adding explicit state transitions so it's easier to reason about what should happen at a given stage in the lifetime of a query or a fragment instance. On the coordinator side. The coordinator's view of the query: https://github.com/apache/impala/commit/6ca87e46736a1e591ed7d7d5fee05b4b4d2fbb50 On the fragment instance state side. A FIS's view of its own execution: https://github.com/apache/impala/blob/e12ee485cf4c77203b144c053ee167509cc39374/be/src/runtime/fragment-instance-state.h#L182-L203 We don't have something like this for the QueryState class which maintains query wide state per executor. Adding it should make the lifecycle of a query from an executors point of view much easier to reason about. Additional info: This was identified as part of work for IMPALA-2990/IMPALA-4063, and is a precursor to it. was: We've recently been improving our query lifecycle by adding explicit state transitions so it's easier to reason about what should happen at a given stage in the lifetime of a query or a fragment instance. On the coordinator side. The coordinator's view of the query: https://github.com/apache/impala/commit/6ca87e46736a1e591ed7d7d5fee05b4b4d2fbb50 On the fragment instance state side. A FIS's view of its own execution: https://github.com/apache/impala/blob/e12ee485cf4c77203b144c053ee167509cc39374/be/src/runtime/fragment-instance-state.h#L182-L203 We don't have something like this for the QueryState class which maintains query wide state per executor. Adding it should make the lifecycle of a query from an executors point of view much easier to reason about. Additional info: This was identified as part of work for IMPALA-2990, and is a precursor to it. > Implement a state machine for the QueryState class > -- > > Key: IMPALA-7163 > URL: https://issues.apache.org/jira/browse/IMPALA-7163 > Project: IMPALA > Issue Type: Improvement >Reporter: Sailesh Mukil >Assignee: Sailesh Mukil >Priority: Major > > We've recently been improving our query lifecycle by adding explicit state > transitions so it's easier to reason about what should happen at a given > stage in the lifetime of a query or a fragment instance. > On the coordinator side. The coordinator's view of the query: > https://github.com/apache/impala/commit/6ca87e46736a1e591ed7d7d5fee05b4b4d2fbb50 > On the fragment instance state side. A FIS's view of its own execution: > https://github.com/apache/impala/blob/e12ee485cf4c77203b144c053ee167509cc39374/be/src/runtime/fragment-instance-state.h#L182-L203 > We don't have something like this for the QueryState class which maintains > query wide state per executor. Adding it should make the lifecycle of a query > from an executors point of view much easier to reason about. > Additional info: This was identified as part of work for > IMPALA-2990/IMPALA-4063, and is a precursor to it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7161) Bootstrap's handling of JAVA_HOME needs improvement
[
https://issues.apache.org/jira/browse/IMPALA-7161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510039#comment-16510039
]
Philip Zeyliger commented on IMPALA-7161:
-
This all sounds good to me. I think if there's a working java on the path, we
should use it.
I found the following works for me:
{code}
export JAVA_HOME="$(jrunscript -e
'java.lang.System.out.println(java.lang.System.getProperty("java.home"));')"
{code}
I think the {{readlink}} approach is good too.
> Bootstrap's handling of JAVA_HOME needs improvement
> ---
>
> Key: IMPALA-7161
> URL: https://issues.apache.org/jira/browse/IMPALA-7161
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.13.0, Impala 3.1.0
>Reporter: Joe McDonnell
>Priority: Major
>
> bin/bootstrap_system.sh installs the Java SDK and sets JAVA_HOME in the
> current shell. It also adds a command to the bin/impala-config-local.sh to
> export JAVA_HOME there. This doesn't do the job.
> bin/impala-config.sh tests for JAVA_HOME at the very start of the script,
> before it has sourced bin/impala-config-local.sh. So, the user doesn't have a
> way of developing over the long term without manually setting up JAVA_HOME.
> bin/impala-config.sh also doesn't detect the system JAVA_HOME. For Ubuntu
> 16.04, this is fairly simple and if a developer has their system JDK set up
> appropriately, it would make sense to use it. For example:
>
> {noformat}
> # If javac exists, then the system has a Java SDK (JRE does not have javac).
> # Follow the symbolic links and use this to determine the system's JAVA_HOME.
> if [ -L /usr/bin/javac ]; then
> SYSTEM_JAVA_HOME=$(readlink -f /usr/bin/javac | sed "s:bin/javac::")
> fi
> export JAVA_HOME="${JAVA_HOME:-${SYSTEM_JAVA_HOME}}"{noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7162) idle_query_timeout setting in Impala needs more clarification
[ https://issues.apache.org/jira/browse/IMPALA-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510040#comment-16510040 ] Tim Armstrong commented on IMPALA-7162: --- After IMPALA-1575 cancelled queries don't consume slots in admission control (thankfully!). > idle_query_timeout setting in Impala needs more clarification > - > > Key: IMPALA-7162 > URL: https://issues.apache.org/jira/browse/IMPALA-7162 > Project: IMPALA > Issue Type: Documentation > Components: Docs >Affects Versions: Impala 2.6.0 >Reporter: Laurel Hale >Priority: Minor > Labels: documentation > > 1) Please refer to the following Impala documentation: > [http://impala.apache.org/docs/build3x/html/topics/impala_timeouts.html] > 2) Please see the section where idle_query_timeout is explained. > * > Request:- > * > We've mentioned that idle_query_timeout cancels the query. > Can we please elaborate it further and mention that query still stays open > (after being cancelled) and consumes a slot in Impala admission control? > Therefore, though idle_query_timeout helps in saving some resources at Impala > end, you may still want to configure idle_Session_timeout to same value since > idle_Session_timeout even closes the query for idle sessions (instead of just > cancelling) though the requirement for it to trigger is session should stay > idle. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7162) idle_query_timeout setting in Impala needs more clarification
Laurel Hale created IMPALA-7162: --- Summary: idle_query_timeout setting in Impala needs more clarification Key: IMPALA-7162 URL: https://issues.apache.org/jira/browse/IMPALA-7162 Project: IMPALA Issue Type: Documentation Components: Docs Affects Versions: Impala 2.6.0 Reporter: Laurel Hale 1) Please refer to the following Impala documentation: [http://impala.apache.org/docs/build3x/html/topics/impala_timeouts.html] 2) Please see the section where idle_query_timeout is explained. * Request:- * We've mentioned that idle_query_timeout cancels the query. Can we please elaborate it further and mention that query still stays open (after being cancelled) and consumes a slot in Impala admission control? Therefore, though idle_query_timeout helps in saving some resources at Impala end, you may still want to configure idle_Session_timeout to same value since idle_Session_timeout even closes the query for idle sessions (instead of just cancelling) though the requirement for it to trigger is session should stay idle. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7145) Leak of memory from OpenSSL when spill-to-disk encryption is enabled
[
https://issues.apache.org/jira/browse/IMPALA-7145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-7145:
--
Summary: Leak of memory from OpenSSL when spill-to-disk encryption is
enabled (was: Memory leak when looping test_spilling_naaj)
> Leak of memory from OpenSSL when spill-to-disk encryption is enabled
>
>
> Key: IMPALA-7145
> URL: https://issues.apache.org/jira/browse/IMPALA-7145
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.0, Impala 2.3.0, Impala 2.5.0, Impala 2.4.0,
> Impala 2.6.0, Impala 2.7.0, Impala 2.8.0, Impala 2.9.0, Impala 2.10.0, Impala
> 2.11.0, Impala 2.12.0, Impala 2.13.0, Impala 3.1.0, 2.2.0, 2.1.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: resource-management
>
> I tried looping this test for a while to flush out a different issue and
> discovered a memory leak:
> {noformat}
> while impala-py.test tests/query_test/test_spilling.py -k naaj -n4 --verbose;
> do date; done
> {noformat}
> Eventually the test fails with "Memory Limit Exceeded" and there is a huge
> amount of untracked memory reported on /memz.
> I tried running the heap growth profiler and the only suspicious thing I saw
> was CRYPTO_malloc, similar to IMPALA-6792. My current hypothesis is that it's
> something related to disk spill encryption and we haven't noticed it before
> because we had disk spill encryption off.
> If I disable encryption "start-impala-cluster.py
> --impalad_args=--disk_spill_encryption=false" then I don't see the leak.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7145) Memory leak when looping test_spilling_naaj
[
https://issues.apache.org/jira/browse/IMPALA-7145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510009#comment-16510009
]
Tim Armstrong commented on IMPALA-7145:
---
It looks like IMPALA-6128 made things about 4x worse, but there was always some
memory leaked.
{noformat}
AES-GCM:
SUMMARY: AddressSanitizer: 13440 byte(s) leaked in 32 allocation(s).
AES-CTR:
SUMMARY: AddressSanitizer: 13440 byte(s) leaked in 32 allocation(s).
AES-CFB:
SUMMARY: AddressSanitizer: 3697 byte(s) leaked in 14 allocation(s).
{noformat}
> Memory leak when looping test_spilling_naaj
> ---
>
> Key: IMPALA-7145
> URL: https://issues.apache.org/jira/browse/IMPALA-7145
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.0, Impala 2.3.0, Impala 2.5.0, Impala 2.4.0,
> Impala 2.6.0, Impala 2.7.0, Impala 2.8.0, Impala 2.9.0, Impala 2.10.0, Impala
> 2.11.0, Impala 2.12.0, Impala 2.13.0, Impala 3.1.0, 2.2.0, 2.1.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: resource-management
>
> I tried looping this test for a while to flush out a different issue and
> discovered a memory leak:
> {noformat}
> while impala-py.test tests/query_test/test_spilling.py -k naaj -n4 --verbose;
> do date; done
> {noformat}
> Eventually the test fails with "Memory Limit Exceeded" and there is a huge
> amount of untracked memory reported on /memz.
> I tried running the heap growth profiler and the only suspicious thing I saw
> was CRYPTO_malloc, similar to IMPALA-6792. My current hypothesis is that it's
> something related to disk spill encryption and we haven't noticed it before
> because we had disk spill encryption off.
> If I disable encryption "start-impala-cluster.py
> --impalad_args=--disk_spill_encryption=false" then I don't see the leak.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7161) Bootstrap's handling of JAVA_HOME needs improvement
[
https://issues.apache.org/jira/browse/IMPALA-7161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509900#comment-16509900
]
Joe McDonnell commented on IMPALA-7161:
---
Additional issue:
Suppose a user sets JAVA_HOME in their environment like they are supposed to.
Suppose they also have JAVA_HOME in bin/impala-config-local.sh like
bin/bootstrap_system.sh writes. The two can get out of sync. The JAVA would be
from the environment JAVA_HOME, but after setting JAVA,
bin/impala-config-local.sh's value for JAVA_HOME would overwrite the
environment variable and JAVA_HOME would be that value. These could point to
two different places.
> Bootstrap's handling of JAVA_HOME needs improvement
> ---
>
> Key: IMPALA-7161
> URL: https://issues.apache.org/jira/browse/IMPALA-7161
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.13.0, Impala 3.1.0
>Reporter: Joe McDonnell
>Priority: Major
>
> bin/bootstrap_system.sh installs the Java SDK and sets JAVA_HOME in the
> current shell. It also adds a command to the bin/impala-config-local.sh to
> export JAVA_HOME there. This doesn't do the job.
> bin/impala-config.sh tests for JAVA_HOME at the very start of the script,
> before it has sourced bin/impala-config-local.sh. So, the user doesn't have a
> way of developing over the long term without manually setting up JAVA_HOME.
> bin/impala-config.sh also doesn't detect the system JAVA_HOME. For Ubuntu
> 16.04, this is fairly simple and if a developer has their system JDK set up
> appropriately, it would make sense to use it. For example:
>
> {noformat}
> # If javac exists, then the system has a Java SDK (JRE does not have javac).
> # Follow the symbolic links and use this to determine the system's JAVA_HOME.
> if [ -L /usr/bin/javac ]; then
> SYSTEM_JAVA_HOME=$(readlink -f /usr/bin/javac | sed "s:bin/javac::")
> fi
> export JAVA_HOME="${JAVA_HOME:-${SYSTEM_JAVA_HOME}}"{noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7156) Many authorization tests failed with "User 'jenkins' does not have privileges to access"
[
https://issues.apache.org/jira/browse/IMPALA-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fredy Wijaya resolved IMPALA-7156.
--
Resolution: Fixed
Fix Version/s: Impala 2.13.0
> Many authorization tests failed with "User 'jenkins' does not have privileges
> to access"
>
>
> Key: IMPALA-7156
> URL: https://issues.apache.org/jira/browse/IMPALA-7156
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.13.0
>Reporter: Tim Armstrong
>Assignee: Fredy Wijaya
>Priority: Blocker
> Labels: broken-build, flaky
> Fix For: Impala 2.13.0
>
> Attachments: 0001-Fix-IMPALA-7156-breakage-in-Sentry.patch,
> jenkins-priv-logs.tar.gz
>
>
> Failed tests:
> {noformat}
>
> authorization.test_grant_revoke.TestGrantRevoke.test_role_privilege_case[exec_option:
> {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
> 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
> 'exec_single_node_rows_threshold': 0} | table_format: text/none]
> org.apache.impala.analysis.AuthorizationTest.TestServerLevelInsert[1]
> org.apache.impala.analysis.AuthorizationTest.TestFunction[1]
> org.apache.impala.analysis.AuthorizationTest.TestDropDatabase[1]
> org.apache.impala.analysis.AuthorizationTest.TestServerLevelSelect[1]
> org.apache.impala.analysis.AuthorizationTest.TestTruncate[1]
> org.apache.impala.analysis.AuthorizationTest.TestShowCreateTable[1]
> org.apache.impala.analysis.AuthorizationTest.TestShortUsernameUsed[1]
> org.apache.impala.analysis.AuthorizationTest.TestComputeStatsTable[1]
> org.apache.impala.analysis.AuthorizationTest.TestHs2GetColumns[1]
> org.apache.impala.analysis.AuthorizationTest.TestLoad[1]
> org.apache.impala.analysis.AuthorizationTest.TestCreateDatabase[1]
> org.apache.impala.analysis.AuthorizationTest.TestInsert[1]
> org.apache.impala.analysis.AuthorizationTest.TestSelect[1]
> org.apache.impala.analysis.AuthorizationTest.TestCreateView[1]
> org.apache.impala.analysis.AuthorizationTest.TestServerLevelAlter[1]
> org.apache.impala.analysis.AuthorizationTest.TestResetMetadata[1]
> org.apache.impala.analysis.AuthorizationTest.TestExplain[1]
> org.apache.impala.analysis.AuthorizationTest.TestCommentOn[1]
> org.apache.impala.analysis.AuthorizationTest.TestWithClause[1]
> org.apache.impala.analysis.AuthorizationTest.TestAlterView[1]
> org.apache.impala.analysis.AuthorizationTest.TestShowPermissions[1]
> org.apache.impala.analysis.AuthorizationTest.TestServerLevelDrop[1]
> org.apache.impala.analysis.AuthorizationTest.TestDropView[1]
> org.apache.impala.analysis.AuthorizationTest.TestShowDbResultsFiltered[1]
>
> org.apache.impala.analysis.AuthorizationTest.TestShowTableResultsFiltered[1]
> org.apache.impala.analysis.AuthorizationTest.TestUnion[1]
> org.apache.impala.analysis.AuthorizationTest.TestUseDb[1]
> org.apache.impala.analysis.AuthorizationTest.TestDropStats[1]
> org.apache.impala.analysis.AuthorizationTest.TestDropTable[1]
> org.apache.impala.analysis.AuthorizationTest.TestHs2GetSchema[1]
> org.apache.impala.analysis.AuthorizationTest.TestHs2GetTables[1]
> org.apache.impala.analysis.AuthorizationTest.TestDescribeDb[1]
> org.apache.impala.analysis.AuthorizationTest.TestAlterTable[1]
> org.apache.impala.analysis.AuthorizationTest.TestDescribe[1]
> org.apache.impala.analysis.AuthorizationTest.TestServerLevelCreate[1]
> org.apache.impala.analysis.AuthorizationTest.TestCreateTable[1]
> {noformat}
> The errors are all along the lines of:
> {noformat}
> org.apache.impala.catalog.AuthorizationException: User 'jenkins' does not
> have privileges to execute 'INSERT' on: functional_parquet.alltypes
> at
> org.apache.impala.authorization.AuthorizationChecker.checkAccess(AuthorizationChecker.java:146)
> at
> org.apache.impala.analysis.AnalysisContext.authorizePrivilegeRequest(AnalysisContext.java:567)
> at
> org.apache.impala.analysis.AnalysisContext.authorize(AnalysisContext.java:530)
> at
> org.apache.impala.analysis.AnalysisContext.analyzeAndAuthorize(AnalysisContext.java:409)
> at
> org.apache.impala.common.FrontendTestBase.parseAndAnalyze(FrontendTestBase.java:429)
> at
> org.apache.impala.analysis.AuthorizationTest.AuthzOk(AuthorizationTest.java:2841)
> at
> org.apache.impala.analysis.AuthorizationTest.AuthzOk(AuthorizationTest.java:2836)
> at
> org.apache.impala.analysis.AuthorizationTest.AuthzOk(AuthorizationTest.java:2832)
> at
> org.apache.impala.analysis.AuthorizationTest.TestTruncate(AuthorizationTest.java:1417)
> {noformat}
>
[jira] [Work started] (IMPALA-6664) Tag log statements with query-ids
[ https://issues.apache.org/jira/browse/IMPALA-6664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-6664 started by Zoram Thanga. > Tag log statements with query-ids > - > > Key: IMPALA-6664 > URL: https://issues.apache.org/jira/browse/IMPALA-6664 > Project: IMPALA > Issue Type: Improvement > Components: Backend, Frontend >Affects Versions: Impala 2.12.0 >Reporter: bharath v >Assignee: Zoram Thanga >Priority: Major > Labels: supportability > > The idea is to include query-id in each logged statement in > coordinator/finsts/planner. This helps to collect query specific logs by just > grep'ing through the logs with the query-id. > This is hopefully easier to implement, now that we have the query debug info > available in the thread context [1]. > [1] https://issues.apache.org/jira/browse/IMPALA-3703 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-6665) Tag CatalogOp logs with query IDs
[ https://issues.apache.org/jira/browse/IMPALA-6665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoram Thanga reassigned IMPALA-6665: Assignee: Zoram Thanga > Tag CatalogOp logs with query IDs > - > > Key: IMPALA-6665 > URL: https://issues.apache.org/jira/browse/IMPALA-6665 > Project: IMPALA > Issue Type: Improvement > Components: Catalog >Affects Versions: Impala 2.12.0 >Reporter: bharath v >Assignee: Zoram Thanga >Priority: Major > Labels: supportability > > Similar to IMPALA-6664. The idea is to improve catalog server logging by > adding query-ID to each of the Catalog server log statements. This helps map > Catalog errors to specific queries, which is currently not possible. > Raising a separate jira for the Catalog server since fixing it could be a > little tricker than the other components since we don't have the query hash > readily available in the Catalog context. We need to augment the Catalog RPCs > with this data. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-6664) Tag log statements with query-ids
[ https://issues.apache.org/jira/browse/IMPALA-6664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoram Thanga reassigned IMPALA-6664: Assignee: Zoram Thanga > Tag log statements with query-ids > - > > Key: IMPALA-6664 > URL: https://issues.apache.org/jira/browse/IMPALA-6664 > Project: IMPALA > Issue Type: Improvement > Components: Backend, Frontend >Affects Versions: Impala 2.12.0 >Reporter: bharath v >Assignee: Zoram Thanga >Priority: Major > Labels: supportability > > The idea is to include query-id in each logged statement in > coordinator/finsts/planner. This helps to collect query specific logs by just > grep'ing through the logs with the query-id. > This is hopefully easier to implement, now that we have the query debug info > available in the thread context [1]. > [1] https://issues.apache.org/jira/browse/IMPALA-3703 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Reopened] (IMPALA-5765) Flaky tpc-ds data loading
[
https://issues.apache.org/jira/browse/IMPALA-5765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Volker reopened IMPALA-5765:
-
> Flaky tpc-ds data loading
> -
>
> Key: IMPALA-5765
> URL: https://issues.apache.org/jira/browse/IMPALA-5765
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.10.0
>Reporter: Matthew Jacobs
>Assignee: Philip Zeyliger
>Priority: Critical
> Labels: flaky
>
> Saw this on a number of gerrit-verify-dryrun jobs:
> {code}
> 23:49:37 Loading TPC-DS data (logging to
> /home/ubuntu/Impala/logs/data_loading/load-tpcds.log)...
> 23:55:39 FAILED (Took: 6 min 2 sec)
> 23:55:39 'load-data tpcds core' failed. Tail of log:
> 23:55:39 ss_net_profit,
> 23:55:39 ss_sold_date_sk
> 23:55:39 from store_sales_unpartitioned
> 23:55:39 WHERE ss_sold_date_sk < 2451272
> 23:55:39 distribute by ss_sold_date_sk
> 23:55:39 INFO : Query ID =
> ubuntu_2017073123_26963c6a-a58b-4cad-b0c7-c3790f9b22dc
> 23:55:39 INFO : Total jobs = 1
> 23:55:39 INFO : Launching Job 1 out of 1
> 23:55:39 INFO : Starting task [Stage-1:MAPRED] in serial mode
> 23:55:39 INFO : Number of reduce tasks not specified. Estimated from input
> data size: 2
> 23:55:39 INFO : In order to change the average load for a reducer (in bytes):
> 23:55:39 INFO : set hive.exec.reducers.bytes.per.reducer=
> 23:55:39 INFO : In order to limit the maximum number of reducers:
> 23:55:39 INFO : set hive.exec.reducers.max=
> 23:55:39 INFO : In order to set a constant number of reducers:
> 23:55:39 INFO : set mapreduce.job.reduces=
> 23:55:39 INFO : number of splits:2
> 23:55:39 INFO : Submitting tokens for job: job_local1252085428_0826
> 23:55:39 INFO : The url to track the job: http://localhost:8080/
> 23:55:39 INFO : Job running in-process (local Hadoop)
> 23:55:39 INFO : 2017-07-31 23:55:06,606 Stage-1 map = 0%, reduce = 0%
> 23:55:39 INFO : 2017-07-31 23:55:13,609 Stage-1 map = 100%, reduce = 0%
> 23:55:39 INFO : 2017-07-31 23:55:28,621 Stage-1 map = 100%, reduce = 33%
> 23:55:39 ERROR : Ended Job = job_local1252085428_0826 with errors
> 23:55:39 ERROR : FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> 23:55:39 INFO : MapReduce Jobs Launched:
> 23:55:39 INFO : Stage-Stage-1: HDFS Read: 26483258512 HDFS Write:
> 19378762131 FAIL
> 23:55:39 INFO : Total MapReduce CPU Time Spent: 0 msec
> 23:55:39 INFO : Completed executing
> command(queryId=ubuntu_2017073123_26963c6a-a58b-4cad-b0c7-c3790f9b22dc);
> Time taken: 33.276 seconds
> 23:55:39 Error: Error while processing statement: FAILED: Execution Error,
> return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> (state=08S01,code=2)
> 23:55:39 java.sql.SQLException: Error while processing statement: FAILED:
> Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> 23:55:39 at
> org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:292)
> 23:55:39 at
> org.apache.hive.beeline.Commands.executeInternal(Commands.java:989)
> 23:55:39 at org.apache.hive.beeline.Commands.execute(Commands.java:1203)
> 23:55:39 at org.apache.hive.beeline.Commands.sql(Commands.java:1117)
> 23:55:39 at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1176)
> 23:55:39 at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:1010)
> 23:55:39 at org.apache.hive.beeline.BeeLine.executeFile(BeeLine.java:987)
> 23:55:39 at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:914)
> 23:55:39 at
> org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:518)
> 23:55:39 at org.apache.hive.beeline.BeeLine.main(BeeLine.java:501)
> 23:55:39 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 23:55:39 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 23:55:39 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 23:55:39 at java.lang.reflect.Method.invoke(Method.java:606)
> 23:55:39 at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> 23:55:39 at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> 23:55:39
> 23:55:39 Closing: 0: jdbc:hive2://localhost:11050/default;auth=none
> 23:55:39 Error executing file from Hive: load-tpcds-core-hive-generated.sql
> 23:55:39 Error in /home/ubuntu/Impala/testdata/bin/create-load-data.sh at
> line 48: LOAD_DATA_ARGS=""
> {code}
> https://jenkins.impala.io/job/ubuntu-14.04-from-scratch/1827/
> It's been reported a few times in the last week. Here's another failed job
> reported on dev@:
> https://jenkins.impala.io/job/ubuntu-14.04-from-scratch/1807/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (IMPALA-5765) Flaky tpc-ds data loading
[
https://issues.apache.org/jira/browse/IMPALA-5765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509874#comment-16509874
]
Lars Volker commented on IMPALA-5765:
-
I saw this again today.
> Flaky tpc-ds data loading
> -
>
> Key: IMPALA-5765
> URL: https://issues.apache.org/jira/browse/IMPALA-5765
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.10.0
>Reporter: Matthew Jacobs
>Assignee: Philip Zeyliger
>Priority: Critical
> Labels: flaky
>
> Saw this on a number of gerrit-verify-dryrun jobs:
> {code}
> 23:49:37 Loading TPC-DS data (logging to
> /home/ubuntu/Impala/logs/data_loading/load-tpcds.log)...
> 23:55:39 FAILED (Took: 6 min 2 sec)
> 23:55:39 'load-data tpcds core' failed. Tail of log:
> 23:55:39 ss_net_profit,
> 23:55:39 ss_sold_date_sk
> 23:55:39 from store_sales_unpartitioned
> 23:55:39 WHERE ss_sold_date_sk < 2451272
> 23:55:39 distribute by ss_sold_date_sk
> 23:55:39 INFO : Query ID =
> ubuntu_2017073123_26963c6a-a58b-4cad-b0c7-c3790f9b22dc
> 23:55:39 INFO : Total jobs = 1
> 23:55:39 INFO : Launching Job 1 out of 1
> 23:55:39 INFO : Starting task [Stage-1:MAPRED] in serial mode
> 23:55:39 INFO : Number of reduce tasks not specified. Estimated from input
> data size: 2
> 23:55:39 INFO : In order to change the average load for a reducer (in bytes):
> 23:55:39 INFO : set hive.exec.reducers.bytes.per.reducer=
> 23:55:39 INFO : In order to limit the maximum number of reducers:
> 23:55:39 INFO : set hive.exec.reducers.max=
> 23:55:39 INFO : In order to set a constant number of reducers:
> 23:55:39 INFO : set mapreduce.job.reduces=
> 23:55:39 INFO : number of splits:2
> 23:55:39 INFO : Submitting tokens for job: job_local1252085428_0826
> 23:55:39 INFO : The url to track the job: http://localhost:8080/
> 23:55:39 INFO : Job running in-process (local Hadoop)
> 23:55:39 INFO : 2017-07-31 23:55:06,606 Stage-1 map = 0%, reduce = 0%
> 23:55:39 INFO : 2017-07-31 23:55:13,609 Stage-1 map = 100%, reduce = 0%
> 23:55:39 INFO : 2017-07-31 23:55:28,621 Stage-1 map = 100%, reduce = 33%
> 23:55:39 ERROR : Ended Job = job_local1252085428_0826 with errors
> 23:55:39 ERROR : FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> 23:55:39 INFO : MapReduce Jobs Launched:
> 23:55:39 INFO : Stage-Stage-1: HDFS Read: 26483258512 HDFS Write:
> 19378762131 FAIL
> 23:55:39 INFO : Total MapReduce CPU Time Spent: 0 msec
> 23:55:39 INFO : Completed executing
> command(queryId=ubuntu_2017073123_26963c6a-a58b-4cad-b0c7-c3790f9b22dc);
> Time taken: 33.276 seconds
> 23:55:39 Error: Error while processing statement: FAILED: Execution Error,
> return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> (state=08S01,code=2)
> 23:55:39 java.sql.SQLException: Error while processing statement: FAILED:
> Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> 23:55:39 at
> org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:292)
> 23:55:39 at
> org.apache.hive.beeline.Commands.executeInternal(Commands.java:989)
> 23:55:39 at org.apache.hive.beeline.Commands.execute(Commands.java:1203)
> 23:55:39 at org.apache.hive.beeline.Commands.sql(Commands.java:1117)
> 23:55:39 at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1176)
> 23:55:39 at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:1010)
> 23:55:39 at org.apache.hive.beeline.BeeLine.executeFile(BeeLine.java:987)
> 23:55:39 at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:914)
> 23:55:39 at
> org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:518)
> 23:55:39 at org.apache.hive.beeline.BeeLine.main(BeeLine.java:501)
> 23:55:39 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 23:55:39 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 23:55:39 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 23:55:39 at java.lang.reflect.Method.invoke(Method.java:606)
> 23:55:39 at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> 23:55:39 at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> 23:55:39
> 23:55:39 Closing: 0: jdbc:hive2://localhost:11050/default;auth=none
> 23:55:39 Error executing file from Hive: load-tpcds-core-hive-generated.sql
> 23:55:39 Error in /home/ubuntu/Impala/testdata/bin/create-load-data.sh at
> line 48: LOAD_DATA_ARGS=""
> {code}
> https://jenkins.impala.io/job/ubuntu-14.04-from-scratch/1827/
> It's been reported a few times in the last week. Here's another failed job
> reported on dev@:
> https://jenkins.impala.io/job/ubuntu-14.04-from-scratch/1807/
--
This message
[jira] [Updated] (IMPALA-7145) Memory leak when looping test_spilling_naaj
[
https://issues.apache.org/jira/browse/IMPALA-7145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-7145:
--
Affects Version/s: Impala 2.0
Impala 2.3.0
Impala 2.5.0
Impala 2.4.0
Impala 2.6.0
Impala 2.7.0
Impala 2.8.0
Impala 2.9.0
Impala 2.10.0
Impala 2.11.0
Impala 2.12.0
2.1.0
2.2.0
The bug was present in all versions of the disk spill encryption code:
[https://github.com/apache/impala/commit/41de2e167a1083bd991864d10f7eae976abc9812#diff-7deeb73dab3e893393aa665cb0eb0858R900]
It's not clear if IMPALA-6128 made it worse or not.
> Memory leak when looping test_spilling_naaj
> ---
>
> Key: IMPALA-7145
> URL: https://issues.apache.org/jira/browse/IMPALA-7145
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.0, Impala 2.3.0, Impala 2.5.0, Impala 2.4.0,
> Impala 2.6.0, Impala 2.7.0, Impala 2.8.0, Impala 2.9.0, Impala 2.10.0, Impala
> 2.11.0, Impala 2.12.0, Impala 2.13.0, Impala 3.1.0, 2.2.0, 2.1.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: resource-management
>
> I tried looping this test for a while to flush out a different issue and
> discovered a memory leak:
> {noformat}
> while impala-py.test tests/query_test/test_spilling.py -k naaj -n4 --verbose;
> do date; done
> {noformat}
> Eventually the test fails with "Memory Limit Exceeded" and there is a huge
> amount of untracked memory reported on /memz.
> I tried running the heap growth profiler and the only suspicious thing I saw
> was CRYPTO_malloc, similar to IMPALA-6792. My current hypothesis is that it's
> something related to disk spill encryption and we haven't noticed it before
> because we had disk spill encryption off.
> If I disable encryption "start-impala-cluster.py
> --impalad_args=--disk_spill_encryption=false" then I don't see the leak.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Closed] (IMPALA-6408) [DOCS] Description of "shuffle" hint does not mention changes in IMPALA-3930
[ https://issues.apache.org/jira/browse/IMPALA-6408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni closed IMPALA-6408. --- Resolution: Fixed Fix Version/s: Impala 3.1.0 Impala 2.13.0 > [DOCS] Description of "shuffle" hint does not mention changes in IMPALA-3930 > > > Key: IMPALA-6408 > URL: https://issues.apache.org/jira/browse/IMPALA-6408 > Project: IMPALA > Issue Type: Bug > Components: Docs >Reporter: Csaba Ringhofer >Assignee: Alex Rodoni >Priority: Minor > Fix For: Impala 2.13.0, Impala 3.1.0 > > > The change in IMPALA-3930 states that if only one partition is written > (because all partitioning columns are constant or the target table is not > partitioned), then the "shuffle" hint leads to a plan where all rows are > merged at the coordinator where the table sink is executed. > The documentation of the "shuffle" hint does not mention this behavior. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-7150) Crash in Reflection::invoke_method()
[
https://issues.apache.org/jira/browse/IMPALA-7150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Volker reassigned IMPALA-7150:
---
Assignee: Vuk Ercegovac
> Crash in Reflection::invoke_method()
>
>
> Key: IMPALA-7150
> URL: https://issues.apache.org/jira/browse/IMPALA-7150
> Project: IMPALA
> Issue Type: Bug
> Components: Backend, Frontend
>Affects Versions: Impala 2.13.0, Impala 3.1.0
>Reporter: Lars Volker
>Assignee: Vuk Ercegovac
>Priority: Critical
> Labels: broken-build, crash, flaky
>
> We've seen several crashes in the past involving
> {{Reflection::invoke_method()}}. The stack trace looks like this:
> {noformat}
> CORE: ./core.1527786725.23658.impalad
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/be/buil'.
> Program terminated with signal 6, Aborted.
> #0 0x7fca4ff4a5c9 in raise () from /lib64/libc.so.6
> To enable execution of this file add
> add-auto-load-safe-path
> /data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/Impala-Toolchain/gcc-4.9.2/lib64/libstdc++.so.6.0.20-gdb.py
> line to your configuration file "/var/lib/jenkins/.gdbinit".
> To completely disable this security protection add
> set auto-load safe-path /
> line to your configuration file "/var/lib/jenkins/.gdbinit".
> For more information about this security protection see the
> "Auto-loading safe path" section in the GDB manual. E.g., run from the shell:
> info "(gdb)Auto-loading safe path"
> #0 0x7fca4ff4a5c9 in raise () from /lib64/libc.so.6
> #1 0x7fca4ff4bcd8 in abort () from /lib64/libc.so.6
> #2 0x7fca52db61a5 in os::abort(bool) () from
> /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
> #3 0x7fca52f46843 in VMError::report_and_die() () from
> /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
> #4 0x7fca52dbb562 in JVM_handle_linux_signal () from
> /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
> #5 0x7fca52db24f3 in signalHandler(int, siginfo*, void*) () from
> /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
> #6
> #7 0x7fca52e29621 in Reflection::invoke(instanceKlassHandle,
> methodHandle, Handle, bool, objArrayHandle, BasicType, objArrayHandle, bool,
> Thread*) () from
> /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
> #8 0x7fca52e2cd27 in Reflection::invoke_method(oopDesc*, Handle,
> objArrayHandle, Thread*) () from
> /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
> #9 0x7fca52bc339b in JVM_InvokeMethod () from
> /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
> #10 0x7fca36f49d6d in ?? ()
> #11 0x0006600753f0 in ?? ()
> #12 0x00066006f000 in ?? ()
> #13 0x000660079328 in ?? ()
> #14 0x00065f3e6550 in ?? ()
> #15 0x0007c0733028 in ?? ()
> #16 0x7fc98ff20bf0 in ?? ()
> #17 0x00039d85edd0 in ?? ()
> #18 0x7fca36f48cb6 in ?? ()
> #19 0x0006600753f0 in ?? ()
> #20 0x7fca36f48f0c in ?? ()
> #21 0x00140001 in ?? ()
> #22 0x in ?? ()
> {noformat}
> From the core file I was unable to figure out what had happened.
> Unfortunately we also did not preserver the hs_err_pid* files until recently
> (fixed [here|https://gerrit.cloudera.org/#/c/10594/]).
> We should track this down when it happens again.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7150) Crash in Reflection::invoke_method()
[
https://issues.apache.org/jira/browse/IMPALA-7150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509759#comment-16509759
]
Lars Volker commented on IMPALA-7150:
-
[~vukercegovac] - I'm assigning this to you, thinking you might have an idea
where to look next. Please feel free to find someone else or assign back to me
if you're swamped. Thanks :)
> Crash in Reflection::invoke_method()
>
>
> Key: IMPALA-7150
> URL: https://issues.apache.org/jira/browse/IMPALA-7150
> Project: IMPALA
> Issue Type: Bug
> Components: Backend, Frontend
>Affects Versions: Impala 2.13.0, Impala 3.1.0
>Reporter: Lars Volker
>Assignee: Vuk Ercegovac
>Priority: Critical
> Labels: broken-build, crash, flaky
>
> We've seen several crashes in the past involving
> {{Reflection::invoke_method()}}. The stack trace looks like this:
> {noformat}
> CORE: ./core.1527786725.23658.impalad
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/be/buil'.
> Program terminated with signal 6, Aborted.
> #0 0x7fca4ff4a5c9 in raise () from /lib64/libc.so.6
> To enable execution of this file add
> add-auto-load-safe-path
> /data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/Impala-Toolchain/gcc-4.9.2/lib64/libstdc++.so.6.0.20-gdb.py
> line to your configuration file "/var/lib/jenkins/.gdbinit".
> To completely disable this security protection add
> set auto-load safe-path /
> line to your configuration file "/var/lib/jenkins/.gdbinit".
> For more information about this security protection see the
> "Auto-loading safe path" section in the GDB manual. E.g., run from the shell:
> info "(gdb)Auto-loading safe path"
> #0 0x7fca4ff4a5c9 in raise () from /lib64/libc.so.6
> #1 0x7fca4ff4bcd8 in abort () from /lib64/libc.so.6
> #2 0x7fca52db61a5 in os::abort(bool) () from
> /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
> #3 0x7fca52f46843 in VMError::report_and_die() () from
> /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
> #4 0x7fca52dbb562 in JVM_handle_linux_signal () from
> /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
> #5 0x7fca52db24f3 in signalHandler(int, siginfo*, void*) () from
> /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
> #6
> #7 0x7fca52e29621 in Reflection::invoke(instanceKlassHandle,
> methodHandle, Handle, bool, objArrayHandle, BasicType, objArrayHandle, bool,
> Thread*) () from
> /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
> #8 0x7fca52e2cd27 in Reflection::invoke_method(oopDesc*, Handle,
> objArrayHandle, Thread*) () from
> /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
> #9 0x7fca52bc339b in JVM_InvokeMethod () from
> /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
> #10 0x7fca36f49d6d in ?? ()
> #11 0x0006600753f0 in ?? ()
> #12 0x00066006f000 in ?? ()
> #13 0x000660079328 in ?? ()
> #14 0x00065f3e6550 in ?? ()
> #15 0x0007c0733028 in ?? ()
> #16 0x7fc98ff20bf0 in ?? ()
> #17 0x00039d85edd0 in ?? ()
> #18 0x7fca36f48cb6 in ?? ()
> #19 0x0006600753f0 in ?? ()
> #20 0x7fca36f48f0c in ?? ()
> #21 0x00140001 in ?? ()
> #22 0x in ?? ()
> {noformat}
> From the core file I was unable to figure out what had happened.
> Unfortunately we also did not preserver the hs_err_pid* files until recently
> (fixed [here|https://gerrit.cloudera.org/#/c/10594/]).
> We should track this down when it happens again.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-5706) Parallelise read I/O in sorter
[ https://issues.apache.org/jira/browse/IMPALA-5706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509278#comment-16509278 ] ASF subversion and git services commented on IMPALA-5706: - Commit 88366bab07db3defa14ce00ac4f02ad97ec54176 in impala's branch refs/heads/master from [~gaborkaszab] [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=88366ba ] IMPALA-5706: Spilling sort optimisations This patch covers multiple changes with the purpose of optimizing spilling sort mechanism: - Remove the hard-coded maximum limit of buffers that can be used for merging the sorted runs. Instead this number is calculated based on the available memory through buffer pool. - The already sorted runs are distributed more optimally between the last intermediate merge and the final merge to avoid that a heavy intermediate merge is followed by a light final merge. - Right before starting the merging phase Sorter tries to allocate additional memory through the buffer pool. - An output run is not allocated anymore for the final merge. Note, double-buffering the runs during a merge was also planned with this patch. However, performance testing showed that except some exotic queries with unreasonably small amount of buffer pool memory available double-buffering doesn't add to the overall performance. It's basically because the half of the available buffers have to be sacrificed to do double-buffering and as a result the merge tree can get deeper. In addition the amount of I/O wait time is not reaching the level where double-buffering could countervail the reduced number of runs during a particular merge. Performance measurements were made during manual testing to verify that this is in fact an optimization: - In case doing a sort on top of a join when working with a restricted amount of memory then the Sort node successfully allocates additional memory right before the merging phase. This is feasible because once Join finishes sending new input data and calls InputDone() then it releases memory that can be picked up by the Sorter. This results in shallower merging trees (more runs grabbed for a merge). - On a multi-node cluster I verified that in cases when at least one merging step is done then this change reduces the execution time for sorts. - The more merging steps are done the bigger the performance gain is compared to the baseline. Change-Id: I74857c1694802e81f1cfc765d2b4e8bc644387f9 Reviewed-on: http://gerrit.cloudera.org:8080/9943 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Parallelise read I/O in sorter > -- > > Key: IMPALA-5706 > URL: https://issues.apache.org/jira/browse/IMPALA-5706 > Project: IMPALA > Issue Type: Sub-task > Components: Backend >Affects Versions: Impala 2.10.0 >Reporter: Tim Armstrong >Assignee: Gabor Kaszab >Priority: Major > Labels: resource-management, spill > Fix For: Impala 3.1.0 > > > IMPALA-3200 offers an opportunity to improve the spilling sort algorithm: > * Use the reliability of reservations to select the most efficient order to > conduct merges in (rather than greedily trying to maximise the fan-in of the > current merge). We want to minimise the depth of the merge tree, then > structure the tree based on the preferred fan-in. > * Do multiple-buffering of the stream being written (this happens > automatically if there are free buffers in the BufferPool client). > * Do multiple-buffering of the streams being read, instead of blocking on > read I/O frequently. > More concretely, the idea is to implement double-buffering of spilled input > runs by calling BufferPool::Pin() early to prefetch the second page in each > input Run. Currently only one page per input run is pinned, which means that > the sorter frequently blocks on I/O. > I'd suggest doing this in two steps. > The first step is to change how the fan-in of each merge run is selected. We > know the number of runs to be merged and the buffer reservation that is > available, so we can compute the maximum possible fan-in of each merge step > (assuming 1 buffer for the output run and 1 buffer for each input run to the > merge). We can then calculate the minimum number of rounds of merging > required and, based on that, decide how the runs should be merged (you could > think about it as a tree of merge operations). I think we want to reduce the > number of bytes written to disk. E.g. if we have 5 buffers and 8 input runs, > we should merge input runs (1,2,3,4) then merge that intermediate runs with > runs (5,6,7). It's reasonable to assume that the input runs are all > approximate the same size. > ee53ddb389549247f5bfe760d446dc7b3b963a29 actually removed some logic along >
[jira] [Resolved] (IMPALA-5706) Parallelise read I/O in sorter
[
https://issues.apache.org/jira/browse/IMPALA-5706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Kaszab resolved IMPALA-5706.
--
Resolution: Fixed
Fix Version/s: Impala 3.1.0
https://gerrit.cloudera.org/#/c/9943/
> Parallelise read I/O in sorter
> --
>
> Key: IMPALA-5706
> URL: https://issues.apache.org/jira/browse/IMPALA-5706
> Project: IMPALA
> Issue Type: Sub-task
> Components: Backend
>Affects Versions: Impala 2.10.0
>Reporter: Tim Armstrong
>Assignee: Gabor Kaszab
>Priority: Major
> Labels: resource-management, spill
> Fix For: Impala 3.1.0
>
>
> IMPALA-3200 offers an opportunity to improve the spilling sort algorithm:
> * Use the reliability of reservations to select the most efficient order to
> conduct merges in (rather than greedily trying to maximise the fan-in of the
> current merge). We want to minimise the depth of the merge tree, then
> structure the tree based on the preferred fan-in.
> * Do multiple-buffering of the stream being written (this happens
> automatically if there are free buffers in the BufferPool client).
> * Do multiple-buffering of the streams being read, instead of blocking on
> read I/O frequently.
> More concretely, the idea is to implement double-buffering of spilled input
> runs by calling BufferPool::Pin() early to prefetch the second page in each
> input Run. Currently only one page per input run is pinned, which means that
> the sorter frequently blocks on I/O.
> I'd suggest doing this in two steps.
> The first step is to change how the fan-in of each merge run is selected. We
> know the number of runs to be merged and the buffer reservation that is
> available, so we can compute the maximum possible fan-in of each merge step
> (assuming 1 buffer for the output run and 1 buffer for each input run to the
> merge). We can then calculate the minimum number of rounds of merging
> required and, based on that, decide how the runs should be merged (you could
> think about it as a tree of merge operations). I think we want to reduce the
> number of bytes written to disk. E.g. if we have 5 buffers and 8 input runs,
> we should merge input runs (1,2,3,4) then merge that intermediate runs with
> runs (5,6,7). It's reasonable to assume that the input runs are all
> approximate the same size.
> ee53ddb389549247f5bfe760d446dc7b3b963a29 actually removed some logic along
> those lines because it didn't work with the old buffer management scheme. The
> logic before that commit might provide some ideas. There are also some
> related TODOs in Sorter::MergeIntermediateRuns() and Sorter::CreateMerger()
> to simplify how the number of input runs is decided and how the merger is set
> up:
> {code}
> // TODO: once we have reliable reservations (IMPALA-3200), we should
> calculate this
> // based on the available reservations.
>
> // TODO: this isn't optimal: we could defer creating the merged run if we
> have
> // reliable reservations (IMPALA-3200).
> ...
> // TODO: IMPALA-3200: we should not need this logic once we have
> reliable
> // reservations (IMPALA-3200).
> {code}
> The second step would be to adjust the logic from the first step to reserve 2
> buffers per input and output run and then implement the logic to call Pin()
> earlier to prefetch the page after the current page.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
