[jira] [Commented] (IMPALA-7169) TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
[
https://issues.apache.org/jira/browse/IMPALA-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514604#comment-16514604
]
Tianyi Wang commented on IMPALA-7169:
-
[~tarmstrong] To check the most recent checkpoint we need to parse the file
names in Trash and choose the one with the latest time stamp. HDFS might create
a new checkpoint/remove an old one concurrently so we still need to come up
with a way to lock the trash folder, or retry something in a loop, or else.
> TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
> --
>
> Key: IMPALA-7169
> URL: https://issues.apache.org/jira/browse/IMPALA-7169
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.13.0
>Reporter: Lars Volker
>Assignee: Tianyi Wang
>Priority: Critical
> Labels: broken-build, flaky
>
> {noformat}
> F
> metadata/test_hdfs_encryption.py::TestHdfsEncryption::()::test_drop_partition_encrypt
> metadata/test_hdfs_encryption.py:202: in test_drop_partition_encrypt
> assert self.hdfs_client.exists(
> E assert 0xba6ee90>>('/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt')
> E+ where > =
> .exists
> E+where 0xba6ee90> = 0xba6ed50>.hdfs_client
> E+ and
> '/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt'
> = ('jenkins',
> 'test_encryption_db')
> E+where =
> '/user/{0}/.Trash/Current/test-warehouse/{1}.db/t1/j=1/j1.txt'.format
> E+and 'jenkins' = ()
> E+ where = getpass.getuser
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7169) TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
[
https://issues.apache.org/jira/browse/IMPALA-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514600#comment-16514600
]
Tim Armstrong commented on IMPALA-7169:
---
It might not be so bad though to have a helper function like
check_file_in_trash(user, path) or whatever, but I guess really we just want to
set it to something reasonable then move on with our lives :)
> TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
> --
>
> Key: IMPALA-7169
> URL: https://issues.apache.org/jira/browse/IMPALA-7169
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.13.0
>Reporter: Lars Volker
>Assignee: Tianyi Wang
>Priority: Critical
> Labels: broken-build, flaky
>
> {noformat}
> F
> metadata/test_hdfs_encryption.py::TestHdfsEncryption::()::test_drop_partition_encrypt
> metadata/test_hdfs_encryption.py:202: in test_drop_partition_encrypt
> assert self.hdfs_client.exists(
> E assert 0xba6ee90>>('/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt')
> E+ where > =
> .exists
> E+where 0xba6ee90> = 0xba6ed50>.hdfs_client
> E+ and
> '/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt'
> = ('jenkins',
> 'test_encryption_db')
> E+where =
> '/user/{0}/.Trash/Current/test-warehouse/{1}.db/t1/j=1/j1.txt'.format
> E+and 'jenkins' = ()
> E+ where = getpass.getuser
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7169) TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
[
https://issues.apache.org/jira/browse/IMPALA-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514599#comment-16514599
]
Tim Armstrong commented on IMPALA-7169:
---
I don't think there's a major downside for a test cluster, except junk might
accumulate more than before (we can probably live with this TBH).
> TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
> --
>
> Key: IMPALA-7169
> URL: https://issues.apache.org/jira/browse/IMPALA-7169
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.13.0
>Reporter: Lars Volker
>Assignee: Tianyi Wang
>Priority: Critical
> Labels: broken-build, flaky
>
> {noformat}
> F
> metadata/test_hdfs_encryption.py::TestHdfsEncryption::()::test_drop_partition_encrypt
> metadata/test_hdfs_encryption.py:202: in test_drop_partition_encrypt
> assert self.hdfs_client.exists(
> E assert 0xba6ee90>>('/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt')
> E+ where > =
> .exists
> E+where 0xba6ee90> = 0xba6ed50>.hdfs_client
> E+ and
> '/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt'
> = ('jenkins',
> 'test_encryption_db')
> E+where =
> '/user/{0}/.Trash/Current/test-warehouse/{1}.db/t1/j=1/j1.txt'.format
> E+and 'jenkins' = ()
> E+ where = getpass.getuser
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7169) TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
[
https://issues.apache.org/jira/browse/IMPALA-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514596#comment-16514596
]
Tianyi Wang commented on IMPALA-7169:
-
[~tarmstrong] We can. But test_drop_partition_encrypt is not the only test that
looks into trash. The easier option it to set fs.trash.interval to a large
value - do you know any downside of doing so?
> TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
> --
>
> Key: IMPALA-7169
> URL: https://issues.apache.org/jira/browse/IMPALA-7169
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.13.0
>Reporter: Lars Volker
>Assignee: Tianyi Wang
>Priority: Critical
> Labels: broken-build, flaky
>
> {noformat}
> F
> metadata/test_hdfs_encryption.py::TestHdfsEncryption::()::test_drop_partition_encrypt
> metadata/test_hdfs_encryption.py:202: in test_drop_partition_encrypt
> assert self.hdfs_client.exists(
> E assert 0xba6ee90>>('/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt')
> E+ where > =
> .exists
> E+where 0xba6ee90> = 0xba6ed50>.hdfs_client
> E+ and
> '/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt'
> = ('jenkins',
> 'test_encryption_db')
> E+where =
> '/user/{0}/.Trash/Current/test-warehouse/{1}.db/t1/j=1/j1.txt'.format
> E+and 'jenkins' = ()
> E+ where = getpass.getuser
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7169) TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
[
https://issues.apache.org/jira/browse/IMPALA-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514594#comment-16514594
]
Tim Armstrong commented on IMPALA-7169:
---
Can we modify the tests that look in the trash to check both "Current" and the
most recent checkpoint?
> TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
> --
>
> Key: IMPALA-7169
> URL: https://issues.apache.org/jira/browse/IMPALA-7169
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.13.0
>Reporter: Lars Volker
>Assignee: Tianyi Wang
>Priority: Critical
> Labels: broken-build, flaky
>
> {noformat}
> F
> metadata/test_hdfs_encryption.py::TestHdfsEncryption::()::test_drop_partition_encrypt
> metadata/test_hdfs_encryption.py:202: in test_drop_partition_encrypt
> assert self.hdfs_client.exists(
> E assert 0xba6ee90>>('/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt')
> E+ where > =
> .exists
> E+where 0xba6ee90> = 0xba6ed50>.hdfs_client
> E+ and
> '/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt'
> = ('jenkins',
> 'test_encryption_db')
> E+where =
> '/user/{0}/.Trash/Current/test-warehouse/{1}.db/t1/j=1/j1.txt'.format
> E+and 'jenkins' = ()
> E+ where = getpass.getuser
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-5937) Docs are missing some query options
[ https://issues.apache.org/jira/browse/IMPALA-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni updated IMPALA-5937: Description: I noticed that the following options show up in "SET" in impala-shell but don't have corresponding documentation entries. I know BUFFER_POOL_LIMIT is mentioned in IMPALA-5655. --BUFFER_POOL_LIMIT-- -DECIMAL_V2- -DEFAULT_SPILLABLE_BUFFER_SIZE- -DISABLE_CODEGEN_ROWS_THRESHOLD- ENABLE_EXPR_REWRITES -MAX_ROW_SIZE- -MIN_SPILLABLE_BUFFER_SIZE- -PARQUET_ARRAY_RESOLUTION- PARQUET_DICTIONARY_FILTERING PARQUET_READ_STATISTICS -STRICT_MODE /Dev option that Greg and Tim recommended not to doc- was: I noticed that the following options show up in "SET" in impala-shell but don't have corresponding documentation entries. I know BUFFER_POOL_LIMIT is mentioned in IMPALA-5655. --BUFFER_POOL_LIMIT-- -DECIMAL_V2- -DEFAULT_SPILLABLE_BUFFER_SIZE- DISABLE_CODEGEN_ROWS_THRESHOLD ENABLE_EXPR_REWRITES -MAX_ROW_SIZE- -MIN_SPILLABLE_BUFFER_SIZE- -PARQUET_ARRAY_RESOLUTION- PARQUET_DICTIONARY_FILTERING PARQUET_READ_STATISTICS -STRICT_MODE /Dev option that Greg and Tim recommended not to doc- > Docs are missing some query options > --- > > Key: IMPALA-5937 > URL: https://issues.apache.org/jira/browse/IMPALA-5937 > Project: IMPALA > Issue Type: Bug > Components: Docs >Reporter: Philip Zeyliger >Assignee: Alex Rodoni >Priority: Major > > I noticed that the following options show up in "SET" in impala-shell but > don't have corresponding documentation entries. I know BUFFER_POOL_LIMIT is > mentioned in IMPALA-5655. > --BUFFER_POOL_LIMIT-- > -DECIMAL_V2- > -DEFAULT_SPILLABLE_BUFFER_SIZE- > -DISABLE_CODEGEN_ROWS_THRESHOLD- > ENABLE_EXPR_REWRITES > -MAX_ROW_SIZE- > -MIN_SPILLABLE_BUFFER_SIZE- > -PARQUET_ARRAY_RESOLUTION- > PARQUET_DICTIONARY_FILTERING > PARQUET_READ_STATISTICS > -STRICT_MODE /Dev option that Greg and Tim recommended not to doc- -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7171) Add docs for Kudu insert partitioning/sorting
[ https://issues.apache.org/jira/browse/IMPALA-7171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514584#comment-16514584 ] Alex Rodoni commented on IMPALA-7171: - https://gerrit.cloudera.org/#/c/10737/ > Add docs for Kudu insert partitioning/sorting > - > > Key: IMPALA-7171 > URL: https://issues.apache.org/jira/browse/IMPALA-7171 > Project: IMPALA > Issue Type: Improvement > Components: Docs >Reporter: Thomas Tauber-Marshall >Assignee: Alex Rodoni >Priority: Major > Labels: docs > > On the page: > http://impala.apache.org/docs/build3x/html/topics/impala_kudu.html, at the > end of the section: "Impala DML Support for Kudu Tables (INSERT, UPDATE, > DELETE, UPSERT)", we should add text like: > Starting from Impala 2.9, Impala will automatically add a partition and sort > step to INSERTs before sending the rows to Kudu. Since Kudu partitions and > sorts rows on write, pre-partitioning and sorting takes some of the load off > of Kudu, and helps ensure that large INSERTs complete without timing out, but > it may slow down the end-to-end performance of the INSERT. Starting from > Impala 2.10, the hints "/* +noshuffle,noclustered */" may be used to turn > this pre-partitioning and sorting off. Additionally, since sorting may > consume a lot of memory, users should consider setting a "mem_limit" for > these queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6153) Prevent Coordinator::UpdateFilter() running after query exec resources are released
[ https://issues.apache.org/jira/browse/IMPALA-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514572#comment-16514572 ] Tim Armstrong commented on IMPALA-6153: --- Yeah I agree with Dan. I think the reader/writer lock would have the right effect (I think?) but we should have a clear explanation for how it fits into the state machine of the class and then see if the reader/writer lock is the simplest possible mechanism to achieve the effect. I think there's one tricky issue around liveness that we should think through. Potentially a thread in UpdateFilter could be in an RPC and take a while to exit and clean itself up. Blocking progress of another thread can have other consequences, e.g. if it's being called from an RPC handler. It might be the right thing to do, but we should make sure to do the right thing. > Prevent Coordinator::UpdateFilter() running after query exec resources are > released > --- > > Key: IMPALA-6153 > URL: https://issues.apache.org/jira/browse/IMPALA-6153 > Project: IMPALA > Issue Type: Bug > Components: Backend >Reporter: Sailesh Mukil >Assignee: Pooja Nilangekar >Priority: Major > Labels: query-lifecycle, runtime-filters > > Coordinator::UpdateFilter() and CoordinatorBackendState::PublishFilter() run > independent of the lifecycle of any fragment instance. This is problematic > during query teardown. > Specifically we should not release resources for a query if any one of those > above functions are still running for that query and we also should not not > start running the above methods after resources are released for the query. > Also, the 'rpc_params' in UpdateFilter() could potentially hold large amounts > of untracked memory, so we should track it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Comment Edited] (IMPALA-6153) Prevent Coordinator::UpdateFilter() running after query exec resources are released
[
https://issues.apache.org/jira/browse/IMPALA-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514558#comment-16514558
]
Dan Hecht edited comment on IMPALA-6153 at 6/16/18 12:14 AM:
-
We've been trying to clarify query lifecycle events and also differentiate
control structures from resources, rather than relying on adding ad-hoc locks.
Control structures remain alive for the life of the owning object (either
Coordinator on coordinator side or QueryState on executor side), whereas
resources are freed soon after query execution has finished. So I think we
should step back and see if we can rationalize the dependencies and what kind
of dependencies we really have, before resorting to adding a lock (not saying
we can't do that for sure, but let's consider the lifecycle of these things
first).
Note also that we already have {{filter_lock_}}.
was (Author: dhecht):
We've been trying to clarify query lifecycle events and also differentiate
control structures from resources, rather than relying on adding ad-hoc locks.
Control structures remain alive for the life of the owning object (either
Coordinator on coordinator side or QueryState on executor side), whereas
resources are freed soon after query execution has finished. So I think we
should step back and see if we can rationalize the dependencies and what kind
of dependencies we really have, before resorting to adding a lock (not saying
we can't do that for sure, but let's consider the lifecycle of these things
first).
> Prevent Coordinator::UpdateFilter() running after query exec resources are
> released
> ---
>
> Key: IMPALA-6153
> URL: https://issues.apache.org/jira/browse/IMPALA-6153
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Reporter: Sailesh Mukil
>Assignee: Pooja Nilangekar
>Priority: Major
> Labels: query-lifecycle, runtime-filters
>
> Coordinator::UpdateFilter() and CoordinatorBackendState::PublishFilter() run
> independent of the lifecycle of any fragment instance. This is problematic
> during query teardown.
> Specifically we should not release resources for a query if any one of those
> above functions are still running for that query and we also should not not
> start running the above methods after resources are released for the query.
> Also, the 'rpc_params' in UpdateFilter() could potentially hold large amounts
> of untracked memory, so we should track it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6153) Prevent Coordinator::UpdateFilter() running after query exec resources are released
[ https://issues.apache.org/jira/browse/IMPALA-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514558#comment-16514558 ] Dan Hecht commented on IMPALA-6153: --- We've been trying to clarify query lifecycle events and also differentiate control structures from resources, rather than relying on adding ad-hoc locks. Control structures remain alive for the life of the owning object (either Coordinator on coordinator side or QueryState on executor side), whereas resources are freed soon after query execution has finished. So I think we should step back and see if we can rationalize the dependencies and what kind of dependencies we really have, before resorting to adding a lock (not saying we can't do that for sure, but let's consider the lifecycle of these things first). > Prevent Coordinator::UpdateFilter() running after query exec resources are > released > --- > > Key: IMPALA-6153 > URL: https://issues.apache.org/jira/browse/IMPALA-6153 > Project: IMPALA > Issue Type: Bug > Components: Backend >Reporter: Sailesh Mukil >Assignee: Pooja Nilangekar >Priority: Major > Labels: query-lifecycle, runtime-filters > > Coordinator::UpdateFilter() and CoordinatorBackendState::PublishFilter() run > independent of the lifecycle of any fragment instance. This is problematic > during query teardown. > Specifically we should not release resources for a query if any one of those > above functions are still running for that query and we also should not not > start running the above methods after resources are released for the query. > Also, the 'rpc_params' in UpdateFilter() could potentially hold large amounts > of untracked memory, so we should track it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Comment Edited] (IMPALA-7169) TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
[
https://issues.apache.org/jira/browse/IMPALA-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514539#comment-16514539
]
Tianyi Wang edited comment on IMPALA-7169 at 6/16/18 12:10 AM:
---
We set fs.trash.interval = 1440 so that HDFS supposedly shouldn't do the
checkpointing within a day, which doesn't work. HDFS will do the checkpointing
at the next UNIX time which is a multiple of 1440 * 60 * 1000. The failed build
happened to run this test at around 17:00:00 so it failed.
was (Author: tianyiwang):
We set fs.trash.interval = 1440 so that HDFS supposedly shouldn't do the
checkpointing within a day, which doesn't work. HDFS will do the checkpointing
at the next UNIX time which is a multiple of 1440 * 60. The failed build
happened to run this test at around 17:00:00 so it failed.
> TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
> --
>
> Key: IMPALA-7169
> URL: https://issues.apache.org/jira/browse/IMPALA-7169
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.13.0
>Reporter: Lars Volker
>Assignee: Tianyi Wang
>Priority: Critical
> Labels: broken-build, flaky
>
> {noformat}
> F
> metadata/test_hdfs_encryption.py::TestHdfsEncryption::()::test_drop_partition_encrypt
> metadata/test_hdfs_encryption.py:202: in test_drop_partition_encrypt
> assert self.hdfs_client.exists(
> E assert 0xba6ee90>>('/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt')
> E+ where > =
> .exists
> E+where 0xba6ee90> = 0xba6ed50>.hdfs_client
> E+ and
> '/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt'
> = ('jenkins',
> 'test_encryption_db')
> E+where =
> '/user/{0}/.Trash/Current/test-warehouse/{1}.db/t1/j=1/j1.txt'.format
> E+and 'jenkins' = ()
> E+ where = getpass.getuser
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Comment Edited] (IMPALA-7169) TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
[
https://issues.apache.org/jira/browse/IMPALA-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514539#comment-16514539
]
Tianyi Wang edited comment on IMPALA-7169 at 6/16/18 12:10 AM:
---
We set fs.trash.interval = 1440 so that HDFS supposedly shouldn't do the
checkpointing within a day, which doesn't work. HDFS will do the checkpointing
at the next UNIX time which is a multiple of 1440 * 60. The failed build
happened to run this test at around 17:00:00 so it failed.
was (Author: tianyiwang):
We set fs.trash.interval = 1440 so that HDFS supposedly shouldn't do the
checkpointing within a day.
The build took less than a day but HDFS did it when this test is running.
My current theory is that there are some leftover in the trash in our HDFS
snapshot. The timestamp (which is part of the filename) on those leftover is
older than a day so that HDFS thought more than a day has passed.
> TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
> --
>
> Key: IMPALA-7169
> URL: https://issues.apache.org/jira/browse/IMPALA-7169
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.13.0
>Reporter: Lars Volker
>Assignee: Tianyi Wang
>Priority: Critical
> Labels: broken-build, flaky
>
> {noformat}
> F
> metadata/test_hdfs_encryption.py::TestHdfsEncryption::()::test_drop_partition_encrypt
> metadata/test_hdfs_encryption.py:202: in test_drop_partition_encrypt
> assert self.hdfs_client.exists(
> E assert 0xba6ee90>>('/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt')
> E+ where > =
> .exists
> E+where 0xba6ee90> = 0xba6ed50>.hdfs_client
> E+ and
> '/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt'
> = ('jenkins',
> 'test_encryption_db')
> E+where =
> '/user/{0}/.Trash/Current/test-warehouse/{1}.db/t1/j=1/j1.txt'.format
> E+and 'jenkins' = ()
> E+ where = getpass.getuser
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6153) Prevent Coordinator::UpdateFilter() running after query exec resources are released
[ https://issues.apache.org/jira/browse/IMPALA-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514548#comment-16514548 ] Pooja Nilangekar commented on IMPALA-6153: -- I looked at the code and spoke to [~sailesh] this morning. One approach would be to added a reader/writer lock in the Coordinator. Each time the UpdateFilter function gets called, it would hold a reader lock throughout its execution and the ReleaseExecResources function would have to acquire a writer lock instead of the filter_lock_. I wanted to check if you think this approach would suffice or should I explore other directions? > Prevent Coordinator::UpdateFilter() running after query exec resources are > released > --- > > Key: IMPALA-6153 > URL: https://issues.apache.org/jira/browse/IMPALA-6153 > Project: IMPALA > Issue Type: Bug > Components: Backend >Reporter: Sailesh Mukil >Assignee: Pooja Nilangekar >Priority: Major > Labels: query-lifecycle, runtime-filters > > Coordinator::UpdateFilter() and CoordinatorBackendState::PublishFilter() run > independent of the lifecycle of any fragment instance. This is problematic > during query teardown. > Specifically we should not release resources for a query if any one of those > above functions are still running for that query and we also should not not > start running the above methods after resources are released for the query. > Also, the 'rpc_params' in UpdateFilter() could potentially hold large amounts > of untracked memory, so we should track it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Issue Comment Deleted] (IMPALA-6153) Prevent Coordinator::UpdateFilter() running after query exec resources are released
[ https://issues.apache.org/jira/browse/IMPALA-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pooja Nilangekar updated IMPALA-6153: - Comment: was deleted (was: I ) > Prevent Coordinator::UpdateFilter() running after query exec resources are > released > --- > > Key: IMPALA-6153 > URL: https://issues.apache.org/jira/browse/IMPALA-6153 > Project: IMPALA > Issue Type: Bug > Components: Backend >Reporter: Sailesh Mukil >Assignee: Pooja Nilangekar >Priority: Major > Labels: query-lifecycle, runtime-filters > > Coordinator::UpdateFilter() and CoordinatorBackendState::PublishFilter() run > independent of the lifecycle of any fragment instance. This is problematic > during query teardown. > Specifically we should not release resources for a query if any one of those > above functions are still running for that query and we also should not not > start running the above methods after resources are released for the query. > Also, the 'rpc_params' in UpdateFilter() could potentially hold large amounts > of untracked memory, so we should track it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6153) Prevent Coordinator::UpdateFilter() running after query exec resources are released
[ https://issues.apache.org/jira/browse/IMPALA-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514543#comment-16514543 ] Pooja Nilangekar commented on IMPALA-6153: -- I > Prevent Coordinator::UpdateFilter() running after query exec resources are > released > --- > > Key: IMPALA-6153 > URL: https://issues.apache.org/jira/browse/IMPALA-6153 > Project: IMPALA > Issue Type: Bug > Components: Backend >Reporter: Sailesh Mukil >Assignee: Pooja Nilangekar >Priority: Major > Labels: query-lifecycle, runtime-filters > > Coordinator::UpdateFilter() and CoordinatorBackendState::PublishFilter() run > independent of the lifecycle of any fragment instance. This is problematic > during query teardown. > Specifically we should not release resources for a query if any one of those > above functions are still running for that query and we also should not not > start running the above methods after resources are released for the query. > Also, the 'rpc_params' in UpdateFilter() could potentially hold large amounts > of untracked memory, so we should track it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Comment Edited] (IMPALA-7169) TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
[
https://issues.apache.org/jira/browse/IMPALA-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514539#comment-16514539
]
Tianyi Wang edited comment on IMPALA-7169 at 6/15/18 11:53 PM:
---
We set fs.trash.interval = 1440 so that HDFS supposedly shouldn't do the
checkpointing within a day.
The build took less than a day but HDFS did it when this test is running.
My current theory is that there are some leftover in the trash in our HDFS
snapshot. The timestamp (which is part of the filename) on those leftover is
older than a day so that HDFS thought more than a day has passed.
was (Author: tianyiwang):
OK. So we set fs.trash.interval = 1440 so that HDFS supposedly shouldn't do the
checkpointing within a day.
The build took less than a day but HDFS did it when this test is running.
My current theory is that there are some leftover in the trash in our HDFS
snapshot. The timestamp (which is part of the filename) on those leftover is
older than a day so that HDFS thought more than a day has passed.
> TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
> --
>
> Key: IMPALA-7169
> URL: https://issues.apache.org/jira/browse/IMPALA-7169
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.13.0
>Reporter: Lars Volker
>Assignee: Tianyi Wang
>Priority: Critical
> Labels: broken-build, flaky
>
> {noformat}
> F
> metadata/test_hdfs_encryption.py::TestHdfsEncryption::()::test_drop_partition_encrypt
> metadata/test_hdfs_encryption.py:202: in test_drop_partition_encrypt
> assert self.hdfs_client.exists(
> E assert 0xba6ee90>>('/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt')
> E+ where > =
> .exists
> E+where 0xba6ee90> = 0xba6ed50>.hdfs_client
> E+ and
> '/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt'
> = ('jenkins',
> 'test_encryption_db')
> E+where =
> '/user/{0}/.Trash/Current/test-warehouse/{1}.db/t1/j=1/j1.txt'.format
> E+and 'jenkins' = ()
> E+ where = getpass.getuser
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-7171) Add docs for Kudu insert partitioning/sorting
[ https://issues.apache.org/jira/browse/IMPALA-7171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-7171 started by Alex Rodoni. --- > Add docs for Kudu insert partitioning/sorting > - > > Key: IMPALA-7171 > URL: https://issues.apache.org/jira/browse/IMPALA-7171 > Project: IMPALA > Issue Type: Improvement > Components: Docs >Reporter: Thomas Tauber-Marshall >Assignee: Alex Rodoni >Priority: Major > Labels: docs > > On the page: > http://impala.apache.org/docs/build3x/html/topics/impala_kudu.html, at the > end of the section: "Impala DML Support for Kudu Tables (INSERT, UPDATE, > DELETE, UPSERT)", we should add text like: > Starting from Impala 2.9, Impala will automatically add a partition and sort > step to INSERTs before sending the rows to Kudu. Since Kudu partitions and > sorts rows on write, pre-partitioning and sorting takes some of the load off > of Kudu, and helps ensure that large INSERTs complete without timing out, but > it may slow down the end-to-end performance of the INSERT. Starting from > Impala 2.10, the hints "/* +noshuffle,noclustered */" may be used to turn > this pre-partitioning and sorting off. Additionally, since sorting may > consume a lot of memory, users should consider setting a "mem_limit" for > these queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Comment Edited] (IMPALA-7169) TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
[
https://issues.apache.org/jira/browse/IMPALA-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514539#comment-16514539
]
Tianyi Wang edited comment on IMPALA-7169 at 6/15/18 11:52 PM:
---
OK. So we set fs.trash.interval = 1440 so that HDFS supposedly shouldn't do the
checkpointing within a day.
The build took less than a day but HDFS did it when this test is running.
My current theory is that there are some leftover in the trash in our HDFS
snapshot. The timestamp (which is part of the filename) on those leftover is
older than a day so that HDFS thought more than a day has passed.
was (Author: tianyiwang):
OK. So cloudera's Hadoopa fs.trash.interval = 1440 so that HDFS supposedly
shouldn't do the checkpointing within a day.
The build took less than a day but HDFS did it when this test is running.
My current theory is that there are some leftover in the trash in our HDFS
snapshot. The timestamp (which is part of the filename) on those leftover is
older than a day so that HDFS thought more than a day has passed.
> TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
> --
>
> Key: IMPALA-7169
> URL: https://issues.apache.org/jira/browse/IMPALA-7169
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.13.0
>Reporter: Lars Volker
>Assignee: Tianyi Wang
>Priority: Critical
> Labels: broken-build, flaky
>
> {noformat}
> F
> metadata/test_hdfs_encryption.py::TestHdfsEncryption::()::test_drop_partition_encrypt
> metadata/test_hdfs_encryption.py:202: in test_drop_partition_encrypt
> assert self.hdfs_client.exists(
> E assert 0xba6ee90>>('/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt')
> E+ where > =
> .exists
> E+where 0xba6ee90> = 0xba6ed50>.hdfs_client
> E+ and
> '/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt'
> = ('jenkins',
> 'test_encryption_db')
> E+where =
> '/user/{0}/.Trash/Current/test-warehouse/{1}.db/t1/j=1/j1.txt'.format
> E+and 'jenkins' = ()
> E+ where = getpass.getuser
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7169) TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
[
https://issues.apache.org/jira/browse/IMPALA-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514539#comment-16514539
]
Tianyi Wang commented on IMPALA-7169:
-
OK. So cloudera's Hadoopa fs.trash.interval = 1440 so that HDFS supposedly
shouldn't do the checkpointing within a day.
The build took less than a day but HDFS did it when this test is running.
My current theory is that there are some leftover in the trash in our HDFS
snapshot. The timestamp (which is part of the filename) on those leftover is
older than a day so that HDFS thought more than a day has passed.
> TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
> --
>
> Key: IMPALA-7169
> URL: https://issues.apache.org/jira/browse/IMPALA-7169
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.13.0
>Reporter: Lars Volker
>Assignee: Tianyi Wang
>Priority: Critical
> Labels: broken-build, flaky
>
> {noformat}
> F
> metadata/test_hdfs_encryption.py::TestHdfsEncryption::()::test_drop_partition_encrypt
> metadata/test_hdfs_encryption.py:202: in test_drop_partition_encrypt
> assert self.hdfs_client.exists(
> E assert 0xba6ee90>>('/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt')
> E+ where > =
> .exists
> E+where 0xba6ee90> = 0xba6ed50>.hdfs_client
> E+ and
> '/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt'
> = ('jenkins',
> 'test_encryption_db')
> E+where =
> '/user/{0}/.Trash/Current/test-warehouse/{1}.db/t1/j=1/j1.txt'.format
> E+and 'jenkins' = ()
> E+ where = getpass.getuser
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-6249) Expose CMAKE_BUILD_TYPE via web UI for build type detection
[ https://issues.apache.org/jira/browse/IMPALA-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong updated IMPALA-6249: -- Priority: Minor (was: Major) > Expose CMAKE_BUILD_TYPE via web UI for build type detection > --- > > Key: IMPALA-6249 > URL: https://issues.apache.org/jira/browse/IMPALA-6249 > Project: IMPALA > Issue Type: Improvement > Components: Infrastructure >Reporter: Tim Armstrong >Priority: Minor > > IMPALA-6241 added a .cmake_build_type file with the CMAKE_BUILD_TYPE value > for the last build. The file is used to detect the type of the build that the > python tests are running against. However, this assumes that the tests are > running from the same directory that the Impala cluster under test was built > from, which isn't necessarily true for all dev workflows and for remote > cluster tests. > It would be convenient if CMAKE_BUILD_TYPE was exposed from the Impalad web > UI. Currently we expose DEBUG/RELEASE depending on the value of NDEBUG - see > GetVersionString() and impalad-host:25000/?json=true, but we could expose the > precise build type, then allow the python tests to parse it from the web UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7169) TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
[
https://issues.apache.org/jira/browse/IMPALA-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514480#comment-16514480
]
Tianyi Wang commented on IMPALA-7169:
-
The log in the failed build:
{noformat}
2018-06-11 16:59:54,285 INFO org.apache.hadoop.hdfs.StateChange: DIR*
completeFile: /test-warehouse/test_enc
ryption_db.db/t1/j=3/j3.txt is closed by DFSClient_NONMAPREDUCE_-192125689_43
2018-06-11 17:00:00,067 INFO org.apache.hadoop.fs.TrashPolicyDefault:
TrashPolicyDefault#deleteCheckpoint fo
r trashRoot: hdfs://localhost:20500/user/jenkins/.Trash
2018-06-11 17:00:00,072 INFO org.apache.hadoop.fs.TrashPolicyDefault: Created
trash checkpoint: /user/jenkin
s/.Trash/18061117
2018-06-11 17:00:00,894 WARN org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:
jenkins (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not exist:
/user/jenkins/.Trash/Current
/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt
{noformat}
The log of a success build
{noformat}
2018-06-13 22:20:24,736 INFO org.apache.hadoop.hdfs.StateChange: DIR*
completeFile: /test-warehouse/test_encryption_db.db/
t1/j=3/j3.txt is closed by DFSClient_NONMAPREDUCE_-2038519081_43
2018-06-13 22:20:31,100 WARN org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:jenkins (auth:
SIMPLE) cause:java.io.FileNotFoundException: File does not exist:
/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt
2018-06-13 22:20:31,108 WARN org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:jenkins (auth:
SIMPLE) cause:java.io.FileNotFoundException: File does not exist:
/test-warehouse/test_encryption_db.db/t1/j=1
2018-06-13 22:20:31,168 INFO BlockStateChange: BLOCK* addToInvalidates:
blk_1073755231_14407 127.0.0.1:31000 127.0.0.1:310
01 127.0.0.1:31002
2018-06-13 22:20:33,664 INFO BlockStateChange: BLOCK* BlockManager: ask
127.0.0.1:31001 to delete [blk_1073755231_14407]
2018-06-13 22:20:36,665 INFO BlockStateChange: BLOCK* BlockManager: ask
127.0.0.1:31002 to delete [blk_1073755231_14407]
2018-06-13 22:20:37,098 WARN org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:jenkins (auth:
SIMPLE) cause:java.io.FileNotFoundException: File does not exist:
/test-warehouse/test_encryption_db.db/t1/j=2/j2.txt
2018-06-13 22:20:37,107 WARN org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:jenkins (auth:
SIMPLE) cause:java.io.FileNotFoundException: File does not exist:
/test-warehouse/test_encryption_db.db/t1/j=2
2018-06-13 22:20:37,167 INFO BlockStateChange: BLOCK* addToInvalidates:
blk_1073755232_14408 127.0.0.1:31001 127.0.0.1:310
02 127.0.0.1:31000
2018-06-13 22:20:39,665 INFO BlockStateChange: BLOCK* BlockManager: ask
127.0.0.1:31002 to delete [blk_1073755232_14408]
2018-06-13 22:20:42,666 INFO BlockStateChange: BLOCK* BlockManager: ask
127.0.0.1:31001 to delete [blk_1073755232_14408]
2018-06-13 22:20:43,107 WARN org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:jenkins (auth:
SIMPLE) cause:java.io.FileNotFoundException: File does not exist:
/test-warehouse/test_encryption_db.db/t1/j=3/j3.txt
2018-06-13 22:20:43,117 WARN org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:jenkins (auth:
SIMPLE) cause:java.io.FileNotFoundException: File does not exist:
/test-warehouse/test_encryption_db.db/t1/j=3
2018-06-13 22:20:45,666 INFO BlockStateChange: BLOCK* BlockManager: ask
127.0.0.1:31000 to delete [blk_1073755232_14408, b
lk_1073755231_14407]
2018-06-13 22:20:59,342 INFO BlockStateChange: BLOCK* addToInvalidates:
blk_1073755230_14406 127.0.0.1:31001 127.0.0.1:310
02 127.0.0.1:31000
{noformat}
Maybe the "TrashPolicyDefault#deleteCheckpoint" operation raced with something.
> TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
> --
>
> Key: IMPALA-7169
> URL: https://issues.apache.org/jira/browse/IMPALA-7169
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.13.0
>Reporter: Lars Volker
>Assignee: Tianyi Wang
>Priority: Critical
> Labels: broken-build, flaky
>
> {noformat}
> F
> metadata/test_hdfs_encryption.py::TestHdfsEncryption::()::test_drop_partition_encrypt
> metadata/test_hdfs_encryption.py:202: in test_drop_partition_encrypt
> assert self.hdfs_client.exists(
> E assert 0xba6ee90>>('/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt')
> E+ where > =
> .exists
> E+where 0xba6ee90> = 0xba6ed50>.hdfs_client
> E+ and
> '/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt'
> = ('jenkins',
> 'test_encryption_db')
> E+where =
>
[jira] [Resolved] (IMPALA-7125) test_cancellation execute_cancel_test() should have a delay before fetch
[ https://issues.apache.org/jira/browse/IMPALA-7125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dan Hecht resolved IMPALA-7125. --- Resolution: Not A Problem Actually, I think we get this coverage already from failure/test_failpoints.py > test_cancellation execute_cancel_test() should have a delay before fetch > > > Key: IMPALA-7125 > URL: https://issues.apache.org/jira/browse/IMPALA-7125 > Project: IMPALA > Issue Type: Improvement > Components: Infrastructure >Affects Versions: Impala 3.0 >Reporter: Dan Hecht >Assignee: Dan Hecht >Priority: Major > > We should exercise cancel rpc when fetch has not been called by either adding > a delay before fetch rpc, or not executing fetch rpc sometimes. The > cancellation side effects are (unfortunately) a bit different depending on > whether the fetch path picks up the cancellation, so this is should be > exercised explicitly. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-6942) "Cancelled due to unreachable impalad(s)" error message is misleading
[ https://issues.apache.org/jira/browse/IMPALA-6942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dan Hecht resolved IMPALA-6942. --- Resolution: Fixed Fix Version/s: Impala 3.1.0 commit 76f8ff4a1507c3a7f86eb58dbac191024b59890e Author: Dan Hecht Date: Thu Jun 14 13:05:49 2018 -0700 IMPALA-6942: Reword error message to say "Failed" rather than "Cancelled" In this case, the query is failing. It happens to use the cancellation path to cleanup, but from a user's perspective this is a query failure not a cancellation. Reword the message to reflect that. Change-Id: I4d8e755aef196e5c25205094af9c8486eb899344 Reviewed-on: http://gerrit.cloudera.org:8080/10717 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > "Cancelled due to unreachable impalad(s)" error message is misleading > - > > Key: IMPALA-6942 > URL: https://issues.apache.org/jira/browse/IMPALA-6942 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 3.0 >Reporter: Dan Hecht >Assignee: Dan Hecht >Priority: Major > Fix For: Impala 3.1.0 > > > The error message "Cancelled due to unreachable impalad(s)" would be better > as "Failed due to unreachable impalad(s)" since the query has failed. The > code happens to trigger cancellation via the same path as client > cancellation, but this is really a query failure. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7175) In a local FS build, test_native_functions_race thinks there are 2 impalads where there should be 1
[
https://issues.apache.org/jira/browse/IMPALA-7175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514316#comment-16514316
]
Vuk Ercegovac commented on IMPALA-7175:
---
I see the following that's due to the ImpalaCluster() call on L325:
MainThread: Found 2 impalad/1 statestored/1 catalogd process(es)
The test is run multiple times and all of the other runs it look like this:
MainThread: Found 1 impalad/1 statestored/1 catalogd process(es)
I see a lot of these types of messages that are likely thrown at L132 when a
listed process is not found:
MainThread: no process found with pid 20961
...
For ee-tests, I assume the cluster nodes are *not* expected to be killed or
die.
> In a local FS build, test_native_functions_race thinks there are 2 impalads
> where there should be 1
> ---
>
> Key: IMPALA-7175
> URL: https://issues.apache.org/jira/browse/IMPALA-7175
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.1.0
>Reporter: Tianyi Wang
>Assignee: Vuk Ercegovac
>Priority: Critical
> Labels: broken-build
>
> In TestUdfExecution.test_native_functions_race, the test checks the number of
> impalads at the beginning and end of the test. In a local build there should
> be only 1 impalad but somehow the test found 2 at the beginning of the test
> and failed.
> {noformat}
> Stacktrace
> query_test/test_udfs.py:379: in test_native_functions_race
> assert len(cluster.impalads) == exp_num_impalads
> E assert 1 == 2
> E+ where 1 = len([ 0xc9ffa90>])
> E+where [ 0xc9ffa90>] = 0x6a5d510>.impalads
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7154) Error making 'dropDatabase' RPC to Hive Metastore
[
https://issues.apache.org/jira/browse/IMPALA-7154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514246#comment-16514246
]
Tianyi Wang commented on IMPALA-7154:
-
Saw it again. I took a brief look. The problem was that the connection to HMS
was lost after the drop database commend was sent. HMS client however didn't
know the command took effect and retried so it failed.
{noformat}
I0614 21:46:06.395942 14644 ClientCnxn.java:975] Opening socket connection to
server localhost/0:0:0:
0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
W0614 21:46:06.396255 14644 ClientCnxn.java:1102] Session 0x0 for server null,
unexpected error, clos
ing socket connection and attempting reconnect
Java exception follows:
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
W0614 21:46:06.548285 12859 RetryingMetaStoreClient.java:148] MetaStoreClient
lost connection. Attemp
ting to reconnect.
Java exception follows:
org.apache.thrift.transport.TTransportException:
java.net.SocketTimeoutException: Read timed out
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_drop_database(ThriftHiveMet
astore.java:733)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.drop_database(ThriftHiveMetastor
e.java:718)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropDatabase(HiveMetaStoreClient.java:810)
at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:101
)
at com.sun.proxy.$Proxy5.dropDatabase(Unknown Source)
at
org.apache.impala.service.CatalogOpExecutor.dropDatabase(CatalogOpExecutor.java:1305)
at
org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:300)
at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:146)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
... 16 more
I0614 21:46:06.550910 12859 HiveMetaStoreClient.java:547] Closed a connection
to metastore, current c
onnections: 9
I0614 21:46:06.550966 12859 HiveMetaStoreClient.java:391] Trying to connect to
metastore with URI thr
ift://localhost:9083
I0614 21:46:06.551259 12859 HiveMetaStoreClient.java:465] Opened a connection
to metastore, current c
onnections: 10
I0614 21:46:06.551961 12859 HiveMetaStoreClient.java:518] Connected to
metastore.
I0614 21:46:06.555757 12859 jni-util.cc:230]
org.apache.impala.common.ImpalaRuntimeException: Error m
aking 'dropDatabase' RPC to Hive Metastore:
at
org.apache.impala.service.CatalogOpExecutor.dropDatabase(CatalogOpExecutor.java:1309)
at
org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:300)
at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:146)
Caused by: NoSuchObjectException(message:test_fuzz_nested_types_67367717)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_database_result$get_database_result
StandardScheme.read(ThriftHiveMetastore.java:16387)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_database_result$get_database_result
StandardScheme.read(ThriftHiveMetastore.java:16364)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_database_result.read(ThriftHiveMeta
store.java:16295)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMeta
store.java:702)
at
[jira] [Commented] (IMPALA-6086) Use of permanent function should require SELECT privilege on DB
[ https://issues.apache.org/jira/browse/IMPALA-6086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514241#comment-16514241 ] Fredy Wijaya commented on IMPALA-6086: -- Yup, that seems like the right approach. > Use of permanent function should require SELECT privilege on DB > --- > > Key: IMPALA-6086 > URL: https://issues.apache.org/jira/browse/IMPALA-6086 > Project: IMPALA > Issue Type: Bug > Components: Catalog, Security >Affects Versions: Impala 2.9.0, Impala 3.1.0 >Reporter: Zoram Thanga >Assignee: Zoram Thanga >Priority: Minor > > A user that has no privilege on a database should not be able to execute any > permanent functions in that database. This is currently possible, and should > be fixed, so that the user must have SELECT privilege to execute permanent > functions. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-3040) test_caching_ddl failed with unexpected get_num_cache_requests
[
https://issues.apache.org/jira/browse/IMPALA-3040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyi Wang updated IMPALA-3040:
Labels: broken-build flaky (was: flaky)
> test_caching_ddl failed with unexpected get_num_cache_requests
> --
>
> Key: IMPALA-3040
> URL: https://issues.apache.org/jira/browse/IMPALA-3040
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 2.5.0
>Reporter: Tim Armstrong
>Assignee: Tianyi Wang
>Priority: Major
> Labels: broken-build, flaky
>
> This test failed on the integration job.
> http://sandbox.jenkins.cloudera.com/view/Impala/view/Builds%20-%202.5.0%20release/job/impala-cdh5.7.0-integration/3/testReport/junit/query_test.test_hdfs_caching/TestHdfsCachingDdl/test_caching_ddl_exec_optiondisable_codegen___False___abort_on_error___1___exec_single_node_rows_threshold___0___batch_size___0___num_nodes___0table_format__text_none_/
> {code}
> query_test.test_hdfs_caching.TestHdfsCachingDdl.test_caching_ddl[exec_option:
> {'disable_codegen': False, 'abort_on_error': 1,
> 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} |
> table_format: text/none] (from pytest)
> Failing for the past 1 build (Since Failed#3 )
> Took 47 sec.
> add description
> Error Message
> assert 9 == 10 + where 10 = get_num_cache_requests()
> Stacktrace
> query_test/test_hdfs_caching.py:132: in test_caching_ddl
> assert num_entries_pre == get_num_cache_requests()
> E assert 9 == 10
> E+ where 10 = get_num_cache_requests()
> Standard Error
> -- connecting to: localhost:21000
> -- executing against localhost:21000
> use default;
> SET sync_ddl=1;
> -- executing against localhost:21000
> drop database if exists `cachedb` cascade;
> -- executing against localhost:21000
> create database cachedb;
> -- executing against localhost:21000
> use functional;
> SET disable_codegen=False;
> SET abort_on_error=1;
> SET exec_single_node_rows_threshold=0;
> SET batch_size=0;
> SET num_nodes=0;
> -- executing against localhost:21000
> use cachedb;
> -- executing against localhost:21000
> create table cached_tbl_nopart (i int) cached in 'testPool';
> -- executing against localhost:21000
> insert into cached_tbl_nopart select 1;
> -- executing against localhost:21000
> select * from cached_tbl_nopart;
> -- executing against localhost:21000
> create table like_cached_tbl_nopart like cached_tbl_nopart;
> -- executing against localhost:21000
> show table stats like_cached_tbl_nopart;
> -- executing against localhost:21000
> show table stats cached_tbl_nopart;
> -- executing against localhost:21000
> alter table cached_tbl_nopart set uncached;
> -- executing against localhost:21000
> show table stats cached_tbl_nopart;
> -- executing against localhost:21000
> drop table if exists cached_tbl_part;
> -- executing against localhost:21000
> create table cached_tbl_part (i int) partitioned by (j int) cached in
> 'testPool' with replication = 9;
> -- executing against localhost:21000
> alter table cached_tbl_part add partition (j=0);
> -- executing against localhost:21000
> alter table cached_tbl_part add partition (j=1) uncached;
> -- executing against localhost:21000
> alter table cached_tbl_part add partition (j=2) cached in 'testPool';
> -- executing against localhost:21000
> show partitions cached_tbl_part;
> -- executing against localhost:21000
> drop table if exists cached_tbl_part;
> -- executing against localhost:21000
> create table cached_tbl_part (i int) partitioned by (j int) cached in
> 'testPool';
> -- executing against localhost:21000
> alter table cached_tbl_part add partition (j=0);
> -- executing against localhost:21000
> alter table cached_tbl_part add partition (j=1) uncached;
> -- executing against localhost:21000
> alter table cached_tbl_part add partition (j=2) cached in 'testPool';
> -- executing against localhost:21000
> show partitions cached_tbl_part;
> -- executing against localhost:21000
> alter table cached_tbl_part partition (j=2) set uncached;
> -- executing against localhost:21000
> show partitions cached_tbl_part;
> -- executing against localhost:21000
> alter table cached_tbl_part partition (j=2) set uncached;
> -- executing against localhost:21000
> show partitions cached_tbl_part;
> -- executing against localhost:21000
> alter table cached_tbl_part partition (j=1) set cached in 'testPool';
> -- executing against localhost:21000
> show partitions cached_tbl_part;
> -- executing against localhost:21000
> insert into cached_tbl_part partition(j) values(3, 3), (4, 4);
> -- executing against localhost:21000
> alter table cached_tbl_part partition (j=3) set cached in 'testPool' with
> replication = 4;
> -- executing against localhost:21000
> show partitions cached_tbl_part;
> --
[jira] [Updated] (IMPALA-7115) Set a default THREAD_RESERVATION_LIMIT value
[
https://issues.apache.org/jira/browse/IMPALA-7115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-7115:
--
Target Version: Impala 3.1.0 (was: Impala 2.13.0, Impala 3.1.0)
> Set a default THREAD_RESERVATION_LIMIT value
>
>
> Key: IMPALA-7115
> URL: https://issues.apache.org/jira/browse/IMPALA-7115
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Major
> Labels: resource-management
>
> As a follow on to IMPALA-6035, we should set a default value that actually
> will help protect again insanely complex queries.
> Motivating discussion is here:
> https://gerrit.cloudera.org/#/c/10365/9/common/thrift/ImpalaInternalService.thrift
> {quote}
> Tim Armstrong
> 1:11 PM
> Dan suggested setting a default here. I started doing some experiments to see
> what our current practical limits are.
> On stock Ubuntu 16.04 I start getting thread_resource_error at around 8000
> reserved threads. I'm not sure that the config reflects what people would use
> on production systems so continuing to investigate.
> Dan Hecht
> 1:31 PM
> We could also consider choosing a default dynamically based on the OS's
> setting, if that's necessary.
> Tim Armstrong
> 3:45 PM
> I increased some of the configs (I think I was limited by
> /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max == 12288) and now it
> got oom-killed at ~26000 threads.
> I think unfortunately there are a lot of different OS knobs that impact this
> and they seem to evolve over time, so it's probably not feasible with a
> reasonable amount of effort to get it working on all common Linux distros.
> I was thinking ~5000, since 1000-2000 plan nodes is the most I've seen for a
> query running successfully in production.
> Maybe I should do this in a follow-on change, since we probably also want to
> add a test query at or near this limit.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-6656) Metrics for time spent in BufferAllocator
[ https://issues.apache.org/jira/browse/IMPALA-6656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong updated IMPALA-6656: -- Target Version: Impala 3.1.0 (was: Impala 2.13.0) > Metrics for time spent in BufferAllocator > - > > Key: IMPALA-6656 > URL: https://issues.apache.org/jira/browse/IMPALA-6656 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > Labels: observability, resource-management > > We should track the total time spent and the time spent in TCMalloc so we can > understand where time is going globally. > I think we should shard these metrics across the arenas so we can see if the > problem is just per-arena, and also to avoid contention between threads when > updating the metrics. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6086) Use of permanent function should require SELECT privilege on DB
[
https://issues.apache.org/jira/browse/IMPALA-6086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514221#comment-16514221
]
Zoram Thanga commented on IMPALA-6086:
--
In case it was not clear, here's a sample session. User 'dummy_user' has no
privileges:
{noformat}
[localhost:21000] default> show roles;
Query: show roles
ERROR: AuthorizationException: User 'dummy_user' does not have privileges to
access the requested policy metadata or Sentry Service is unavailable.
[localhost:21000] default> show tables;
Query: show tables
ERROR: AuthorizationException: User 'dummy_user' does not have privileges to
access: default.*.*
[localhost:21000] default> select has_vowels('abcdef');
Query: select has_vowels('abcdef')
Query submitted at: 2018-06-15 11:22:39 (Coordinator:
http://zoram-desktop:25000)
Query progress can be monitored at:
http://zoram-desktop:25000/query_plan?query_id=8347149d8e057744:12f35393
+--+
| default.has_vowels('abcdef') |
+--+
| true |
+--+
Fetched 1 row(s) in 0.12s
{noformat}
With my patch above, the result is different:
{noformat}
[localhost:21000] default> show roles;
Query: show roles
ERROR: AuthorizationException: User 'dummy_user' does not have privileges to
access the requested policy metadata or Sentry Service is unavailable.
[localhost:21000] default> show tables;
Query: show tables
ERROR: AuthorizationException: User 'dummy_user' does not have privileges to
access: default.*.*
[localhost:21000] default> select has_vowels('abcdef');
Query: select has_vowels('abcdef')
Query submitted at: 2018-06-15 11:30:40 (Coordinator:
http://zoram-desktop:25000)
ERROR: AuthorizationException: User 'dummy_user' does not have privileges to
access: default
[localhost:21000] default>
{noformat}
What do you think, [~fredyw]?
> Use of permanent function should require SELECT privilege on DB
> ---
>
> Key: IMPALA-6086
> URL: https://issues.apache.org/jira/browse/IMPALA-6086
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog, Security
>Affects Versions: Impala 2.9.0, Impala 3.1.0
>Reporter: Zoram Thanga
>Assignee: Zoram Thanga
>Priority: Minor
>
> A user that has no privilege on a database should not be able to execute any
> permanent functions in that database. This is currently possible, and should
> be fixed, so that the user must have SELECT privilege to execute permanent
> functions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6994) Avoid reloading a table's HMS data for file-only operations
[ https://issues.apache.org/jira/browse/IMPALA-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514214#comment-16514214 ] Pranay Singh commented on IMPALA-6994: -- [~jeszyb] I have made the below change, that is currently in review, here is a brief description of the change made. The problem is that while inserting a new row to an unpartitioned HDFS table, or to an existing partition, the catalogd makes unecessary call to Hive Meta Store to do a getTable() and list the partitions of the table, when in fact only the file metadata for HDFS tables needs to be updated. This extra call to HMS introduces a point of failure, we need to handle for error scenario when Hive MetaStore crashes. This change removes the extra call to Hive Meta Store for the case when a row is inserted to an existing partition in HDFS table or when a row is added to unpartitioned table. Thus, an optimization as it reduces the call to Hive Meta Store during Update of Catalog > Avoid reloading a table's HMS data for file-only operations > --- > > Key: IMPALA-6994 > URL: https://issues.apache.org/jira/browse/IMPALA-6994 > Project: IMPALA > Issue Type: Improvement > Components: Catalog >Affects Versions: Impala 2.12.0 >Reporter: Balazs Jeszenszky >Assignee: Pranay Singh >Priority: Major > > Reloading file metadata for HDFS tables (e.g. as a final step in an 'insert') > is done via > https://github.com/apache/impala/blob/branch-2.12.0/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L628 > , which calls > https://github.com/apache/impala/blob/branch-2.12.0/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java#L1243 > HdfsTable.load has no option to only load file metadata. HMS metadata will > also be reloaded every time, which is an unnecessary overhead (and potential > point of failure) when adding files to existing locations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7179) Consider changing --allow_multiple_scratch_dirs_per_device default
[ https://issues.apache.org/jira/browse/IMPALA-7179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514185#comment-16514185 ] Balazs Jeszenszky commented on IMPALA-7179: --- I agree this should be default, this behaviour is what I would intuitively expect. > Consider changing --allow_multiple_scratch_dirs_per_device default > -- > > Key: IMPALA-7179 > URL: https://issues.apache.org/jira/browse/IMPALA-7179 > Project: IMPALA > Issue Type: Sub-task > Components: Backend >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Critical > Labels: resource-management > > I've seen multiple instances of Impala users being tripped up by this > behaviour and zero instances of it being useful (although it's possible that > it helped someone and they didn't notice). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-5168) Codegen hash computation in DataStreamSender::Send for partition exchange.
[
https://issues.apache.org/jira/browse/IMPALA-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Ho resolved IMPALA-5168.
Resolution: Fixed
[https://github.com/apache/impala/commit/51ff47d05e6edf80ca07805c68b417dec789d85b]
{noformat}
IMPALA-5168: Codegen HASH_PARTITIONED KrpcDataStreamSender::Send()
This change codegens the hash partitioning logic of
KrpcDataStreamSender::Send() when the partitioning strategy
is HASH_PARTITIONED. It does so by unrolling the loop which
evaluates each row against the partitioning expressions and
hashes the result. It also replaces the number of channels
of that sender with a constant at runtime.
With this change, we get reasonable speedup with some benchmarks:
++---+-++++
| Workload | File Format | Avg (s) | Delta(Avg) | GeoMean(s) |
Delta(GeoMean) |
++---+-++++
| TPCH(_300) | parquet / none / none | 20.03 | -6.44% | 13.56 |
-7.15% |
++---+-++++
+-+---+-++++
| Workload| File Format | Avg (s) | Delta(Avg) |
GeoMean(s) | Delta(GeoMean) |
+-+---+-++++
| TARGETED-PERF(_300) | parquet / none / none | 58.59 | -5.56% | 12.28
| -5.30% |
+-+---+-++++
+-+---+-++++
| Workload| File Format | Avg (s) | Delta(Avg) |
GeoMean(s) | Delta(GeoMean) |
+-+---+-++++
| TPCDS-UNMODIFIED(_1000) | parquet / none / none | 15.60 | -3.10% | 7.16
| -4.33% |
+-+---+-++++
+---+---+-++++
| Workload | File Format | Avg (s) | Delta(Avg) | GeoMean(s)
| Delta(GeoMean) |
+---+---+-++++
| TPCH_NESTED(_300) | parquet / none / none | 30.93 | -3.02% | 17.46
| -4.71% |
+---+---+-++++
Change-Id: I1c44cc9312c062cc7a5a4ac9156ceaa31fb887ff
Reviewed-on: http://gerrit.cloudera.org:8080/10421
Reviewed-by: Michael Ho
Tested-by: Impala Public Jenkins
{noformat}
> Codegen hash computation in DataStreamSender::Send for partition exchange.
> ---
>
> Key: IMPALA-5168
> URL: https://issues.apache.org/jira/browse/IMPALA-5168
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Affects Versions: Impala 2.6.0, Impala 2.7.0, Impala 2.8.0, Impala 2.9.0,
> Impala 2.10.0, Impala 2.11.0
>Reporter: Mostafa Mokhtar
>Assignee: Michael Ho
>Priority: Major
> Labels: perfomance
>
> Hash partition computation for exchange operators can benefit from codegen,
> profile data ~20% of CPU in the fragment thread is consumed by
> RawValue::GetHashValueFnv & ExprContext::GetValue
> {code}
> // hash-partition batch's rows across channels
> int num_channels = channels_.size();
> for (int i = 0; i < batch->num_rows(); ++i) {
> TupleRow* row = batch->GetRow(i);
> uint32_t hash_val = HashUtil::FNV_SEED;
> for (int i = 0; i < partition_expr_ctxs_.size(); ++i) {
> ExprContext* ctx = partition_expr_ctxs_[i];
> void* partition_val = ctx->GetValue(row);
> // We can't use the crc hash function here because it does not result
> // in uncorrelated hashes with different seeds. Instead we must use
> // fnv hash.
> // TODO: fix crc hash/GetHashValue()
> hash_val =
> RawValue::GetHashValueFnv(partition_val, ctx->root()->type(),
> hash_val);
> }
> ExprContext::FreeLocalAllocations(partition_expr_ctxs_);
> RETURN_IF_ERROR(channels_[hash_val % num_channels]->AddRow(row));
> }
> {code}
> |Function Stack| Effective Time % |
> |Total|100%|
> | clone|99%|
> | start_thread|99%|
> | thread_proxy|99%|
> |boost::detail::thread_data::run|99%|
> | boost::_bi::bind_t | operator() | impala::Thread::SuperviseThread|99%|
> |boost::function0::operator()|99%|
> |
[jira] [Created] (IMPALA-7179) Consider changing --allow_multiple_scratch_dirs_per_device default
Tim Armstrong created IMPALA-7179: - Summary: Consider changing --allow_multiple_scratch_dirs_per_device default Key: IMPALA-7179 URL: https://issues.apache.org/jira/browse/IMPALA-7179 Project: IMPALA Issue Type: Sub-task Components: Backend Reporter: Tim Armstrong Assignee: Tim Armstrong I've seen multiple instances of Impala users being tripped up by this behaviour and zero instances of it being useful (although it's possible that it helped someone and they didn't notice). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-2937) Introduce sampled statistics
[ https://issues.apache.org/jira/browse/IMPALA-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mostafa Mokhtar resolved IMPALA-2937. - Resolution: Fixed Fix Version/s: Impala 2.12.0 > Introduce sampled statistics > > > Key: IMPALA-2937 > URL: https://issues.apache.org/jira/browse/IMPALA-2937 > Project: IMPALA > Issue Type: Sub-task > Components: Backend >Affects Versions: Impala 2.2 >Reporter: Mostafa Mokhtar >Priority: Minor > Labels: planner, supportability > Fix For: Impala 2.12.0 > > > Stats computation can be expensive, introduce sampled scans for Parquet and > text files to speedup stats computation. > Database will have a default sampling rate which can be overridden at table > level. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7138) Fix detection and handling of Device Mapper volumes
[
https://issues.apache.org/jira/browse/IMPALA-7138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-7138:
--
Summary: Fix detection and handling of Device Mapper volumes (was: Impala
gets confused by multiple scratch directories on Device Mapper volumes)
> Fix detection and handling of Device Mapper volumes
> ---
>
> Key: IMPALA-7138
> URL: https://issues.apache.org/jira/browse/IMPALA-7138
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Major
> Labels: resource-management
>
> I've heard a couple of instances of Impala doing the wrong thing when
> multiple scratch directories are present on a single DeviceMapper logical
> volume. E.g. in the following setup, we only use one directory as a disk.
> {noformat}
> >> mount list
> /dev/mapper/xx01-lv_data01 on /data01 type ext4 (rw,relatime,data=ordered)
> /dev/mapper/xx02-lv_data02 on /data02 type ext4 (rw,relatime,data=ordered)
> /dev/mapper/xx03-lv_data03 on /data03 type ext4 (rw,relatime,data=ordered)
> /dev/mapper/xx04-lv_data04 on /data04 type ext4 (rw,relatime,data=ordered)
> /dev/mapper/xx05-lv_data05 on /data05 type ext4 (rw,relatime,data=ordered)
> /dev/mapper/xx06-lv_data06 on /data06 type ext4 (rw,relatime,data=ordered)
> /dev/mapper/xx07-lv_data07 on /data07 type ext4 (rw,relatime,data=ordered)
> /dev/mapper/xx08-lv_data08 on /data08 type ext4 (rw,relatime,data=ordered)
> /dev/mapper/xx09-lv_data09 on /data09 type ext4 (rw,relatime,data=ordered)
> /dev/mapper/xx10-lv_data10 on /data10 type ext4 (rw,relatime,data=ordered)
> /dev/mapper/xx11-lv_data11 on /data11 type ext4 (rw,relatime,data=ordered)
> /dev/mapper/xx12-lv_data12 on /data12 type ext4 (rw,relatime,data=ordered)
> Scratch dirs are:
> /data01/impala/impalad,/data02/impala/impalad,/data03/impala/impalad,/data04/impala/impalad,/data05/impala/impalad,/data06/impala/impalad,/data07/impala/impalad,/data08/impala/impalad,/data09/impala/impalad,/data10/impala/impalad,/data11/impala/impalad,/data12/impala/impalad
> ~~~ tmp-file-mgr.cc:122] Using scratch directory
> /data01/impala/impalad/impala-scratch on disk13.
> {noformat}
> A workaround is to set --allow_multiple_scratch_dirs_per_device=true
> There are a few questions here:
> # Does the deduplication logic even make sense to have?
> # Do we need to fix how we treat Device mapper volumes?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7138) Impala gets confused by multiple scratch directories on Device Mapper volumes
[
https://issues.apache.org/jira/browse/IMPALA-7138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514058#comment-16514058
]
Tim Armstrong commented on IMPALA-7138:
---
This is probably also a problem for the Disk I/O manager since it will have
only one I/O queue per device and the sizing of that will be messed up.
> Impala gets confused by multiple scratch directories on Device Mapper volumes
> -
>
> Key: IMPALA-7138
> URL: https://issues.apache.org/jira/browse/IMPALA-7138
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Major
> Labels: resource-management
>
> I've heard a couple of instances of Impala doing the wrong thing when
> multiple scratch directories are present on a single DeviceMapper logical
> volume. E.g. in the following setup, we only use one directory as a disk.
> {noformat}
> >> mount list
> /dev/mapper/xx01-lv_data01 on /data01 type ext4 (rw,relatime,data=ordered)
> /dev/mapper/xx02-lv_data02 on /data02 type ext4 (rw,relatime,data=ordered)
> /dev/mapper/xx03-lv_data03 on /data03 type ext4 (rw,relatime,data=ordered)
> /dev/mapper/xx04-lv_data04 on /data04 type ext4 (rw,relatime,data=ordered)
> /dev/mapper/xx05-lv_data05 on /data05 type ext4 (rw,relatime,data=ordered)
> /dev/mapper/xx06-lv_data06 on /data06 type ext4 (rw,relatime,data=ordered)
> /dev/mapper/xx07-lv_data07 on /data07 type ext4 (rw,relatime,data=ordered)
> /dev/mapper/xx08-lv_data08 on /data08 type ext4 (rw,relatime,data=ordered)
> /dev/mapper/xx09-lv_data09 on /data09 type ext4 (rw,relatime,data=ordered)
> /dev/mapper/xx10-lv_data10 on /data10 type ext4 (rw,relatime,data=ordered)
> /dev/mapper/xx11-lv_data11 on /data11 type ext4 (rw,relatime,data=ordered)
> /dev/mapper/xx12-lv_data12 on /data12 type ext4 (rw,relatime,data=ordered)
> Scratch dirs are:
> /data01/impala/impalad,/data02/impala/impalad,/data03/impala/impalad,/data04/impala/impalad,/data05/impala/impalad,/data06/impala/impalad,/data07/impala/impalad,/data08/impala/impalad,/data09/impala/impalad,/data10/impala/impalad,/data11/impala/impalad,/data12/impala/impalad
> ~~~ tmp-file-mgr.cc:122] Using scratch directory
> /data01/impala/impalad/impala-scratch on disk13.
> {noformat}
> A workaround is to set --allow_multiple_scratch_dirs_per_device=true
> There are a few questions here:
> # Does the deduplication logic even make sense to have?
> # Do we need to fix how we treat Device mapper volumes?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-6816) Statestore spends a lot of time in GetMinSubscriberTopicVersion()
[
https://issues.apache.org/jira/browse/IMPALA-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-6816 started by Tim Armstrong.
-
> Statestore spends a lot of time in GetMinSubscriberTopicVersion()
> -
>
> Key: IMPALA-6816
> URL: https://issues.apache.org/jira/browse/IMPALA-6816
> Project: IMPALA
> Issue Type: Bug
> Components: Distributed Exec
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Minor
> Labels: admission-control, statestore
>
> {noformat}
> Samples: 13K of event 'cycles', Event count (approx.): 1200870513
> 20.23% statestored impalad [.]
> impala::Statestore::GetMinSubscriberTopicVersion(std::string const&,
> std::string*)
>7.68% statestored [kernel.kallsyms][k] find_busiest_group
>3.46% statestored impalad [.]
> impala::Statestore::Subscriber::LastTopicVersionProcessed(std::string const&)
> const
>3.26% statestored libc-2.12.so [.] __memcmp_sse4_1
>1.41% statestored [kernel.kallsyms][k] find_next_bit
>1.40% statestored [kernel.kallsyms][k] cpumask_next_and
>1.21% statestored libpthread-2.12.so [.] pthread_mutex_lock
>1.04% statestored libc-2.12.so [.] memcpy
>1.01% statestored [kernel.kallsyms][k] _spin_lock
>0.98% statestored impalad [.] 0x0088f903
>0.93% statestored impalad [.] 0x0088f8f5
>0.91% statestored impalad [.] 0x0088f8ea
>0.85% statestored [kernel.kallsyms][k] ixgbe_xmit_frame_ring
>0.77% statestored impalad [.] 0x0088f8e3
>0.75% statestored impalad [.] 0x0088f900
>0.75% statestored impalad [.]
> impala::Statestore::IsPrioritizedTopic(std::string const&)
>0.73% statestored impalad [.] 0x0088f8fa
>0.72% statestored impalad [.] operator new[](unsigned
> long)
>0.68% statestored [kernel.kallsyms][k] tcp_recvmsg
>0.67% statestored impalad [.] 0x0088f8fd
>0.66% statestored impalad [.]
> impala::Statestore::Topic::BuildDelta(std::string const&, long,
> impala::TTopicDelta*)
>0.61% statestored [kernel.kallsyms][k] thread_return
>0.60% statestored impalad [.] 0x0088f8f2
>0.60% statestored libstdc++.so.6 [.]
> std::string::compare(std::string const&) const
>0.59% statestored impalad [.] 0x0088f8e6
>0.56% statestored impalad [.] 0x0088f8ee
>0.56% statestored libcrypto.so.1.0.1e [.] aesni_encrypt
>0.55% statestored impalad [.] 0x0088f8e0
>0.55% statestored [kernel.kallsyms][k] tcp_transmit_skb
>0.53% statestored [kernel.kallsyms][k] fget_light
>0.51% statestored impalad [.] std::_Rb_tree std::pair >,
> std::_Select1st0.50% statestored impalad [.]
> apache::thrift::transport::TVirtualTransport apache::thrift::transport::TBufferBase>::readAll_virt(unsigned char*
>0.50% statestored impalad [.]
> impala::Statestore::DoSubscriberUpdate(impala::Statestore::UpdateKind, int,
> impala::Statestore::ScheduledSubscriberUpdate const&)
>0.49% statestored libssl.so.1.0.1e [.] tls1_enc
>0.48% statestored libssl.so.1.0.1e [.] ssl3_read_bytes
> {noformat}
> We are spending most of our time computing this for non-catalog topics, where
> it's not even used.
> There are a couple of ways we could fix this that I can think of:
> * Avoid including this information for topics where we're not interested in it
> * Cache or precompute the value somehow to avoid iterating over all
> subscribers every time
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-5490) Upgrade to GCC 7.1
[ https://issues.apache.org/jira/browse/IMPALA-5490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong reassigned IMPALA-5490: - Assignee: (was: Tim Armstrong) > Upgrade to GCC 7.1 > -- > > Key: IMPALA-5490 > URL: https://issues.apache.org/jira/browse/IMPALA-5490 > Project: IMPALA > Issue Type: Improvement > Components: Infrastructure >Reporter: Tim Armstrong >Priority: Minor > > This has some nice features like the [[nodiscard]] annotation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-5604) Document DISABLE_CODEGEN_ROWS_THRESHOLD
[ https://issues.apache.org/jira/browse/IMPALA-5604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong resolved IMPALA-5604. --- Resolution: Fixed Fix Version/s: Impala 3.1.0 Impala 2.13.0 commit 1b0a027ffb216d729fe68e7240313af77550ae08 Author: Tim Armstrong Date: Thu Jun 14 16:13:58 2018 -0700 IMPALA-5604: document DISABLE_CODEGEN_ROWS_THRESHOLD Also fix a couple of nits in EXEC_SINGLE_NODE_ROWS_THRESHOLD. Change-Id: I709cd55e3869888feb645f85e61a99901d41d479 Reviewed-on: http://gerrit.cloudera.org:8080/10727 Reviewed-by: Alex Rodoni Tested-by: Impala Public Jenkins > Document DISABLE_CODEGEN_ROWS_THRESHOLD > --- > > Key: IMPALA-5604 > URL: https://issues.apache.org/jira/browse/IMPALA-5604 > Project: IMPALA > Issue Type: Task > Components: Docs >Affects Versions: Impala 2.10.0 >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > Fix For: Impala 2.13.0, Impala 3.1.0 > > > We just checking in DISABLE_CODEGEN_ROWS_THRESHOLD to master with default > value 50,000. > The option uses the planner's estimates to determine whether to disable > codegen for a query based on the maximum number of rows flowing through any > part of the query plan. > It is similar to EXEC_SINGLE_NODE_ROWS_THRESHOLD in behaviour except the > threshold is per-node rows instead of total rows. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-6035) Add query option that rejects queries based on query complexity
[ https://issues.apache.org/jira/browse/IMPALA-6035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong resolved IMPALA-6035. --- Resolution: Fixed Fix Version/s: Impala 3.1.0 Impala 2.13.0 > Add query option that rejects queries based on query complexity > --- > > Key: IMPALA-6035 > URL: https://issues.apache.org/jira/browse/IMPALA-6035 > Project: IMPALA > Issue Type: Sub-task > Components: Distributed Exec >Reporter: Mostafa Mokhtar >Assignee: Tim Armstrong >Priority: Major > Labels: admission-control > Fix For: Impala 2.13.0, Impala 3.1.0 > > > Expose query option to reject queries with a large number of remote > fragments. > Queries with a large number of fragments can negatively affect cluster > behavior. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6035) Add query option that rejects queries based on query complexity
[ https://issues.apache.org/jira/browse/IMPALA-6035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514047#comment-16514047 ] Tim Armstrong commented on IMPALA-6035: --- commit d8ed07f11227257254682fff3b62de8834291d66 Author: Tim Armstrong Date: Thu May 31 16:25:26 2018 -0700 IMPALA-6035: Add query options to limit thread reservation Adds two options: THREAD_RESERVATION_LIMIT and THREAD_RESERVATION_AGGREGATE_LIMIT, which are both enforced by admission control based on planner resource requirements and the schedule. The mechanism used is the same as the minimum reservation checks. THREAD_RESERVATION_LIMIT limits the total number of reserved threads in fragments scheduled on a single backend. THREAD_RESERVATION_AGGREGATE_LIMIT limits the sum of reserved threads across all fragments. This also slightly improves the minimum reservation error message to include the host name. Testing: Added end-to-end tests that exercise the code paths. Ran core tests. Change-Id: I5b5bbbdad5cd6b24442eb6c99a4d38c2ad710007 Reviewed-on: http://gerrit.cloudera.org:8080/10365 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Add query option that rejects queries based on query complexity > --- > > Key: IMPALA-6035 > URL: https://issues.apache.org/jira/browse/IMPALA-6035 > Project: IMPALA > Issue Type: Sub-task > Components: Distributed Exec >Reporter: Mostafa Mokhtar >Assignee: Tim Armstrong >Priority: Major > Labels: admission-control > Fix For: Impala 2.13.0, Impala 3.1.0 > > > Expose query option to reject queries with a large number of remote > fragments. > Queries with a large number of fragments can negatively affect cluster > behavior. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7174) TestAdmissionController.test_cancellation failed with incorrect total-admitted metric
[
https://issues.apache.org/jira/browse/IMPALA-7174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514040#comment-16514040
]
Tim Armstrong commented on IMPALA-7174:
---
---
IMPALA-7174: fix test_cancellation for RELEASE builds
The test was DOA when run against a release build because the debug
actions that it depends on were disabled. The fix is to enable the
debug actions for release builds, similar to other debug actions.
I assume the original motivation of the NDEBUG checks was to avoid
adding overhead to release builds. The cost is minimised by quickly
checking whether the string is empty before proceeding with any
further work.
Also remove wonky exception handling - the test was swallowing
exceptions but we don't expect that code to throw exceptions.
Testing:
Looped the test on a release build.
Change-Id: I41da7b5ac58a468a8ed11969906f63df6d4b
Reviewed-on: [http://gerrit.cloudera.org:8080/10722]
Reviewed-by: Impala Public Jenkins
<[impala-public-jenk...@cloudera.com|mailto:impala-public-jenk...@cloudera.com]>
Tested-by: Impala Public Jenkins
<[impala-public-jenk...@cloudera.com|mailto:impala-public-jenk...@cloudera.com]>
> TestAdmissionController.test_cancellation failed with incorrect
> total-admitted metric
> -
>
> Key: IMPALA-7174
> URL: https://issues.apache.org/jira/browse/IMPALA-7174
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Tianyi Wang
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: broken-build
> Fix For: Impala 2.13.0, Impala 3.1.0
>
> Attachments:
> impalad.ec2-m2-4xlarge-centos-6-4-01f7.vpc.cloudera.com.jenkins.log.INFO.20180614-060607.2553
>
>
> The failed revision is ee9a9b6c5000cf915716a15ea8a0b3605290a9a5, an
> descendant of 'IMPALA-5216: Make admission control queuing async'.
> {noformat}
> Stacktrace
> custom_cluster/test_admission_controller.py:557: in test_cancellation
> assert self.cluster.impalads[0].service.get_metric_value(
> E assert 0 == 3
> E+ where 0 = 0x4ed1b90>>('admission-controller.total-admitted.default-pool')
> E+where > =
> 0x4ed1b90>.get_metric_value
> E+ where 0x4ed1b90> = 0x52474d0>.service
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-6874) Add more tests for mixed-format tables, including parquet
[ https://issues.apache.org/jira/browse/IMPALA-6874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong reassigned IMPALA-6874: - Assignee: (was: Tim Armstrong) > Add more tests for mixed-format tables, including parquet > - > > Key: IMPALA-6874 > URL: https://issues.apache.org/jira/browse/IMPALA-6874 > Project: IMPALA > Issue Type: Test > Components: Infrastructure >Affects Versions: Impala 2.12.0 >Reporter: Tim Armstrong >Priority: Major > > We only have a single very basic table with mixed formats that I can see. It > would be good to have a larger table that includes parquet, so we can > exercise some of the more interesting memory reservation code paths. I think > the requirements are: > * Non-tiny files > * Includes Parquet and at least one row-based format > * Ideally a mix of file sizes -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7174) TestAdmissionController.test_cancellation failed with incorrect total-admitted metric
[
https://issues.apache.org/jira/browse/IMPALA-7174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-7174.
---
Resolution: Fixed
Fix Version/s: Impala 3.1.0
Impala 2.13.0
> TestAdmissionController.test_cancellation failed with incorrect
> total-admitted metric
> -
>
> Key: IMPALA-7174
> URL: https://issues.apache.org/jira/browse/IMPALA-7174
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Tianyi Wang
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: broken-build
> Fix For: Impala 2.13.0, Impala 3.1.0
>
> Attachments:
> impalad.ec2-m2-4xlarge-centos-6-4-01f7.vpc.cloudera.com.jenkins.log.INFO.20180614-060607.2553
>
>
> The failed revision is ee9a9b6c5000cf915716a15ea8a0b3605290a9a5, an
> descendant of 'IMPALA-5216: Make admission control queuing async'.
> {noformat}
> Stacktrace
> custom_cluster/test_admission_controller.py:557: in test_cancellation
> assert self.cluster.impalads[0].service.get_metric_value(
> E assert 0 == 3
> E+ where 0 = 0x4ed1b90>>('admission-controller.total-admitted.default-pool')
> E+where > =
> 0x4ed1b90>.get_metric_value
> E+ where 0x4ed1b90> = 0x52474d0>.service
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7178) Reduce logging for common data errors
[ https://issues.apache.org/jira/browse/IMPALA-7178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Csaba Ringhofer updated IMPALA-7178: Description: Some data errors (for example out-of-range parquet timestamps) can dominate logs if a table contains a large number of rows with invalid data. If an error has its own error code (see common/thrift/generate_error_codes.py), then these errors are already aggregated to the user (RuntimeState::LogError()) for every query, but the logs will contain a new line for every occurrence. This is not too useful most of times, as the log lines will repeat the same information (the corrupt data itself is not logged as it can be sensitive information). The best would to reduce logging without loosing information: - the first occurrence of an error should be logged (per query/fragment/table/file/column) to help the investigation of cases where the data error leads to other errors and to avoid breaking log analyzer tools that search for the current format - other occurrences can be aggregated, like "in query Q table T column C XY error occurred N times" An extra goal is to avoid calling RuntimeState::LogError() for other occurrences than the first one, as RuntimeState::LogError() uses a (per fragment) lock. was: Some data errors (for example out-of-range parquet timestamps) can dominate logs if a table contains a large number of rows with invalid data. If an error has its own error code (see common/thrift/generate_error_codes.py), then these errors are already aggregated to the user (RuntimeState::LogError()) for every query, but the logs will contain a new line for every occurrence. This not too useful most of times, as the log lines will repeat the same information (the corrupt data itself is not logged as it can be sensitive information). The best would to reduce logging without loosing information: - the first occurrence of an error should be logged (per query/fragment/table/file/column) to help investigation of cases where the data error leads to other errors and to avoid breaking log analyzer tools that search for the current format - other occurrences can be aggregated, like "in query Q table T column C XY error occurred N times" An extra goal is to avoid calling RuntimeState::LogError() for other occurrences than the first one, as RuntimeState::LogError() uses a lock. > Reduce logging for common data errors > - > > Key: IMPALA-7178 > URL: https://issues.apache.org/jira/browse/IMPALA-7178 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Reporter: Csaba Ringhofer >Assignee: Csaba Ringhofer >Priority: Major > > Some data errors (for example out-of-range parquet timestamps) can dominate > logs if a table contains a large number of rows with invalid data. If an > error has its own error code (see common/thrift/generate_error_codes.py), > then these errors are already aggregated to the user > (RuntimeState::LogError()) for every query, but the logs will contain a new > line for every occurrence. This is not too useful most of times, as the log > lines will repeat the same information (the corrupt data itself is not logged > as it can be sensitive information). > The best would to reduce logging without loosing information: > - the first occurrence of an error should be logged (per > query/fragment/table/file/column) to help the investigation of cases where > the data error leads to other errors and to avoid breaking log analyzer tools > that search for the current format > - other occurrences can be aggregated, like "in query Q table T column C XY > error occurred N times" > An extra goal is to avoid calling RuntimeState::LogError() for other > occurrences than the first one, as RuntimeState::LogError() uses a (per > fragment) lock. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7178) Reduce logging for common data errors
Csaba Ringhofer created IMPALA-7178: --- Summary: Reduce logging for common data errors Key: IMPALA-7178 URL: https://issues.apache.org/jira/browse/IMPALA-7178 Project: IMPALA Issue Type: Improvement Components: Backend Reporter: Csaba Ringhofer Assignee: Csaba Ringhofer Some data errors (for example out-of-range parquet timestamps) can dominate logs if a table contains a large number of rows with invalid data. If an error has its own error code (see common/thrift/generate_error_codes.py), then these errors are already aggregated to the user (RuntimeState::LogError()) for every query, but the logs will contain a new line for every occurrence. This not too useful most of times, as the log lines will repeat the same information (the corrupt data itself is not logged as it can be sensitive information). The best would to reduce logging without loosing information: - the first occurrence of an error should be logged (per query/fragment/table/file/column) to help investigation of cases where the data error leads to other errors and to avoid breaking log analyzer tools that search for the current format - other occurrences can be aggregated, like "in query Q table T column C XY error occurred N times" An extra goal is to avoid calling RuntimeState::LogError() for other occurrences than the first one, as RuntimeState::LogError() uses a lock. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
