[jira] [Commented] (IMPALA-7766) Perform constant folding within an expression
[
https://issues.apache.org/jira/browse/IMPALA-7766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664552#comment-16664552
]
Paul Rogers commented on IMPALA-7766:
-
[~tarmstrong], makes sense, that seems how a parser would reduce
left-associative expressions.
We do have a {{NormalizeExprsRule}}. I'll check if that can do the rewrite you
suggest.
> Perform constant folding within an expression
> -
>
> Key: IMPALA-7766
> URL: https://issues.apache.org/jira/browse/IMPALA-7766
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
>Affects Versions: Impala 3.0
>Reporter: Paul Rogers
>Priority: Minor
>
> Suppose we have the following query:
> {code:sql}
> SELECT id + 1 + 2, 2 + 3 FROM foo;
> {code}
> Then, examine the SELECT after rewrites (using the new {{FullRewriteTest}}.)
> We'd expect content folding. However, constant folding only happens if the
> entire expression is constant, it can't handle sub-expressions. Result:
> {code:sql}
> SELECT id + 1 + 2, 5 FROM foo;
> {code}
> Constant folding within an expression is tricky, and it is not clear what, if
> any, performance gain would be had. Still, it is worth keeping in mind.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7769) Constant folding sometimes introduces an unnecessary cast
[
https://issues.apache.org/jira/browse/IMPALA-7769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Rogers updated IMPALA-7769:
Description:
Consider the following query:
{code:sql}
SELECT IFNULL(NULL + 1, id) FROM alltypessmall;
{code}
Visualize the rewritten query after analysis (using the new
{{FullRewriteTest}}.) We get:
{code:sql}
SELECT CASE WHEN CAST(NULL AS INT) IS NULL THEN id ELSE NULL END FROM
alltypessmall
{code}
(This is what the expression actually contains. The {{toSql()}} method lies and
says that the statement is:
{code:sql}
SELECT CASE WHEN NULL IS NULL THEN id ELSE NULL END FROM alltypessmall
{code}
which causes confusion when debugging.)
Expected:
{code:sql}
SELECT id FROM alltypessmall
{code}
The reason is that there are multiple rewrites, one of which has a flaw.
# Rewrite IFNULL to CASE
# Rewrite NULL + 1 to CAST(NULL AS SMALLINT)
# Try to rewrite CAST(NULL AS SMALLINT) IS NULL, but fail because CAST is not a
literal.
The code in question in {{FoldConstantsRule.apply()}} is:
{code:java}
for (Expr child: expr.getChildren()) if (!child.isLiteral()) return expr;
{code}
In fact, this check is too restrictive. Need a new {{isLiteralLike()}} which
should work like {{IsNullLiteral()}}:
* True if the node is a literal.
* True if the node is a cast of a literal.
(Can't change {{isLiteral()}} since there are places that assume that this
existing predicate indicates that the node is exactly a {{LiteralExpr}}.)
Impala already has a predicate that does what is needed: {{isConstant()}}.
However, the code in {{FoldConstantsRule.apply()}} specifically excludes
calling it:
{code:java}
// Avoid calling Expr.isConstant() because that would lead to repeated
traversals
// of the Expr tree. Assumes the bottom-up application of this rule.
Constant
// children should have been folded at this point.
{code}
The new method solves the repeated traversal problem. With it, the test query
now simplifies to the expected result.
was:
Consider the following query:
{code:sql}
SELECT IFNULL(NULL + 1, id) FROM alltypessmall;
{code}
Visualize the rewritten query after analysis (using the new
{{FullRewriteTest}}.) We get:
{code:sql}
SELECT CASE WHEN CAST(NULL AS INT) IS NULL THEN id ELSE NULL END FROM
alltypessmall
{code}
(This is what the expression actually contains. The {{toSql()}} method lies and
says that the statement is:
{code:sql}
SELECT CASE WHEN NULL IS NULL THEN id ELSE NULL END FROM alltypessmall
{code}
which causes confusion when debugging.)
Expected:
{code:sql}
SELECT id FROM alltypessmall
{code}
The reason is that there are multiple rewrites, one of which has a flaw.
# Rewrite IFNULL to CASE
# Rewrite NULL + 1 to CAST(NULL AS SMALLINT)
# Try to rewrite CAST(NULL AS SMALLINT) IS NULL, but fail because CAST is not a
literal.
The code in question in {{FoldConstantsRule.apply()}} is:
{code:java}
for (Expr child: expr.getChildren()) if (!child.isLiteral()) return expr;
{code}
In fact, this check is too restrictive. Need a new {{isLiteralLike()}} which
should work like {{IsNullLiteral()}}:
* True if the node is a literal.
* True if the node is a cast of a literal.
(Can't change {{isLiteral()}} since there are places that assume that this
existing predicate indicates that the node is exactly a {{LiteralExpr}}.)
Impala already has a predicate that does what is needed: {{isConstant()}}.
However, the code in {{FoldConstantsRule.apply()}} specifically excludes
calling it:
{code:java}
// Avoid calling Expr.isConstant() because that would lead to repeated
traversals
// of the Expr tree. Assumes the bottom-up application of this rule.
Constant
// children should have been folded at this point.
{code}
The new method solves the problem and the test query now simplifies to the
expected result.
> Constant folding sometimes introduces an unnecessary cast
> -
>
> Key: IMPALA-7769
> URL: https://issues.apache.org/jira/browse/IMPALA-7769
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
>Affects Versions: Impala 3.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Consider the following query:
> {code:sql}
> SELECT IFNULL(NULL + 1, id) FROM alltypessmall;
> {code}
> Visualize the rewritten query after analysis (using the new
> {{FullRewriteTest}}.) We get:
> {code:sql}
> SELECT CASE WHEN CAST(NULL AS INT) IS NULL THEN id ELSE NULL END FROM
> alltypessmall
> {code}
> (This is what the expression actually contains. The {{toSql()}} method lies
> and says that the statement is:
> {code:sql}
> SELECT CASE WHEN NULL IS NULL THEN id ELSE NULL END FROM alltypessmall
> {code}
> which causes confusion when debugging.)
> Expected:
> {code:sql}
> SELECT id FROM alltypessmall
>
[jira] [Updated] (IMPALA-7769) Constant folding sometimes introduces an unnecessary cast
[
https://issues.apache.org/jira/browse/IMPALA-7769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Rogers updated IMPALA-7769:
Description:
Consider the following query:
{code:sql}
SELECT IFNULL(NULL + 1, id) FROM alltypessmall;
{code}
Visualize the rewritten query after analysis (using the new
{{FullRewriteTest}}.) We get:
{code:sql}
SELECT CASE WHEN CAST(NULL AS INT) IS NULL THEN id ELSE NULL END FROM
alltypessmall
{code}
(This is what the expression actually contains. The {{toSql()}} method lies and
says that the statement is:
{code:sql}
SELECT CASE WHEN NULL IS NULL THEN id ELSE NULL END FROM alltypessmall
{code}
which causes confusion when debugging.)
Expected:
{code:sql}
SELECT id FROM alltypessmall
{code}
The reason is that there are multiple rewrites, one of which has a flaw.
# Rewrite IFNULL to CASE
# Rewrite NULL + 1 to CAST(NULL AS SMALLINT)
# Try to rewrite CAST(NULL AS SMALLINT) IS NULL, but fail because CAST is not a
literal.
The code in question in {{FoldConstantsRule.apply()}} is:
{code:java}
for (Expr child: expr.getChildren()) if (!child.isLiteral()) return expr;
{code}
In fact, this check is too restrictive. Need a new {{isLiteralLike()}} which
should work like {{IsNullLiteral()}}:
* True if the node is a literal.
* True if the node is a cast of a literal.
(Can't change {{isLiteral()}} since there are places that assume that this
existing predicate indicates that the node is exactly a {{LiteralExpr}}.)
Impala already has a predicate that does what is needed: {{isConstant()}}.
However, the code in {{FoldConstantsRule.apply()}} specifically excludes
calling it:
{code:java}
// Avoid calling Expr.isConstant() because that would lead to repeated
traversals
// of the Expr tree. Assumes the bottom-up application of this rule.
Constant
// children should have been folded at this point.
{code}
The new method solves the problem and the test query now simplifies to the
expected result.
was:
Consider the following query:
{code:sql}
SELECT IFNULL(NULL + 1, id) FROM alltypessmall;
{code}
Visualize the rewritten query after analysis (using the new
{{FullRewriteTest}}.) We get:
{code:sql}
SELECT CASE WHEN CAST(NULL AS INT) IS NULL THEN id ELSE NULL END FROM
alltypessmall
{code}
(This is what the expression actually contains. The {{toSql()}} method lies and
says that the statement is:
{code:sql}
SELECT CASE WHEN NULL IS NULL THEN id ELSE NULL END FROM alltypessmall
{code}
which causes confusion when debugging.)
Expected:
{code:sql}
SELECT id FROM alltypessmall
{code}
The reason is that there are multiple rewrites, one of which has a flaw.
# Rewrite IFNULL to CASE
# Rewrite NULL + 1 to CAST(NULL AS SMALLINT)
# Try to rewrite CAST(NULL AS SMALLINT) IS NULL, but fail because CAST is not a
literal.
The code in question in {{FoldConstantsRule.apply()}} is:
{code:java}
for (Expr child: expr.getChildren()) if (!child.isLiteral()) return expr;
{code}
In fact, this check is too restrictive. Need a new {{isLiteralLike()}} should
work like {{IsNullLiteral()}}:
* True if the node is a literal.
* True if the node is a cast of a literal.
(Can't change {{isLiteral()}} since there are places that assume that this
existing predicate indicates that the node is exactly a {{LiteralExpr}}.)
Impala already has a predicate that does what is needed: {{isConstant()}}.
However, the code in {{FoldConstantsRule.apply()}} specifically excludes
calling it:
{code:java}
// Avoid calling Expr.isConstant() because that would lead to repeated
traversals
// of the Expr tree. Assumes the bottom-up application of this rule.
Constant
// children should have been folded at this point.
{code}
The new method solves the problem and the test query now simplifies to the
expected result.
> Constant folding sometimes introduces an unnecessary cast
> -
>
> Key: IMPALA-7769
> URL: https://issues.apache.org/jira/browse/IMPALA-7769
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
>Affects Versions: Impala 3.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Consider the following query:
> {code:sql}
> SELECT IFNULL(NULL + 1, id) FROM alltypessmall;
> {code}
> Visualize the rewritten query after analysis (using the new
> {{FullRewriteTest}}.) We get:
> {code:sql}
> SELECT CASE WHEN CAST(NULL AS INT) IS NULL THEN id ELSE NULL END FROM
> alltypessmall
> {code}
> (This is what the expression actually contains. The {{toSql()}} method lies
> and says that the statement is:
> {code:sql}
> SELECT CASE WHEN NULL IS NULL THEN id ELSE NULL END FROM alltypessmall
> {code}
> which causes confusion when debugging.)
> Expected:
> {code:sql}
> SELECT id FROM alltypessmall
> {code}
> The reason is that there
[jira] [Updated] (IMPALA-7769) Constant folding sometimes introduces an unnecessary cast
[
https://issues.apache.org/jira/browse/IMPALA-7769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Rogers updated IMPALA-7769:
Description:
Consider the following query:
{code:sql}
SELECT IFNULL(NULL + 1, id) FROM alltypessmall;
{code}
Visualize the rewritten query after analysis (using the new
{{FullRewriteTest}}.) We get:
{code:sql}
SELECT CASE WHEN CAST(NULL AS INT) IS NULL THEN id ELSE NULL END FROM
alltypessmall
{code}
(This is what the expression actually contains. The {{toSql()}} method lies and
says that the statement is:
{code:sql}
SELECT CASE WHEN NULL IS NULL THEN id ELSE NULL END FROM alltypessmall
{code}
which causes confusion when debugging.)
Expected:
{code:sql}
SELECT id FROM alltypessmall
{code}
The reason is that there are multiple rewrites, one of which has a flaw.
# Rewrite IFNULL to CASE
# Rewrite NULL + 1 to CAST(NULL AS SMALLINT)
# Try to rewrite CAST(NULL AS SMALLINT) IS NULL, but fail because CAST is not a
literal.
The code in question in {{FoldConstantsRule.apply()}} is:
{code:java}
for (Expr child: expr.getChildren()) if (!child.isLiteral()) return expr;
{code}
In fact, this check is too restrictive. Need a new {{isLiteralLike()}} should
work like {{IsNullLiteral()}}:
* True if the node is a literal.
* True if the node is a cast of a literal.
(Can't change {{isLiteral()}} since there are places that assume that this
existing predicate indicates that the node is exactly a {{LiteralExpr}}.)
Impala already has a predicate that does what is needed: {{isConstant()}}.
However, the code in {{FoldConstantsRule.apply()}} specifically excludes
calling it:
{code:java}
// Avoid calling Expr.isConstant() because that would lead to repeated
traversals
// of the Expr tree. Assumes the bottom-up application of this rule.
Constant
// children should have been folded at this point.
{code}
The new method solves the problem and the test query now simplifies to the
expected result.
was:
Consider the following query:
{code:sql}
SELECT IFNULL(NULL + 1, id) FROM alltypessmall;
{code}
Visualize the rewritten query after analysis (using the new
{{FullRewriteTest}}.) We get:
{code:sql}
SELECT CASE WHEN NULL IS NULL THEN id ELSE NULL END FROM alltypessmall
{code}
Expected:
{code:sql}
SELECT id FROM alltypessmall
{code}
The reason is that there are multiple rewrites, one of which has a flaw.
# Rewrite IFNULL to CASE
# Rewrite NULL + 1 to CAST(NULL AS SMALLINT)
# Try to rewrite CAST(NULL AS SMALLINT) IS NULL, but fail because CAST is not a
literal.
The code in question in {{FoldConstantsRule.apply()}} is:
{code:java}
for (Expr child: expr.getChildren()) if (!child.isLiteral()) return expr;
{code}
In fact, this check is too restrictive. Need a new {{isLiteralLike()}} should
work like {{IsNullLiteral()}}:
* True if the node is a literal.
* True if the node is a cast of a literal.
(Can't change {{isLiteral()}} since there are places that assume that this
existing predicate indicates that the node is exactly a {{LiteralExpr}}.)
Impala already has a predicate that does what is needed: {{isConstant()}}.
However, the code in {{FoldConstantsRule.apply()}} specifically excludes
calling it:
{code:java}
// Avoid calling Expr.isConstant() because that would lead to repeated
traversals
// of the Expr tree. Assumes the bottom-up application of this rule.
Constant
// children should have been folded at this point.
{code}
The new method solves the problem and the test query now simplifies to the
expected result.
> Constant folding sometimes introduces an unnecessary cast
> -
>
> Key: IMPALA-7769
> URL: https://issues.apache.org/jira/browse/IMPALA-7769
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
>Affects Versions: Impala 3.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Consider the following query:
> {code:sql}
> SELECT IFNULL(NULL + 1, id) FROM alltypessmall;
> {code}
> Visualize the rewritten query after analysis (using the new
> {{FullRewriteTest}}.) We get:
> {code:sql}
> SELECT CASE WHEN CAST(NULL AS INT) IS NULL THEN id ELSE NULL END FROM
> alltypessmall
> {code}
> (This is what the expression actually contains. The {{toSql()}} method lies
> and says that the statement is:
> {code:sql}
> SELECT CASE WHEN NULL IS NULL THEN id ELSE NULL END FROM alltypessmall
> {code}
> which causes confusion when debugging.)
> Expected:
> {code:sql}
> SELECT id FROM alltypessmall
> {code}
> The reason is that there are multiple rewrites, one of which has a flaw.
> # Rewrite IFNULL to CASE
> # Rewrite NULL + 1 to CAST(NULL AS SMALLINT)
> # Try to rewrite CAST(NULL AS SMALLINT) IS NULL, but fail because CAST is not
> a literal.
> The code in question in
[jira] [Created] (IMPALA-7769) Constant folding sometimes introduces an unnecessary cast
Paul Rogers created IMPALA-7769:
---
Summary: Constant folding sometimes introduces an unnecessary cast
Key: IMPALA-7769
URL: https://issues.apache.org/jira/browse/IMPALA-7769
Project: IMPALA
Issue Type: Improvement
Components: Frontend
Affects Versions: Impala 3.0
Reporter: Paul Rogers
Assignee: Paul Rogers
Consider the following query:
{code:sql}
SELECT IFNULL(NULL + 1, id) FROM alltypessmall;
{code}
Visualize the rewritten query after analysis (using the new
{{FullRewriteTest}}.) We get:
{code:sql}
SELECT CASE WHEN NULL IS NULL THEN id ELSE NULL END FROM alltypessmall
{code}
Expected:
{code:sql}
SELECT id FROM alltypessmall
{code}
The reason is that there are multiple rewrites, one of which has a flaw.
# Rewrite IFNULL to CASE
# Rewrite NULL + 1 to CAST(NULL AS SMALLINT)
# Try to rewrite CAST(NULL AS SMALLINT) IS NULL, but fail because CAST is not a
literal.
The code in question in {{FoldConstantsRule.apply()}} is:
{code:java}
for (Expr child: expr.getChildren()) if (!child.isLiteral()) return expr;
{code}
In fact, this check is too restrictive. Need a new {{isLiteralLike()}} should
work like {{IsNullLiteral()}}:
* True if the node is a literal.
* True if the node is a cast of a literal.
(Can't change {{isLiteral()}} since there are places that assume that this
existing predicate indicates that the node is exactly a {{LiteralExpr}}.)
Impala already has a predicate that does what is needed: {{isConstant()}}.
However, the code in {{FoldConstantsRule.apply()}} specifically excludes
calling it:
{code:java}
// Avoid calling Expr.isConstant() because that would lead to repeated
traversals
// of the Expr tree. Assumes the bottom-up application of this rule.
Constant
// children should have been folded at this point.
{code}
The new method solves the problem and the test query now simplifies to the
expected result.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7769) Constant folding sometimes introduces an unnecessary cast
Paul Rogers created IMPALA-7769:
---
Summary: Constant folding sometimes introduces an unnecessary cast
Key: IMPALA-7769
URL: https://issues.apache.org/jira/browse/IMPALA-7769
Project: IMPALA
Issue Type: Improvement
Components: Frontend
Affects Versions: Impala 3.0
Reporter: Paul Rogers
Assignee: Paul Rogers
Consider the following query:
{code:sql}
SELECT IFNULL(NULL + 1, id) FROM alltypessmall;
{code}
Visualize the rewritten query after analysis (using the new
{{FullRewriteTest}}.) We get:
{code:sql}
SELECT CASE WHEN NULL IS NULL THEN id ELSE NULL END FROM alltypessmall
{code}
Expected:
{code:sql}
SELECT id FROM alltypessmall
{code}
The reason is that there are multiple rewrites, one of which has a flaw.
# Rewrite IFNULL to CASE
# Rewrite NULL + 1 to CAST(NULL AS SMALLINT)
# Try to rewrite CAST(NULL AS SMALLINT) IS NULL, but fail because CAST is not a
literal.
The code in question in {{FoldConstantsRule.apply()}} is:
{code:java}
for (Expr child: expr.getChildren()) if (!child.isLiteral()) return expr;
{code}
In fact, this check is too restrictive. Need a new {{isLiteralLike()}} should
work like {{IsNullLiteral()}}:
* True if the node is a literal.
* True if the node is a cast of a literal.
(Can't change {{isLiteral()}} since there are places that assume that this
existing predicate indicates that the node is exactly a {{LiteralExpr}}.)
Impala already has a predicate that does what is needed: {{isConstant()}}.
However, the code in {{FoldConstantsRule.apply()}} specifically excludes
calling it:
{code:java}
// Avoid calling Expr.isConstant() because that would lead to repeated
traversals
// of the Expr tree. Assumes the bottom-up application of this rule.
Constant
// children should have been folded at this point.
{code}
The new method solves the problem and the test query now simplifies to the
expected result.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (IMPALA-7714) Statestore::Subscriber::SetLastTopicVersionProcessed() crashed in AtomicInt64::Store()
[
https://issues.apache.org/jira/browse/IMPALA-7714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-7714.
---
Resolution: Fixed
Fix Version/s: Impala 3.1.0
> Statestore::Subscriber::SetLastTopicVersionProcessed() crashed in
> AtomicInt64::Store()
> --
>
> Key: IMPALA-7714
> URL: https://issues.apache.org/jira/browse/IMPALA-7714
> Project: IMPALA
> Issue Type: Bug
> Components: Distributed Exec
>Affects Versions: Impala 3.1.0
>Reporter: Michael Ho
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: broken-build
> Fix For: Impala 3.1.0
>
> Attachments: dbfd9687-09a9-4ab0-dcd7128b-41a2c5b3.dmp.resolved
>
>
> When running one of the customer cluster tests,
> {{Statestore::Subscriber::SetLastTopicVersionProcessed()}} most likely
> crashed at the following line. It could be a race or something but I didn't
> have time to dig more into it.
> {noformat}
> void Statestore::Subscriber::SetLastTopicVersionProcessed(const TopicId&
> topic_id,
> TopicEntry::Version version) {
> // Safe to call concurrently for different topics because
> 'subscribed_topics' is not
> // modified.
> Topics* subscribed_topics = GetTopicsMapForId(topic_id);
> Topics::iterator topic_it = subscribed_topics->find(topic_id);
> DCHECK(topic_it != subscribed_topics->end());
> topic_it->second.last_version.Store(version); <<-
> }
> {noformat}
> {noformat}
> Error Message
> Minidump generated:
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/logs/custom_cluster_tests/minidumps/statestored/336d9ca9-88dc-4360-6a5adf97-936db5c0.dmp
> Standard Error
> Operating system: Linux
> 0.0.0 Linux 3.10.0-693.5.2.el7.x86_64 #1 SMP Fri Oct 20
> 20:32:50 UTC 2017 x86_64
> CPU: amd64
> family 6 model 85 stepping 4
> 1 CPU
> GPU: UNKNOWN
> Crash reason: SIGSEGV
> Crash address: 0x28
> Process uptime: not available
> Thread 18 (crashed)
> 0
> impalad!impala::Statestore::Subscriber::SetLastTopicVersionProcessed(std::string
> const&, long) [atomicops-internals-x86.h : 300 + 0x0]
> rax = 0x rdx = 0xc34174ed
> rcx = 0x0022c65a25a97b5b rbx = 0x04624e38
> rsi = 0x0070 rdi = 0x04906a79
> rbp = 0x7fd582d81320 rsp = 0x7fd582d812e0
> r8 = 0x9e3779b9r9 = 0x
> r10 = 0x r11 = 0x7fd58da31a90
> r12 = 0x83bfbe948682e9da r13 = 0x04593e20
> r14 = 0x000f r15 = 0x000a
> rip = 0x01022a65
> Found by: given as instruction pointer in context
> 1
> impalad!impala::Statestore::SendTopicUpdate(impala::Statestore::Subscriber*,
> impala::Statestore::UpdateKind, bool*) [statestore.cc : 704 + 0x12]
> rbx = 0x7fd582d813d0 rbp = 0x7fd582d81580
> rsp = 0x7fd582d81330 r12 = 0x04593e00
> r13 = 0x04624dd0 r14 = 0x7fd582d81508
> r15 = 0x7fd582d814f0 rip = 0x010283da
> Found by: call frame info
> 2
> impalad!impala::Statestore::DoSubscriberUpdate(impala::Statestore::UpdateKind,
> int, impala::Statestore::ScheduledSubscriberUpdate const&) [statestore.cc :
> 933 + 0x23]
> rbx = 0x rbp = 0x7fd582d817d0
> rsp = 0x7fd582d81590 r12 = 0x7fd582d81840
> r13 = 0x20c49ba5e353f7cf r14 = 0x01667beb277f
> r15 = 0x7ffc38ca1080 rip = 0x01029064
> Found by: call frame info
> 3
> impalad!impala::ThreadPool::WorkerThread(int)
> [function_template.hpp : 767 + 0x10]
> rbx = 0x7ffc38ca1500 rbp = 0x7fd582d818a0
> rsp = 0x7fd582d817e0 r12 = 0x7ffc38ca1720
> r13 = 0x7fd582d81830 r14 = 0x7fd582d81840
> r15 = 0x rip = 0x01030bdc
> Found by: call frame info
> 4 impalad!impala::Thread::SuperviseThread(std::string const&, std::string
> const&, boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*) [function_template.hpp : 767
> + 0x7]
> rbx = 0x7fd582d81980 rbp = 0x7fd582d81bf0
> rsp = 0x7fd582d818b0 r12 = 0x
> r13 = 0x04658300 r14 = 0x7fd58e6af6a0
> r15 = 0x7ffc38ca07a0 rip = 0x010fec72
> Found by: call frame info
> 5 impalad!boost::detail::thread_data (*)(std::string const&, std::string const&, boost::function,
> impala::ThreadDebugInfo const*, impala::Promise (impala::PromiseMode)0>*), boost::_bi::list5,
> boost::_bi::value, boost::_bi::value >,
> boost::_bi::value,
> boost::_bi::value*> > >
> >::run() [bind.hpp : 525 + 0x6]
> rbx = 0x045f0600 rbp = 0x7fd582d81c50
> rsp =
[jira] [Commented] (IMPALA-6271) Impala daemon should log a message when it's being shut down
[ https://issues.apache.org/jira/browse/IMPALA-6271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664523#comment-16664523 ] ASF subversion and git services commented on IMPALA-6271: - Commit 93ee538c5403509804cb18411e4407352f5b242b in impala's branch refs/heads/master from [~tarmstr...@cloudera.com] [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=93ee538 ] IMPALA-7714: remove unsafe code from signal handlers IMPALA-6271 added LOG statements to some signal handlers and an exit() call to a different signal handler. These functions are not async-signal safe. The fixes are: * Use the write system call directly. I tried using glog's RAW_LOG functionality but had major issues getting it to work. * Call _exit() directly instead of exit() so that static destructors are not run. This is the same default behaviour as SIGTERM. This wans't necessary to prevent this specific crash. Testing: Could reproduce the crash by looping tests/custom_cluster/test_local_catalog.py until a minidump was produced. After this fix it did not reproduce after looping for a few hours. Ran exhaustive build. Change-Id: I52037d6510b9b34ec33d3a8b5492226aee1b9d92 Reviewed-on: http://gerrit.cloudera.org:8080/11777 Tested-by: Impala Public Jenkins Reviewed-by: Tim Armstrong > Impala daemon should log a message when it's being shut down > > > Key: IMPALA-6271 > URL: https://issues.apache.org/jira/browse/IMPALA-6271 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 2.10.0 >Reporter: Zoram Thanga >Assignee: Pranay Singh >Priority: Major > Labels: observability, supportability > Fix For: Impala 3.1.0 > > > At present the Impala daemon does not log any message when it is being shut > down, usually via SIGTERM from management software or OS shutdown. It would > be good to at the very least catch this signal to log a message that we are > going down. This will aid in serviceability. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7714) Statestore::Subscriber::SetLastTopicVersionProcessed() crashed in AtomicInt64::Store()
[
https://issues.apache.org/jira/browse/IMPALA-7714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664522#comment-16664522
]
ASF subversion and git services commented on IMPALA-7714:
-
Commit 93ee538c5403509804cb18411e4407352f5b242b in impala's branch
refs/heads/master from [~tarmstr...@cloudera.com]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=93ee538 ]
IMPALA-7714: remove unsafe code from signal handlers
IMPALA-6271 added LOG statements to some signal handlers and an exit()
call to a different signal handler. These functions are not async-signal
safe.
The fixes are:
* Use the write system call directly. I tried using glog's RAW_LOG
functionality but had major issues getting it to work.
* Call _exit() directly instead of exit() so that static destructors
are not run. This is the same default behaviour as SIGTERM. This
wans't necessary to prevent this specific crash.
Testing:
Could reproduce the crash by looping
tests/custom_cluster/test_local_catalog.py until a minidump was
produced. After this fix it did not reproduce after looping for
a few hours.
Ran exhaustive build.
Change-Id: I52037d6510b9b34ec33d3a8b5492226aee1b9d92
Reviewed-on: http://gerrit.cloudera.org:8080/11777
Tested-by: Impala Public Jenkins
Reviewed-by: Tim Armstrong
> Statestore::Subscriber::SetLastTopicVersionProcessed() crashed in
> AtomicInt64::Store()
> --
>
> Key: IMPALA-7714
> URL: https://issues.apache.org/jira/browse/IMPALA-7714
> Project: IMPALA
> Issue Type: Bug
> Components: Distributed Exec
>Affects Versions: Impala 3.1.0
>Reporter: Michael Ho
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: broken-build
> Fix For: Impala 3.1.0
>
> Attachments: dbfd9687-09a9-4ab0-dcd7128b-41a2c5b3.dmp.resolved
>
>
> When running one of the customer cluster tests,
> {{Statestore::Subscriber::SetLastTopicVersionProcessed()}} most likely
> crashed at the following line. It could be a race or something but I didn't
> have time to dig more into it.
> {noformat}
> void Statestore::Subscriber::SetLastTopicVersionProcessed(const TopicId&
> topic_id,
> TopicEntry::Version version) {
> // Safe to call concurrently for different topics because
> 'subscribed_topics' is not
> // modified.
> Topics* subscribed_topics = GetTopicsMapForId(topic_id);
> Topics::iterator topic_it = subscribed_topics->find(topic_id);
> DCHECK(topic_it != subscribed_topics->end());
> topic_it->second.last_version.Store(version); <<-
> }
> {noformat}
> {noformat}
> Error Message
> Minidump generated:
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/logs/custom_cluster_tests/minidumps/statestored/336d9ca9-88dc-4360-6a5adf97-936db5c0.dmp
> Standard Error
> Operating system: Linux
> 0.0.0 Linux 3.10.0-693.5.2.el7.x86_64 #1 SMP Fri Oct 20
> 20:32:50 UTC 2017 x86_64
> CPU: amd64
> family 6 model 85 stepping 4
> 1 CPU
> GPU: UNKNOWN
> Crash reason: SIGSEGV
> Crash address: 0x28
> Process uptime: not available
> Thread 18 (crashed)
> 0
> impalad!impala::Statestore::Subscriber::SetLastTopicVersionProcessed(std::string
> const&, long) [atomicops-internals-x86.h : 300 + 0x0]
> rax = 0x rdx = 0xc34174ed
> rcx = 0x0022c65a25a97b5b rbx = 0x04624e38
> rsi = 0x0070 rdi = 0x04906a79
> rbp = 0x7fd582d81320 rsp = 0x7fd582d812e0
> r8 = 0x9e3779b9r9 = 0x
> r10 = 0x r11 = 0x7fd58da31a90
> r12 = 0x83bfbe948682e9da r13 = 0x04593e20
> r14 = 0x000f r15 = 0x000a
> rip = 0x01022a65
> Found by: given as instruction pointer in context
> 1
> impalad!impala::Statestore::SendTopicUpdate(impala::Statestore::Subscriber*,
> impala::Statestore::UpdateKind, bool*) [statestore.cc : 704 + 0x12]
> rbx = 0x7fd582d813d0 rbp = 0x7fd582d81580
> rsp = 0x7fd582d81330 r12 = 0x04593e00
> r13 = 0x04624dd0 r14 = 0x7fd582d81508
> r15 = 0x7fd582d814f0 rip = 0x010283da
> Found by: call frame info
> 2
> impalad!impala::Statestore::DoSubscriberUpdate(impala::Statestore::UpdateKind,
> int, impala::Statestore::ScheduledSubscriberUpdate const&) [statestore.cc :
> 933 + 0x23]
> rbx = 0x rbp = 0x7fd582d817d0
> rsp = 0x7fd582d81590 r12 = 0x7fd582d81840
> r13 = 0x20c49ba5e353f7cf r14 = 0x01667beb277f
> r15 = 0x7ffc38ca1080 rip = 0x01029064
> Found by: call frame info
> 3
> impalad!impala::ThreadPool::WorkerThread(int)
> [function_template.hpp : 767 + 0x10]
>
[jira] [Resolved] (IMPALA-7714) Statestore::Subscriber::SetLastTopicVersionProcessed() crashed in AtomicInt64::Store()
[
https://issues.apache.org/jira/browse/IMPALA-7714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-7714.
---
Resolution: Fixed
Fix Version/s: Impala 3.1.0
> Statestore::Subscriber::SetLastTopicVersionProcessed() crashed in
> AtomicInt64::Store()
> --
>
> Key: IMPALA-7714
> URL: https://issues.apache.org/jira/browse/IMPALA-7714
> Project: IMPALA
> Issue Type: Bug
> Components: Distributed Exec
>Affects Versions: Impala 3.1.0
>Reporter: Michael Ho
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: broken-build
> Fix For: Impala 3.1.0
>
> Attachments: dbfd9687-09a9-4ab0-dcd7128b-41a2c5b3.dmp.resolved
>
>
> When running one of the customer cluster tests,
> {{Statestore::Subscriber::SetLastTopicVersionProcessed()}} most likely
> crashed at the following line. It could be a race or something but I didn't
> have time to dig more into it.
> {noformat}
> void Statestore::Subscriber::SetLastTopicVersionProcessed(const TopicId&
> topic_id,
> TopicEntry::Version version) {
> // Safe to call concurrently for different topics because
> 'subscribed_topics' is not
> // modified.
> Topics* subscribed_topics = GetTopicsMapForId(topic_id);
> Topics::iterator topic_it = subscribed_topics->find(topic_id);
> DCHECK(topic_it != subscribed_topics->end());
> topic_it->second.last_version.Store(version); <<-
> }
> {noformat}
> {noformat}
> Error Message
> Minidump generated:
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/logs/custom_cluster_tests/minidumps/statestored/336d9ca9-88dc-4360-6a5adf97-936db5c0.dmp
> Standard Error
> Operating system: Linux
> 0.0.0 Linux 3.10.0-693.5.2.el7.x86_64 #1 SMP Fri Oct 20
> 20:32:50 UTC 2017 x86_64
> CPU: amd64
> family 6 model 85 stepping 4
> 1 CPU
> GPU: UNKNOWN
> Crash reason: SIGSEGV
> Crash address: 0x28
> Process uptime: not available
> Thread 18 (crashed)
> 0
> impalad!impala::Statestore::Subscriber::SetLastTopicVersionProcessed(std::string
> const&, long) [atomicops-internals-x86.h : 300 + 0x0]
> rax = 0x rdx = 0xc34174ed
> rcx = 0x0022c65a25a97b5b rbx = 0x04624e38
> rsi = 0x0070 rdi = 0x04906a79
> rbp = 0x7fd582d81320 rsp = 0x7fd582d812e0
> r8 = 0x9e3779b9r9 = 0x
> r10 = 0x r11 = 0x7fd58da31a90
> r12 = 0x83bfbe948682e9da r13 = 0x04593e20
> r14 = 0x000f r15 = 0x000a
> rip = 0x01022a65
> Found by: given as instruction pointer in context
> 1
> impalad!impala::Statestore::SendTopicUpdate(impala::Statestore::Subscriber*,
> impala::Statestore::UpdateKind, bool*) [statestore.cc : 704 + 0x12]
> rbx = 0x7fd582d813d0 rbp = 0x7fd582d81580
> rsp = 0x7fd582d81330 r12 = 0x04593e00
> r13 = 0x04624dd0 r14 = 0x7fd582d81508
> r15 = 0x7fd582d814f0 rip = 0x010283da
> Found by: call frame info
> 2
> impalad!impala::Statestore::DoSubscriberUpdate(impala::Statestore::UpdateKind,
> int, impala::Statestore::ScheduledSubscriberUpdate const&) [statestore.cc :
> 933 + 0x23]
> rbx = 0x rbp = 0x7fd582d817d0
> rsp = 0x7fd582d81590 r12 = 0x7fd582d81840
> r13 = 0x20c49ba5e353f7cf r14 = 0x01667beb277f
> r15 = 0x7ffc38ca1080 rip = 0x01029064
> Found by: call frame info
> 3
> impalad!impala::ThreadPool::WorkerThread(int)
> [function_template.hpp : 767 + 0x10]
> rbx = 0x7ffc38ca1500 rbp = 0x7fd582d818a0
> rsp = 0x7fd582d817e0 r12 = 0x7ffc38ca1720
> r13 = 0x7fd582d81830 r14 = 0x7fd582d81840
> r15 = 0x rip = 0x01030bdc
> Found by: call frame info
> 4 impalad!impala::Thread::SuperviseThread(std::string const&, std::string
> const&, boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*) [function_template.hpp : 767
> + 0x7]
> rbx = 0x7fd582d81980 rbp = 0x7fd582d81bf0
> rsp = 0x7fd582d818b0 r12 = 0x
> r13 = 0x04658300 r14 = 0x7fd58e6af6a0
> r15 = 0x7ffc38ca07a0 rip = 0x010fec72
> Found by: call frame info
> 5 impalad!boost::detail::thread_data (*)(std::string const&, std::string const&, boost::function,
> impala::ThreadDebugInfo const*, impala::Promise (impala::PromiseMode)0>*), boost::_bi::list5,
> boost::_bi::value, boost::_bi::value >,
> boost::_bi::value,
> boost::_bi::value*> > >
> >::run() [bind.hpp : 525 + 0x6]
> rbx = 0x045f0600 rbp = 0x7fd582d81c50
> rsp =
[jira] [Resolved] (IMPALA-6762) DataStreamRecvr::SenderQueue::GetBatch encounters an exception doing a data_arrival_cv_.Wait(l)
[ https://issues.apache.org/jira/browse/IMPALA-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong resolved IMPALA-6762. --- Resolution: Cannot Reproduce I took another look and agree that it doesn't make sense - there's no way it should be referencing invalid memory here. So it's probably a heap use-after-free, which we can't really track down without a repro (in all likelihood it's been fixed) > DataStreamRecvr::SenderQueue::GetBatch encounters an exception doing a > data_arrival_cv_.Wait(l) > > > Key: IMPALA-6762 > URL: https://issues.apache.org/jira/browse/IMPALA-6762 > Project: IMPALA > Issue Type: Bug >Affects Versions: Impala 2.6.0, Impala 2.13.0 >Reporter: Pranay Singh >Assignee: Pranay Singh >Priority: Major > Labels: crash > > Problem: In the function impala::DataStreamRecvr::SenderQueue::GetBatch() > while > calling data_arrival_cv_.Wait() an exception is encountered in boost > library, which > results in a SIGABRT. The probable cause of this issue is that lock > has been freed. > Note : This problem has been investigated for legacy thrift setup not in a > new KuduRPC setup > Evidence: We have a minidump for the issue seen; the two suspected threads > involved in the issue are listed below. > Thread encountered SIGABRT > Crash reason: SIGABRT > Crash address: 0x3d38b2f > Process uptime: not available > Thread 959 (crashed) > 0 libc-2.17.so + 0x351f7 > rax = 0x rdx = 0x0006 > rcx = 0x rbx = 0x7f1291116f18 > rsi = 0x0001a041 rdi = 0x8b2f > rbp = 0x02ad97c0 rsp = 0x7f102ac0cd48 > r8 = 0x000ar9 = 0x7f102ac0e700 > r10 = 0x0008 r11 = 0x0202 > r12 = 0x7f1291116f00 r13 = 0x7f102ac0cfb0 > r14 = 0x r15 = 0x > rip = 0x7f13ec6601f7 > Found by: given as instruction pointer in context > 1 libc-2.17.so + 0x368e8 > rsp = 0x7f102ac0cd50 rip = 0x7f13ec6618e8 > Found by: stack scanning > . > . > . > 9 impalad! > rax = 0x0001 rdx = 0x0001 > rbx = 0x7f102ac0d390 rbp = 0x7f12c68c13a0 > rsp = 0x7f102ac0d390 r12 = 0x7f12cc820cc0 > r13 = 0x7f1244ab5600 r14 = 0x7f102ac0d4e0 > r15 = 0x0001 rip = 0x0080fe65 > Found by: call frame info > 10 impalad! > rbx = 0x7f102ac0d4e0 rbp = 0x7f1244ab5630 > rsp = 0x7f102ac0d3e0 r12 = 0x7f12cc820cc0 > r13 = 0x7f1244ab5600 r14 = 0x7f102ac0d4e0 > r15 = 0x0001 rip = 0x0080fe8c > Found by: call frame info > 11 impalad! > rbx = 0x rbp = 0x7f1244ab5630 > rsp = 0x7f102ac0d430 r12 = 0x7f12cc820cc0 > r13 = 0x7f1244ab5600 r14 = 0x7f102ac0d4e0 > r15 = 0x0001 rip = 0x00810294 > Found by: call frame info > 12 impalad!impala::DataStreamRecvr::(impala::RowBatch**) > rbx = 0x7f12cc820c60 rbp = 0x7f102ac0d500 > rsp = 0x7f102ac0d4c0 r12 = 0x7f102ac0d530 > r13 = 0x7f12cc820c90 r14 = 0x7f127242f338 > r15 = 0x7f12cc820d48 rip = 0x00a280f3 > Found by: call frame info > 13 impalad!impala::DataStreamRecvr::GetBatch(impala::RowBatch**) > rbx = 0x7f102ac0d5c0 rbp = 0x7f102ac0d5c0 > rsp = 0x7f102ac0d5a0 r12 = 0x7f121f464100 > r13 = 0x7f127242f180 r14 = 0x7f121f464100 > r15 = 0x7f102ac0d760 rip = 0x00a284c3 > Found by: call frame info > 14 impalad!impala::ExchangeNode::FillInputRowBatch(impala::RuntimeState*) > rbx = 0x7f102ac0d690 rbp = 0x7f102ac0d5c0 > rsp = 0x7f102ac0d5b0 r12 = 0x7f121f464100 > r13 = 0x7f127242f180 r14 = 0x7f121f464100 > r15 = 0x7f102ac0d760 rip = 0x00beffa5 > Found by: call frame info > 15 impalad!impala::ExchangeNode::Open(impala::RuntimeState*) > rbx = 0x7f121f464100 rbp = 0x7f102ac0d8d0 > rsp = 0x7f102ac0d640 r12 = 0x7f127242f180 > r13 = 0x7f102ac0d690 r14 = 0x7f121f464100 > r15 = 0x7f102ac0d760 rip = 0x00bf0d9e > Found by: call frame info > Thread 336 > > 13 impalad! [TBufferTransports.h : 69 + 0xe] > rbx = 0x rbp = 0x0004 > rsp = 0x7f13077b9840 r12 = 0x0004 > r13 = 0x7f13077b98b0 r14 = 0x7f12c3f6f270 > r15 = 0x7f12d5a7c034 rip = 0x0080be6e > Found by: call frame info > 14 >
[jira] [Resolved] (IMPALA-6762) DataStreamRecvr::SenderQueue::GetBatch encounters an exception doing a data_arrival_cv_.Wait(l)
[ https://issues.apache.org/jira/browse/IMPALA-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong resolved IMPALA-6762. --- Resolution: Cannot Reproduce I took another look and agree that it doesn't make sense - there's no way it should be referencing invalid memory here. So it's probably a heap use-after-free, which we can't really track down without a repro (in all likelihood it's been fixed) > DataStreamRecvr::SenderQueue::GetBatch encounters an exception doing a > data_arrival_cv_.Wait(l) > > > Key: IMPALA-6762 > URL: https://issues.apache.org/jira/browse/IMPALA-6762 > Project: IMPALA > Issue Type: Bug >Affects Versions: Impala 2.6.0, Impala 2.13.0 >Reporter: Pranay Singh >Assignee: Pranay Singh >Priority: Major > Labels: crash > > Problem: In the function impala::DataStreamRecvr::SenderQueue::GetBatch() > while > calling data_arrival_cv_.Wait() an exception is encountered in boost > library, which > results in a SIGABRT. The probable cause of this issue is that lock > has been freed. > Note : This problem has been investigated for legacy thrift setup not in a > new KuduRPC setup > Evidence: We have a minidump for the issue seen; the two suspected threads > involved in the issue are listed below. > Thread encountered SIGABRT > Crash reason: SIGABRT > Crash address: 0x3d38b2f > Process uptime: not available > Thread 959 (crashed) > 0 libc-2.17.so + 0x351f7 > rax = 0x rdx = 0x0006 > rcx = 0x rbx = 0x7f1291116f18 > rsi = 0x0001a041 rdi = 0x8b2f > rbp = 0x02ad97c0 rsp = 0x7f102ac0cd48 > r8 = 0x000ar9 = 0x7f102ac0e700 > r10 = 0x0008 r11 = 0x0202 > r12 = 0x7f1291116f00 r13 = 0x7f102ac0cfb0 > r14 = 0x r15 = 0x > rip = 0x7f13ec6601f7 > Found by: given as instruction pointer in context > 1 libc-2.17.so + 0x368e8 > rsp = 0x7f102ac0cd50 rip = 0x7f13ec6618e8 > Found by: stack scanning > . > . > . > 9 impalad! > rax = 0x0001 rdx = 0x0001 > rbx = 0x7f102ac0d390 rbp = 0x7f12c68c13a0 > rsp = 0x7f102ac0d390 r12 = 0x7f12cc820cc0 > r13 = 0x7f1244ab5600 r14 = 0x7f102ac0d4e0 > r15 = 0x0001 rip = 0x0080fe65 > Found by: call frame info > 10 impalad! > rbx = 0x7f102ac0d4e0 rbp = 0x7f1244ab5630 > rsp = 0x7f102ac0d3e0 r12 = 0x7f12cc820cc0 > r13 = 0x7f1244ab5600 r14 = 0x7f102ac0d4e0 > r15 = 0x0001 rip = 0x0080fe8c > Found by: call frame info > 11 impalad! > rbx = 0x rbp = 0x7f1244ab5630 > rsp = 0x7f102ac0d430 r12 = 0x7f12cc820cc0 > r13 = 0x7f1244ab5600 r14 = 0x7f102ac0d4e0 > r15 = 0x0001 rip = 0x00810294 > Found by: call frame info > 12 impalad!impala::DataStreamRecvr::(impala::RowBatch**) > rbx = 0x7f12cc820c60 rbp = 0x7f102ac0d500 > rsp = 0x7f102ac0d4c0 r12 = 0x7f102ac0d530 > r13 = 0x7f12cc820c90 r14 = 0x7f127242f338 > r15 = 0x7f12cc820d48 rip = 0x00a280f3 > Found by: call frame info > 13 impalad!impala::DataStreamRecvr::GetBatch(impala::RowBatch**) > rbx = 0x7f102ac0d5c0 rbp = 0x7f102ac0d5c0 > rsp = 0x7f102ac0d5a0 r12 = 0x7f121f464100 > r13 = 0x7f127242f180 r14 = 0x7f121f464100 > r15 = 0x7f102ac0d760 rip = 0x00a284c3 > Found by: call frame info > 14 impalad!impala::ExchangeNode::FillInputRowBatch(impala::RuntimeState*) > rbx = 0x7f102ac0d690 rbp = 0x7f102ac0d5c0 > rsp = 0x7f102ac0d5b0 r12 = 0x7f121f464100 > r13 = 0x7f127242f180 r14 = 0x7f121f464100 > r15 = 0x7f102ac0d760 rip = 0x00beffa5 > Found by: call frame info > 15 impalad!impala::ExchangeNode::Open(impala::RuntimeState*) > rbx = 0x7f121f464100 rbp = 0x7f102ac0d8d0 > rsp = 0x7f102ac0d640 r12 = 0x7f127242f180 > r13 = 0x7f102ac0d690 r14 = 0x7f121f464100 > r15 = 0x7f102ac0d760 rip = 0x00bf0d9e > Found by: call frame info > Thread 336 > > 13 impalad! [TBufferTransports.h : 69 + 0xe] > rbx = 0x rbp = 0x0004 > rsp = 0x7f13077b9840 r12 = 0x0004 > r13 = 0x7f13077b98b0 r14 = 0x7f12c3f6f270 > r15 = 0x7f12d5a7c034 rip = 0x0080be6e > Found by: call frame info > 14 >
[jira] [Updated] (IMPALA-6762) DataStreamRecvr::SenderQueue::GetBatch encounters an exception doing a data_arrival_cv_.Wait(l)
[ https://issues.apache.org/jira/browse/IMPALA-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong updated IMPALA-6762: -- Labels: crash (was: ) > DataStreamRecvr::SenderQueue::GetBatch encounters an exception doing a > data_arrival_cv_.Wait(l) > > > Key: IMPALA-6762 > URL: https://issues.apache.org/jira/browse/IMPALA-6762 > Project: IMPALA > Issue Type: Bug >Affects Versions: Impala 2.6.0, Impala 2.13.0 >Reporter: Pranay Singh >Assignee: Pranay Singh >Priority: Major > Labels: crash > > Problem: In the function impala::DataStreamRecvr::SenderQueue::GetBatch() > while > calling data_arrival_cv_.Wait() an exception is encountered in boost > library, which > results in a SIGABRT. The probable cause of this issue is that lock > has been freed. > Note : This problem has been investigated for legacy thrift setup not in a > new KuduRPC setup > Evidence: We have a minidump for the issue seen; the two suspected threads > involved in the issue are listed below. > Thread encountered SIGABRT > Crash reason: SIGABRT > Crash address: 0x3d38b2f > Process uptime: not available > Thread 959 (crashed) > 0 libc-2.17.so + 0x351f7 > rax = 0x rdx = 0x0006 > rcx = 0x rbx = 0x7f1291116f18 > rsi = 0x0001a041 rdi = 0x8b2f > rbp = 0x02ad97c0 rsp = 0x7f102ac0cd48 > r8 = 0x000ar9 = 0x7f102ac0e700 > r10 = 0x0008 r11 = 0x0202 > r12 = 0x7f1291116f00 r13 = 0x7f102ac0cfb0 > r14 = 0x r15 = 0x > rip = 0x7f13ec6601f7 > Found by: given as instruction pointer in context > 1 libc-2.17.so + 0x368e8 > rsp = 0x7f102ac0cd50 rip = 0x7f13ec6618e8 > Found by: stack scanning > . > . > . > 9 impalad! > rax = 0x0001 rdx = 0x0001 > rbx = 0x7f102ac0d390 rbp = 0x7f12c68c13a0 > rsp = 0x7f102ac0d390 r12 = 0x7f12cc820cc0 > r13 = 0x7f1244ab5600 r14 = 0x7f102ac0d4e0 > r15 = 0x0001 rip = 0x0080fe65 > Found by: call frame info > 10 impalad! > rbx = 0x7f102ac0d4e0 rbp = 0x7f1244ab5630 > rsp = 0x7f102ac0d3e0 r12 = 0x7f12cc820cc0 > r13 = 0x7f1244ab5600 r14 = 0x7f102ac0d4e0 > r15 = 0x0001 rip = 0x0080fe8c > Found by: call frame info > 11 impalad! > rbx = 0x rbp = 0x7f1244ab5630 > rsp = 0x7f102ac0d430 r12 = 0x7f12cc820cc0 > r13 = 0x7f1244ab5600 r14 = 0x7f102ac0d4e0 > r15 = 0x0001 rip = 0x00810294 > Found by: call frame info > 12 impalad!impala::DataStreamRecvr::(impala::RowBatch**) > rbx = 0x7f12cc820c60 rbp = 0x7f102ac0d500 > rsp = 0x7f102ac0d4c0 r12 = 0x7f102ac0d530 > r13 = 0x7f12cc820c90 r14 = 0x7f127242f338 > r15 = 0x7f12cc820d48 rip = 0x00a280f3 > Found by: call frame info > 13 impalad!impala::DataStreamRecvr::GetBatch(impala::RowBatch**) > rbx = 0x7f102ac0d5c0 rbp = 0x7f102ac0d5c0 > rsp = 0x7f102ac0d5a0 r12 = 0x7f121f464100 > r13 = 0x7f127242f180 r14 = 0x7f121f464100 > r15 = 0x7f102ac0d760 rip = 0x00a284c3 > Found by: call frame info > 14 impalad!impala::ExchangeNode::FillInputRowBatch(impala::RuntimeState*) > rbx = 0x7f102ac0d690 rbp = 0x7f102ac0d5c0 > rsp = 0x7f102ac0d5b0 r12 = 0x7f121f464100 > r13 = 0x7f127242f180 r14 = 0x7f121f464100 > r15 = 0x7f102ac0d760 rip = 0x00beffa5 > Found by: call frame info > 15 impalad!impala::ExchangeNode::Open(impala::RuntimeState*) > rbx = 0x7f121f464100 rbp = 0x7f102ac0d8d0 > rsp = 0x7f102ac0d640 r12 = 0x7f127242f180 > r13 = 0x7f102ac0d690 r14 = 0x7f121f464100 > r15 = 0x7f102ac0d760 rip = 0x00bf0d9e > Found by: call frame info > Thread 336 > > 13 impalad! [TBufferTransports.h : 69 + 0xe] > rbx = 0x rbp = 0x0004 > rsp = 0x7f13077b9840 r12 = 0x0004 > r13 = 0x7f13077b98b0 r14 = 0x7f12c3f6f270 > r15 = 0x7f12d5a7c034 rip = 0x0080be6e > Found by: call frame info > 14 > impalad!apache::thrift::protocol::TBinaryProtocolT::readMessageBegin(std::string&, > apache::thrift::protocol::TMessageType&, int&) > rbx = 0x7f13077b98b0 rbp = 0x7f13077b98f8 > rsp = 0x7f13077b98a0 r12 = 0x7f13077b98fc > r13 = 0x7f13077b9900
[jira] [Updated] (IMPALA-6762) DataStreamRecvr::SenderQueue::GetBatch encounters an exception doing a data_arrival_cv_.Wait(l)
[ https://issues.apache.org/jira/browse/IMPALA-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong updated IMPALA-6762: -- Affects Version/s: (was: Impala 2.12.0) (was: Impala 2.11.0) (was: Impala 2.10.0) (was: Impala 2.9.0) (was: Impala 2.7.0) > DataStreamRecvr::SenderQueue::GetBatch encounters an exception doing a > data_arrival_cv_.Wait(l) > > > Key: IMPALA-6762 > URL: https://issues.apache.org/jira/browse/IMPALA-6762 > Project: IMPALA > Issue Type: Bug >Affects Versions: Impala 2.6.0, Impala 2.13.0 >Reporter: Pranay Singh >Assignee: Pranay Singh >Priority: Major > Labels: crash > > Problem: In the function impala::DataStreamRecvr::SenderQueue::GetBatch() > while > calling data_arrival_cv_.Wait() an exception is encountered in boost > library, which > results in a SIGABRT. The probable cause of this issue is that lock > has been freed. > Note : This problem has been investigated for legacy thrift setup not in a > new KuduRPC setup > Evidence: We have a minidump for the issue seen; the two suspected threads > involved in the issue are listed below. > Thread encountered SIGABRT > Crash reason: SIGABRT > Crash address: 0x3d38b2f > Process uptime: not available > Thread 959 (crashed) > 0 libc-2.17.so + 0x351f7 > rax = 0x rdx = 0x0006 > rcx = 0x rbx = 0x7f1291116f18 > rsi = 0x0001a041 rdi = 0x8b2f > rbp = 0x02ad97c0 rsp = 0x7f102ac0cd48 > r8 = 0x000ar9 = 0x7f102ac0e700 > r10 = 0x0008 r11 = 0x0202 > r12 = 0x7f1291116f00 r13 = 0x7f102ac0cfb0 > r14 = 0x r15 = 0x > rip = 0x7f13ec6601f7 > Found by: given as instruction pointer in context > 1 libc-2.17.so + 0x368e8 > rsp = 0x7f102ac0cd50 rip = 0x7f13ec6618e8 > Found by: stack scanning > . > . > . > 9 impalad! > rax = 0x0001 rdx = 0x0001 > rbx = 0x7f102ac0d390 rbp = 0x7f12c68c13a0 > rsp = 0x7f102ac0d390 r12 = 0x7f12cc820cc0 > r13 = 0x7f1244ab5600 r14 = 0x7f102ac0d4e0 > r15 = 0x0001 rip = 0x0080fe65 > Found by: call frame info > 10 impalad! > rbx = 0x7f102ac0d4e0 rbp = 0x7f1244ab5630 > rsp = 0x7f102ac0d3e0 r12 = 0x7f12cc820cc0 > r13 = 0x7f1244ab5600 r14 = 0x7f102ac0d4e0 > r15 = 0x0001 rip = 0x0080fe8c > Found by: call frame info > 11 impalad! > rbx = 0x rbp = 0x7f1244ab5630 > rsp = 0x7f102ac0d430 r12 = 0x7f12cc820cc0 > r13 = 0x7f1244ab5600 r14 = 0x7f102ac0d4e0 > r15 = 0x0001 rip = 0x00810294 > Found by: call frame info > 12 impalad!impala::DataStreamRecvr::(impala::RowBatch**) > rbx = 0x7f12cc820c60 rbp = 0x7f102ac0d500 > rsp = 0x7f102ac0d4c0 r12 = 0x7f102ac0d530 > r13 = 0x7f12cc820c90 r14 = 0x7f127242f338 > r15 = 0x7f12cc820d48 rip = 0x00a280f3 > Found by: call frame info > 13 impalad!impala::DataStreamRecvr::GetBatch(impala::RowBatch**) > rbx = 0x7f102ac0d5c0 rbp = 0x7f102ac0d5c0 > rsp = 0x7f102ac0d5a0 r12 = 0x7f121f464100 > r13 = 0x7f127242f180 r14 = 0x7f121f464100 > r15 = 0x7f102ac0d760 rip = 0x00a284c3 > Found by: call frame info > 14 impalad!impala::ExchangeNode::FillInputRowBatch(impala::RuntimeState*) > rbx = 0x7f102ac0d690 rbp = 0x7f102ac0d5c0 > rsp = 0x7f102ac0d5b0 r12 = 0x7f121f464100 > r13 = 0x7f127242f180 r14 = 0x7f121f464100 > r15 = 0x7f102ac0d760 rip = 0x00beffa5 > Found by: call frame info > 15 impalad!impala::ExchangeNode::Open(impala::RuntimeState*) > rbx = 0x7f121f464100 rbp = 0x7f102ac0d8d0 > rsp = 0x7f102ac0d640 r12 = 0x7f127242f180 > r13 = 0x7f102ac0d690 r14 = 0x7f121f464100 > r15 = 0x7f102ac0d760 rip = 0x00bf0d9e > Found by: call frame info > Thread 336 > > 13 impalad! [TBufferTransports.h : 69 + 0xe] > rbx = 0x rbp = 0x0004 > rsp = 0x7f13077b9840 r12 = 0x0004 > r13 = 0x7f13077b98b0 r14 = 0x7f12c3f6f270 > r15 = 0x7f12d5a7c034 rip = 0x0080be6e > Found by: call frame info > 14 >
[jira] [Resolved] (IMPALA-5680) Avro scanner still crashing when HDFS seek fails
[
https://issues.apache.org/jira/browse/IMPALA-5680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-5680.
---
Resolution: Cannot Reproduce
> Avro scanner still crashing when HDFS seek fails
>
>
> Key: IMPALA-5680
> URL: https://issues.apache.org/jira/browse/IMPALA-5680
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.8.0
>Reporter: Pavas Garg
>Priority: Major
>
> We had a previous Impala Jira
> https://issues.apache.org/jira/browse/IMPALA-4223
> Impala daemon didn't crash but stopped working.
> I0712 16:21:25.917135 11642 status.cc:114] Error seeking to 536870912 in
> file:
> hdfs://nameservice1/envs/production/XXX/data/XXX_ETL-f2h/xxx_logs/dt=2017-07-12/00_0
>
> Error(255): Unknown error 255
> @ 0x8377f9 (unknown)
> @ 0xa606fb (unknown)
> @ 0xa5bbeb (unknown)
> @ 0xa5c064 (unknown)
> @ 0xbd2cd9 (unknown)
> @ 0xbd36b4 (unknown)
> @ 0xe1b65a (unknown)
> @ 0x3a3b007851 (unknown)
> @ 0x3a3a8e894d (unknown)
> I0712 16:21:25.917220 30331 runtime-state.cc:217] Error from query
> 37402db5e53ce683:96f60bf4: Problem parsing file
> hdfs://nameservice1/envs/production/XXX/out-data/XXX_ETL-f2h/XXX_logs/dt=2017-07-12/00_0
> at 536859282
> I0712 16:21:25.917259 30331 runtime-state.cc:217] Error from query
> 37402db5e53ce683:96f60bf4: Error seeking to 536870912 in file:
> hdfs://nameservice1/envs/production/XXX/out-data/XXX_ETL-f2h/XXX_logs/dt=2017-07-12/00_0
>
> Error(255): Unknown error 255
> I0712 16:21:25.928112 11640 status.cc:114] Error seeking to 536870912 in
> file:
> hdfs://nameservice1/envs/production/XXX/out-data/XXX_ETL-f2h/XXX_logs/dt=2017-07-12/02_0
>
> Error(255): Unknown error 255
> @ 0x8377f9 (unknown)
> @ 0xa606fb (unknown)
> @ 0xa5bbeb (unknown)
> @ 0xa5c064 (unknown)
> @ 0xbd2cd9 (unknown)
> @ 0xbd36b4 (unknown)
> @ 0xe1b65a (unknown)
> @ 0x3a3b007851 (unknown)
> @ 0x3a3a8e894d (unknown)
> I0712 16:21:25.928170 30332 runtime-state.cc:217] Error from query
> 37402db5e53ce683:96f60bf4: Problem parsing file
> hdfs://nameservice1/envs/production/XXX/out-data/XXX_ETL-f2h/XXX_logs/dt=2017-07-12/02_0
> at 536867240
> I0712 16:21:25.928185 30332 runtime-state.cc:217] Error from query
> 37402db5e53ce683:96f60bf4: Error seeking to 536870912 in file:
> hdfs://nameservice1/envs/production/XXX/out-data/XXX_ETL-f2h/XXX_logs/dt=2017-07-12/02_0
>
> Error(255): Unknown error 255
> I0712 17:09:47.028455 11343 authentication.cc:499] Registering
> impala/hostn...@domain.com, keytab file
> /var/run/cloudera-scm-agent/process/XXX-impala-IMPALAD/impala.keytab
> ---
> From HDFS namenode logs:
> ---
> 2017-07-12 16:20:43,552 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> allocateBlock:
> /envs/production/XXX/out-data/XXX_ETL-f2h/z_tmp_XXX_logs/dt=2017-07-12/.hive-staging_hive_2017-07-12_16-11-04_471_7577440609227768101-1/_task_tmp.-ext-1/_tmp.02_0.
> BP-707861034-10.13.68.11-1367011211601
> blk_1319449009_1099897417390{blockUCState=UNDER_CONSTRUCTION,
> primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[[DISK]DS-e5b8a4fb-a840-4de2-bd5c-d0de019cb2c8:NORMAL:10.13.68.32:1004|XXX],
>
> ReplicaUnderConstruction[[DISK]DS-5373c471-3289-4563-bff1-65b01c7c906b:NORMAL:10.13.68.30:1004|XXX],
>
> ReplicaUnderConstruction[[DISK]DS-e5988516-2067-4339-b465-f5180869f73b:NORMAL:10.13.68.28:1004|XXX]]}
>
> 2017-07-12 16:21:21,599 INFO org.apache.hadoop.hdfs.StateChange: DIR*
> completeFile:
> /envs/production/XXX/out-data/XXX_ETL-f2h/z_tmp_XXX_logs/dt=2017-07-12/.hive-staging_hive_2017-07-12_16-11-04_471_7577440609227768101-1/_task_tmp.-ext-1/_tmp.02_0
> is closed by DFSClient_attempt_1499250888632_48550_r_02_0_-1168871812_1
> 2017-07-12 16:21:21,729 INFO org.apache.hadoop.hdfs.StateChange: DIR*
> completeFile:
> /envs/production/XXX/out-data/XXX_ETL-f2h/z_tmp_XXX_logs/dt=2017-07-12/.hive-staging_hive_2017-07-12_16-11-04_471_7577440609227768101-1/-ext-10001/tmpstats-2
> is closed by DFSClient_attempt_1499250888632_48550_r_02_0_261413855_1
> 2017-07-12 16:21:24,269 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> allocateBlock:
> /tmp/hive/XXX_prod/f890d0a0-f1bc-4fcd-a828-f882a5aa81e7/hive_2017-07-12_16-14-29_668_7405085067336660683-1/_task_tmp.-mr-10006/_tmp.02_0.
> BP-707861034-10.13.68.11-1367011211601
> blk_1319449246_1099897417627{blockUCState=UNDER_CONSTRUCTION,
> primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[[DISK]DS-15ccd60f-6356-43b0-ae31-11989433f66f:NORMAL:10.13.68.46:1004|XXX],
>
> ReplicaUnderConstruction[[DISK]DS-d40c69bb-7882-42de-a0a7-4a784f4aa14a:NORMAL:10.13.68.73:1004|XXX],
>
>
[jira] [Resolved] (IMPALA-5680) Avro scanner still crashing when HDFS seek fails
[
https://issues.apache.org/jira/browse/IMPALA-5680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-5680.
---
Resolution: Cannot Reproduce
> Avro scanner still crashing when HDFS seek fails
>
>
> Key: IMPALA-5680
> URL: https://issues.apache.org/jira/browse/IMPALA-5680
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.8.0
>Reporter: Pavas Garg
>Priority: Major
>
> We had a previous Impala Jira
> https://issues.apache.org/jira/browse/IMPALA-4223
> Impala daemon didn't crash but stopped working.
> I0712 16:21:25.917135 11642 status.cc:114] Error seeking to 536870912 in
> file:
> hdfs://nameservice1/envs/production/XXX/data/XXX_ETL-f2h/xxx_logs/dt=2017-07-12/00_0
>
> Error(255): Unknown error 255
> @ 0x8377f9 (unknown)
> @ 0xa606fb (unknown)
> @ 0xa5bbeb (unknown)
> @ 0xa5c064 (unknown)
> @ 0xbd2cd9 (unknown)
> @ 0xbd36b4 (unknown)
> @ 0xe1b65a (unknown)
> @ 0x3a3b007851 (unknown)
> @ 0x3a3a8e894d (unknown)
> I0712 16:21:25.917220 30331 runtime-state.cc:217] Error from query
> 37402db5e53ce683:96f60bf4: Problem parsing file
> hdfs://nameservice1/envs/production/XXX/out-data/XXX_ETL-f2h/XXX_logs/dt=2017-07-12/00_0
> at 536859282
> I0712 16:21:25.917259 30331 runtime-state.cc:217] Error from query
> 37402db5e53ce683:96f60bf4: Error seeking to 536870912 in file:
> hdfs://nameservice1/envs/production/XXX/out-data/XXX_ETL-f2h/XXX_logs/dt=2017-07-12/00_0
>
> Error(255): Unknown error 255
> I0712 16:21:25.928112 11640 status.cc:114] Error seeking to 536870912 in
> file:
> hdfs://nameservice1/envs/production/XXX/out-data/XXX_ETL-f2h/XXX_logs/dt=2017-07-12/02_0
>
> Error(255): Unknown error 255
> @ 0x8377f9 (unknown)
> @ 0xa606fb (unknown)
> @ 0xa5bbeb (unknown)
> @ 0xa5c064 (unknown)
> @ 0xbd2cd9 (unknown)
> @ 0xbd36b4 (unknown)
> @ 0xe1b65a (unknown)
> @ 0x3a3b007851 (unknown)
> @ 0x3a3a8e894d (unknown)
> I0712 16:21:25.928170 30332 runtime-state.cc:217] Error from query
> 37402db5e53ce683:96f60bf4: Problem parsing file
> hdfs://nameservice1/envs/production/XXX/out-data/XXX_ETL-f2h/XXX_logs/dt=2017-07-12/02_0
> at 536867240
> I0712 16:21:25.928185 30332 runtime-state.cc:217] Error from query
> 37402db5e53ce683:96f60bf4: Error seeking to 536870912 in file:
> hdfs://nameservice1/envs/production/XXX/out-data/XXX_ETL-f2h/XXX_logs/dt=2017-07-12/02_0
>
> Error(255): Unknown error 255
> I0712 17:09:47.028455 11343 authentication.cc:499] Registering
> impala/hostn...@domain.com, keytab file
> /var/run/cloudera-scm-agent/process/XXX-impala-IMPALAD/impala.keytab
> ---
> From HDFS namenode logs:
> ---
> 2017-07-12 16:20:43,552 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> allocateBlock:
> /envs/production/XXX/out-data/XXX_ETL-f2h/z_tmp_XXX_logs/dt=2017-07-12/.hive-staging_hive_2017-07-12_16-11-04_471_7577440609227768101-1/_task_tmp.-ext-1/_tmp.02_0.
> BP-707861034-10.13.68.11-1367011211601
> blk_1319449009_1099897417390{blockUCState=UNDER_CONSTRUCTION,
> primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[[DISK]DS-e5b8a4fb-a840-4de2-bd5c-d0de019cb2c8:NORMAL:10.13.68.32:1004|XXX],
>
> ReplicaUnderConstruction[[DISK]DS-5373c471-3289-4563-bff1-65b01c7c906b:NORMAL:10.13.68.30:1004|XXX],
>
> ReplicaUnderConstruction[[DISK]DS-e5988516-2067-4339-b465-f5180869f73b:NORMAL:10.13.68.28:1004|XXX]]}
>
> 2017-07-12 16:21:21,599 INFO org.apache.hadoop.hdfs.StateChange: DIR*
> completeFile:
> /envs/production/XXX/out-data/XXX_ETL-f2h/z_tmp_XXX_logs/dt=2017-07-12/.hive-staging_hive_2017-07-12_16-11-04_471_7577440609227768101-1/_task_tmp.-ext-1/_tmp.02_0
> is closed by DFSClient_attempt_1499250888632_48550_r_02_0_-1168871812_1
> 2017-07-12 16:21:21,729 INFO org.apache.hadoop.hdfs.StateChange: DIR*
> completeFile:
> /envs/production/XXX/out-data/XXX_ETL-f2h/z_tmp_XXX_logs/dt=2017-07-12/.hive-staging_hive_2017-07-12_16-11-04_471_7577440609227768101-1/-ext-10001/tmpstats-2
> is closed by DFSClient_attempt_1499250888632_48550_r_02_0_261413855_1
> 2017-07-12 16:21:24,269 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> allocateBlock:
> /tmp/hive/XXX_prod/f890d0a0-f1bc-4fcd-a828-f882a5aa81e7/hive_2017-07-12_16-14-29_668_7405085067336660683-1/_task_tmp.-mr-10006/_tmp.02_0.
> BP-707861034-10.13.68.11-1367011211601
> blk_1319449246_1099897417627{blockUCState=UNDER_CONSTRUCTION,
> primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[[DISK]DS-15ccd60f-6356-43b0-ae31-11989433f66f:NORMAL:10.13.68.46:1004|XXX],
>
> ReplicaUnderConstruction[[DISK]DS-d40c69bb-7882-42de-a0a7-4a784f4aa14a:NORMAL:10.13.68.73:1004|XXX],
>
>
[jira] [Resolved] (IMPALA-414) Impala server cannot detect crash-restart failures reliably
[ https://issues.apache.org/jira/browse/IMPALA-414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong resolved IMPALA-414. -- Resolution: Duplicate I think this is describing IMPALA-2990 since we only need to detect crash-restart failures reliably in order to kill queries that were executing on the failed node. > Impala server cannot detect crash-restart failures reliably > --- > > Key: IMPALA-414 > URL: https://issues.apache.org/jira/browse/IMPALA-414 > Project: IMPALA > Issue Type: Improvement > Components: Distributed Exec >Affects Versions: Impala 1.0.1 >Reporter: Henry Robinson >Priority: Minor > Labels: statestore > > The membership mechanism used to tell Impala servers about failures does not > always detect fast crash-restarts. If a server restarts and re-registers > before the state-store recognises that it has failed, the failure won't get > reported to any other subscriber. > The right way to fix this, I think, is to track a version number in every > subscriber. When a subscriber reconnects, it gets a new version number. For > every query, we track the highest version number of the subscriber known at > that time. Then if any backend executing a query has a higher version number, > it's likely to have restarted since the query started. There might be a > couple of false positives, since a node could conceivably restart between a > scheduling assignment and actually receiving a query, but that's unlikely and > better than false negatives. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-414) Impala server cannot detect crash-restart failures reliably
[ https://issues.apache.org/jira/browse/IMPALA-414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong resolved IMPALA-414. -- Resolution: Duplicate I think this is describing IMPALA-2990 since we only need to detect crash-restart failures reliably in order to kill queries that were executing on the failed node. > Impala server cannot detect crash-restart failures reliably > --- > > Key: IMPALA-414 > URL: https://issues.apache.org/jira/browse/IMPALA-414 > Project: IMPALA > Issue Type: Improvement > Components: Distributed Exec >Affects Versions: Impala 1.0.1 >Reporter: Henry Robinson >Priority: Minor > Labels: statestore > > The membership mechanism used to tell Impala servers about failures does not > always detect fast crash-restarts. If a server restarts and re-registers > before the state-store recognises that it has failed, the failure won't get > reported to any other subscriber. > The right way to fix this, I think, is to track a version number in every > subscriber. When a subscriber reconnects, it gets a new version number. For > every query, we track the highest version number of the subscriber known at > that time. Then if any backend executing a query has a higher version number, > it's likely to have restarted since the query started. There might be a > couple of false positives, since a node could conceivably restart between a > scheduling assignment and actually receiving a query, but that's unlikely and > better than false negatives. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (IMPALA-3950) Impala daemon crash because of C [libc.so.6+0x14a084] __memcpy_ssse3_back+0x1914 frame
[ https://issues.apache.org/jira/browse/IMPALA-3950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong resolved IMPALA-3950. --- Resolution: Information Provided > Impala daemon crash because of C [libc.so.6+0x14a084] > __memcpy_ssse3_back+0x1914 frame > -- > > Key: IMPALA-3950 > URL: https://issues.apache.org/jira/browse/IMPALA-3950 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 2.5.0 >Reporter: Manikandan R >Priority: Minor > Labels: crash > Attachments: hs_err_pid36104.log, query_crash.txt > > > Hello, > I am using CDH-5.7.0-1.cdh5.7.0.p0.45 and Impala version is Impala Shell > v2.5.0-cdh5.7.0. Daemon is crashing quite often these days. When I checked > couple of cases (in logs) in detail, I am seeing the same traces. Please see > below - > # > A fatal error has been detected by the Java Runtime Environment: > # > SIGSEGV (0xb) at pc=0x7f84cee69084, pid=36104, tid=140129948964608 > # > JRE version: Java(TM) SE Runtime Environment (7.0_67-b01) (build 1.7.0_67-b01) > Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 > compressed oops) > Problematic frame: > C [libc.so.6+0x14a084] __memcpy_ssse3_back+0x1914* > # > Failed to write core dump. Core dumps have been disabled. To enable core > dumping, try "ulimit -c unlimited" before starting Java again > # > An error report file with more information is saved as: > /run/cloudera-scm-agent/process/2660-impala-IMPALAD/hs_err_pid36104.log > # > If you would like to submit a bug report, please visit: > http://bugreport.sun.com/bugreport/crash.jsp > I am attaching heap dump and query ran at that same time for reference. > Is there any workaround to fix this issue or Is this really an issue which > needs code fix? > Please let me know if you need any info. > Thanks, > Man -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (IMPALA-3950) Impala daemon crash because of C [libc.so.6+0x14a084] __memcpy_ssse3_back+0x1914 frame
[ https://issues.apache.org/jira/browse/IMPALA-3950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong resolved IMPALA-3950. --- Resolution: Information Provided > Impala daemon crash because of C [libc.so.6+0x14a084] > __memcpy_ssse3_back+0x1914 frame > -- > > Key: IMPALA-3950 > URL: https://issues.apache.org/jira/browse/IMPALA-3950 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 2.5.0 >Reporter: Manikandan R >Priority: Minor > Labels: crash > Attachments: hs_err_pid36104.log, query_crash.txt > > > Hello, > I am using CDH-5.7.0-1.cdh5.7.0.p0.45 and Impala version is Impala Shell > v2.5.0-cdh5.7.0. Daemon is crashing quite often these days. When I checked > couple of cases (in logs) in detail, I am seeing the same traces. Please see > below - > # > A fatal error has been detected by the Java Runtime Environment: > # > SIGSEGV (0xb) at pc=0x7f84cee69084, pid=36104, tid=140129948964608 > # > JRE version: Java(TM) SE Runtime Environment (7.0_67-b01) (build 1.7.0_67-b01) > Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 > compressed oops) > Problematic frame: > C [libc.so.6+0x14a084] __memcpy_ssse3_back+0x1914* > # > Failed to write core dump. Core dumps have been disabled. To enable core > dumping, try "ulimit -c unlimited" before starting Java again > # > An error report file with more information is saved as: > /run/cloudera-scm-agent/process/2660-impala-IMPALAD/hs_err_pid36104.log > # > If you would like to submit a bug report, please visit: > http://bugreport.sun.com/bugreport/crash.jsp > I am attaching heap dump and query ran at that same time for reference. > Is there any workaround to fix this issue or Is this really an issue which > needs code fix? > Please let me know if you need any info. > Thanks, > Man -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-3950) Impala daemon crash because of C [libc.so.6+0x14a084] __memcpy_ssse3_back+0x1914 frame
[ https://issues.apache.org/jira/browse/IMPALA-3950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong updated IMPALA-3950: -- Labels: crash (was: ) > Impala daemon crash because of C [libc.so.6+0x14a084] > __memcpy_ssse3_back+0x1914 frame > -- > > Key: IMPALA-3950 > URL: https://issues.apache.org/jira/browse/IMPALA-3950 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 2.5.0 >Reporter: Manikandan R >Priority: Minor > Labels: crash > Attachments: hs_err_pid36104.log, query_crash.txt > > > Hello, > I am using CDH-5.7.0-1.cdh5.7.0.p0.45 and Impala version is Impala Shell > v2.5.0-cdh5.7.0. Daemon is crashing quite often these days. When I checked > couple of cases (in logs) in detail, I am seeing the same traces. Please see > below - > # > A fatal error has been detected by the Java Runtime Environment: > # > SIGSEGV (0xb) at pc=0x7f84cee69084, pid=36104, tid=140129948964608 > # > JRE version: Java(TM) SE Runtime Environment (7.0_67-b01) (build 1.7.0_67-b01) > Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 > compressed oops) > Problematic frame: > C [libc.so.6+0x14a084] __memcpy_ssse3_back+0x1914* > # > Failed to write core dump. Core dumps have been disabled. To enable core > dumping, try "ulimit -c unlimited" before starting Java again > # > An error report file with more information is saved as: > /run/cloudera-scm-agent/process/2660-impala-IMPALAD/hs_err_pid36104.log > # > If you would like to submit a bug report, please visit: > http://bugreport.sun.com/bugreport/crash.jsp > I am attaching heap dump and query ran at that same time for reference. > Is there any workaround to fix this issue or Is this really an issue which > needs code fix? > Please let me know if you need any info. > Thanks, > Man -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-1780) Catch exceptions thrown by UDFs
[ https://issues.apache.org/jira/browse/IMPALA-1780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong resolved IMPALA-1780. --- Resolution: Later I think the proper solution here is some kind of sandboxing, rather than trying to fix a few cases where UDFs can break things. > Catch exceptions thrown by UDFs > --- > > Key: IMPALA-1780 > URL: https://issues.apache.org/jira/browse/IMPALA-1780 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 2.1.1, Impala 2.3.0 >Reporter: Henry Robinson >Priority: Major > Labels: crash, downgraded > > Catch exceptions thrown by UDFs so Impala doesn't crash. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (IMPALA-1780) Catch exceptions thrown by UDFs
[ https://issues.apache.org/jira/browse/IMPALA-1780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong resolved IMPALA-1780. --- Resolution: Later I think the proper solution here is some kind of sandboxing, rather than trying to fix a few cases where UDFs can break things. > Catch exceptions thrown by UDFs > --- > > Key: IMPALA-1780 > URL: https://issues.apache.org/jira/browse/IMPALA-1780 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 2.1.1, Impala 2.3.0 >Reporter: Henry Robinson >Priority: Major > Labels: crash, downgraded > > Catch exceptions thrown by UDFs so Impala doesn't crash. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-5921) Internal-UDF crashed
[
https://issues.apache.org/jira/browse/IMPALA-5921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664492#comment-16664492
]
Tim Armstrong commented on IMPALA-5921:
---
[~attilaj][~jeszyb] this got neglected since it wasn't marked as critical. Do
we have any clues about what query might have caused this? I looked at the code
but I couldn't see how it could get into that state and it's also been
refactored extensively in master, e.g. by IMPALA-4192
> Internal-UDF crashed
>
>
> Key: IMPALA-5921
> URL: https://issues.apache.org/jira/browse/IMPALA-5921
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 2.7.0
>Reporter: Attila Jeges
>Priority: Major
> Labels: crash, udf
> Attachments: 150226_resolved.txt, hs_err_pid43750.log
>
>
> Impala crashed and a mindump was written. Here's the resolved stack:
> {code}
> Thread 231 (crashed)
> 0 impalad!impala::ScalarFnCall::EvaluateChildren(impala::ExprContext*,
> impala::TupleRow const*, std::vector std::allocator >*) [anyval-util.h : 215 + 0x0]
> rax = 0x7f15a5316090 rdx = 0x0008
> rcx = 0x000a rbx = 0x01c3dfc0
> rsi = 0x rdi = 0x0010
> rbp = 0x rsp = 0x7f161fad80d0
> r8 = 0x7f161fad809cr9 = 0x7f15dca37680
> r10 = 0x r11 = 0x0001
> r12 = 0x0010 r13 = 0x7f0d9ac4ba20
> r14 = 0x7a4d2aa0 r15 = 0x
> rip = 0x008a3209
> Found by: given as instruction pointer in context
> 1 impalad!impala_udf::StringVal
> impala::ScalarFnCall::InterpretEval(impala::ExprContext*,
> impala::TupleRow const*) [scalar-fn-call.cc : 562 + 0x5]
> rbx = 0x7f0d9ac4ba20 rbp = 0x7f0cd921ee40
> rsp = 0x7f161fad8120 r12 = 0x7a4d29a0
> r13 = 0x7c1f09d0 r14 = 0x7f161fad8230
> r15 = 0x7f161fad8250 rip = 0x008acdc6
> Found by: call frame info
> 2 impalad!impala::ScalarFnCall::GetStringVal(impala::ExprContext*,
> impala::TupleRow const*) [scalar-fn-call.cc : 758 + 0x5]
> rbx = 0x7f0d9ac4ba20 rbp = 0x7f15a5316040
> rsp = 0x7f161fad8140 r12 = 0x7f15b7c77900
> r13 = 0x7c1f09d0 r14 = 0x7f161fad8230
> r15 = 0x7f161fad8250 rip = 0x008a3ca5
> Found by: call frame info
> 3 impalad!impala::ExprContext::GetValue(impala::Expr*, impala::TupleRow
> const*) [expr-context.cc : 277 + 0xc]
> rbx = 0x7f0d9ac4ba20 rbp = 0x7f15a5316040
> rsp = 0x7f161fad8150 r12 = 0x7f15b7c77900
> r13 = 0x7c1f09d0 r14 = 0x7f161fad8230
> r15 = 0x7f161fad8250 rip = 0x0085ef3c
> Found by: call frame info
> 4
> impalad!impala::ImpalaServer::QueryExecState::GetRowValue(impala::TupleRow*,
> std::vector >*, std::vector std::allocator >*) [query-exec-state.cc : 837 + 0x5]
> rbx = 0x005a rbp = 0x7f0d7390a000
> rsp = 0x7f161fad81a0 r12 = 0x7f15b7c77900
> r13 = 0x7c1f09d0 r14 = 0x7f161fad8230
> r15 = 0x7f161fad8250 rip = 0x00b24e13
> Found by: call frame info
> 5 impalad!impala::ImpalaServer::QueryExecState::FetchRowsInternal(int,
> impala::ImpalaServer::QueryResultSet*) [query-exec-state.cc : 766 + 0x5]
> rbx = 0x7f0d7390a000 rbp = 0x7f161fad82e0
> rsp = 0x7f161fad81f0 r12 = 0x7f15b80b5200
> r13 = 0x7f161fad8230 r14 = 0x
> r15 = 0x7f161fad8220 rip = 0x00b25bde
> Found by: call frame info
> 6 impalad!impala::ImpalaServer::QueryExecState::FetchRows(int,
> impala::ImpalaServer::QueryResultSet*) [query-exec-state.cc : 656 + 0x19]
> rbx = 0x7f0d7390a000 rbp = 0x7f161fad82e0
> rsp = 0x7f161fad82e0 r12 = 0x7f161fad83b0
> r13 = 0x7f161fad82f0 r14 = 0x7f15b80b5200
> r15 = 0x7f161fad84c0 rip = 0x00b261c9
> Found by: call frame info
> 7 impalad!impala::ImpalaServer::FetchInternal(impala::TUniqueId const&,
> int, bool, apache::hive::service::cli::thrift::TFetchResultsResp*)
> [impala-hs2-server.cc : 524 + 0x5]
> rbx = 0x7f161fad8628 rbp = 0x7f15b80b5200
> rsp = 0x7f161fad8330 r12 = 0x7f161fad83b0
> r13 = 0x7f161fad8470 r14 = 0x7f161fad83c0
> r15 = 0x7f161fad84c0 rip = 0x00b0c672
> Found by: call frame info
> 8
> impalad!impala::ImpalaServer::FetchResults(apache::hive::service::cli::thrift::TFetchResultsResp&,
> apache::hive::service::cli::thrift::TFetchResultsReq const&)
> [impala-hs2-server.cc : 1072 + 0x22]
> rbx = 0x7f161fad8628 rbp = 0x7f161fad84b0
> rsp = 0x7f161fad8440 r12 = 0x7f161fad84e0
>
[jira] [Updated] (IMPALA-1977) Slow cross join query - requires band join implementation
[
https://issues.apache.org/jira/browse/IMPALA-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-1977:
--
Labels: (was: hang)
> Slow cross join query - requires band join implementation
> -
>
> Key: IMPALA-1977
> URL: https://issues.apache.org/jira/browse/IMPALA-1977
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.1.1, Impala 2.2
> Environment: CentOS 6.6, CDH 5.4.0
>Reporter: Isaac Hodes
>Priority: Minor
> Attachments: complete.txt, failed agg profile.txt, oom count
> profile.txt, profile.txt, running select pileup chr20 query.txt, successful
> hash join profile.txt
>
>
> I'm unable to get this query to return; according to Cloudera Manager, it
> hangs around 12%, in F01, which is just an HDFS scan.
> Query (I'd like to be able to run this without restriction the reference and
> bam to specific contigs, as well):
> {code:sql}
> select ref.contig, ref.position, bam.sequence
> from flatbam bam, reference_genome ref
> where ref.contig = 'chr20'
> and bam.contig__contigname = '20'
> and ref.`position` between bam.`start` and bam.`end`
> ;
> {code}
> Some stats (curiously, the job seemed to get to 98% before hanging without
> table stats; I can't drop table stats to repro:
> https://issues.cloudera.org/browse/IMPALA-1976)
> {code}
> > show table stats flatbam;
> +---++-+--+---+-+---+
> | #Rows | #Files | Size| Bytes Cached | Cache Replication | Format |
> Incremental stats |
> +---++-+--+---+-+---+
> | 856755914 | 627| 60.48GB | NOT CACHED | NOT CACHED| PARQUET |
> false |
> +---++-+--+---+-+---+
> Fetched 1 row(s) in 0.08s
> > show table stats reference_genome;
> +++-+--+---+-+---+
> | #Rows | #Files | Size| Bytes Cached | Cache Replication | Format
> | Incremental stats |
> +++-+--+---+-+---+
> | 3137327831 | 91 | 12.63GB | NOT CACHED | NOT CACHED| PARQUET
> | false |
> +++-+--+---+-+---+
> Fetched 1 row(s) in 0.02s
> > select count(*) from flatbam where contig__contigname = '20';
> +--+
> | count(*) |
> +--+
> | 17378815 |
> +--+
> Fetched 1 row(s) in 2.44s
> > select count(*) from reference_genome where contig = 'chr20';
> +--+
> | count(*) |
> +--+
> | 63025520 |
> +--+
> Fetched 1 row(s) in 3.66s
> {code}
> Explain:
> {code}
>
> Estimated Per-Host Requirements: Memory=1.31GB VCores=2
> F02:PLAN FRAGMENT [UNPARTITIONED]
> 04:EXCHANGE [UNPARTITIONED]
> hosts=91 per-host-mem=unavailable
> tuple-ids=0,1 row-size=175B cardinality=3513626239162
> F00:PLAN FRAGMENT [RANDOM]
> DATASTREAM SINK [FRAGMENT=F02, EXCHANGE=04, UNPARTITIONED]
> 02:CROSS JOIN [BROADCAST]
> | predicates: ref.position >= bam.start, ref.position <= bam.`end`
> | hosts=91 per-host-mem=814.93MB
> | tuple-ids=0,1 row-size=175B cardinality=3513626239162
> |
> |--03:EXCHANGE [BROADCAST]
> | hosts=91 per-host-mem=0B
> | tuple-ids=1 row-size=25B cardinality=34859198
> |
> 00:SCAN HDFS [default.flatbam bam, RANDOM]
> partitions=1/1 files=627 size=60.48GB
> predicates: bam.contig__contigname = '20'
> table stats: 856755914 rows total
> column stats: all
> hosts=91 per-host-mem=352.00MB
> tuple-ids=0 row-size=150B cardinality=10079481
> F01:PLAN FRAGMENT [RANDOM]
> DATASTREAM SINK [FRAGMENT=F00, EXCHANGE=03, BROADCAST]
> 01:SCAN HDFS [default.reference_genome ref, RANDOM]
> partitions=1/1 files=91 size=12.63GB
> predicates: ref.contig = 'chr20'
> table stats: 3137327831 rows total
> column stats: all
> hosts=91 per-host-mem=176.00MB
> tuple-ids=1 row-size=25B cardinality=34859198
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-3706) Planner should set min/max per-node buffer pool reservations to optimize query performance
[ https://issues.apache.org/jira/browse/IMPALA-3706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong resolved IMPALA-3706. --- Resolution: Later Unclear if this is the right direction. Needs more thought. > Planner should set min/max per-node buffer pool reservations to optimize > query performance > -- > > Key: IMPALA-3706 > URL: https://issues.apache.org/jira/browse/IMPALA-3706 > Project: IMPALA > Issue Type: Improvement > Components: Frontend >Affects Versions: Impala 2.7.0 >Reporter: Tim Armstrong >Priority: Minor > Labels: planner, resource-management > > Once we have the appropriate min/max reservation mechanisms in-place > (IMPALA-3200), we should make use of them to optimise for query performance > given a memory budget. > E.g. we should preferentially allocate memory to merge aggregations over > preaggregations to reduce or avoid spilling. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (IMPALA-3706) Planner should set min/max per-node buffer pool reservations to optimize query performance
[ https://issues.apache.org/jira/browse/IMPALA-3706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong resolved IMPALA-3706. --- Resolution: Later Unclear if this is the right direction. Needs more thought. > Planner should set min/max per-node buffer pool reservations to optimize > query performance > -- > > Key: IMPALA-3706 > URL: https://issues.apache.org/jira/browse/IMPALA-3706 > Project: IMPALA > Issue Type: Improvement > Components: Frontend >Affects Versions: Impala 2.7.0 >Reporter: Tim Armstrong >Priority: Minor > Labels: planner, resource-management > > Once we have the appropriate min/max reservation mechanisms in-place > (IMPALA-3200), we should make use of them to optimise for query performance > given a memory budget. > E.g. we should preferentially allocate memory to merge aggregations over > preaggregations to reduce or avoid spilling. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7485) test_spilling_naaj hung on jenkins
[
https://issues.apache.org/jira/browse/IMPALA-7485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-7485.
---
Resolution: Cannot Reproduce
There was a cluster of hangs around this time related to IMPALA-7488. I suspect
it's related to that.
> test_spilling_naaj hung on jenkins
> --
>
> Key: IMPALA-7485
> URL: https://issues.apache.org/jira/browse/IMPALA-7485
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Reporter: Csaba Ringhofer
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: broken-build, flaky, flaky-test
> Attachments: resolved_stacks.zip
>
>
> {code}
> query_test/test_spilling.py::TestSpillingDebugActionDimensions::test_spilling_naaj[exec_option:
> {'debug_action': None, 'default_spillable_buffer_size': '256k'} |
> table_format: parquet/none]
> {code}
> seemed hung (was running for more than 4 hours), see
> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/3055/console
> Core dumps and stack traces of impalad were created and the impalad was
> killed. The tests continued without failures after impalad was restarted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7485) test_spilling_naaj hung on jenkins
[
https://issues.apache.org/jira/browse/IMPALA-7485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-7485.
---
Resolution: Cannot Reproduce
There was a cluster of hangs around this time related to IMPALA-7488. I suspect
it's related to that.
> test_spilling_naaj hung on jenkins
> --
>
> Key: IMPALA-7485
> URL: https://issues.apache.org/jira/browse/IMPALA-7485
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Reporter: Csaba Ringhofer
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: broken-build, flaky, flaky-test
> Attachments: resolved_stacks.zip
>
>
> {code}
> query_test/test_spilling.py::TestSpillingDebugActionDimensions::test_spilling_naaj[exec_option:
> {'debug_action': None, 'default_spillable_buffer_size': '256k'} |
> table_format: parquet/none]
> {code}
> seemed hung (was running for more than 4 hours), see
> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/3055/console
> Core dumps and stack traces of impalad were created and the impalad was
> killed. The tests continued without failures after impalad was restarted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IMPALA-7768) Add test for thrift profile compatibility
Joe McDonnell created IMPALA-7768: - Summary: Add test for thrift profile compatibility Key: IMPALA-7768 URL: https://issues.apache.org/jira/browse/IMPALA-7768 Project: IMPALA Issue Type: Improvement Components: Infrastructure Affects Versions: Impala 3.1.0 Reporter: Joe McDonnell There are a variety of tools that consume the thrift runtime profile. As we make improvements, it is useful to be able to maintain compatibility with these tools and verify that we don't unintentionally break existing tools. We should add a test that accesses the thrift runtime profile and accesses various common pieces of useful information. For example, start time, end time, rows produced, and a variety of others are commonly used by these tools and should remain accessible in the current way until we break compatibility. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7768) Add test for thrift profile compatibility
Joe McDonnell created IMPALA-7768: - Summary: Add test for thrift profile compatibility Key: IMPALA-7768 URL: https://issues.apache.org/jira/browse/IMPALA-7768 Project: IMPALA Issue Type: Improvement Components: Infrastructure Affects Versions: Impala 3.1.0 Reporter: Joe McDonnell There are a variety of tools that consume the thrift runtime profile. As we make improvements, it is useful to be able to maintain compatibility with these tools and verify that we don't unintentionally break existing tools. We should add a test that accesses the thrift runtime profile and accesses various common pieces of useful information. For example, start time, end time, rows produced, and a variety of others are commonly used by these tools and should remain accessible in the current way until we break compatibility. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IMPALA-6354) Consider using Guava LoadingCache to cache metadata objects opposed to a ConcurrentHashMap
[ https://issues.apache.org/jira/browse/IMPALA-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664468#comment-16664468 ] Vuk Ercegovac commented on IMPALA-6354: --- https://issues.apache.org/jira/browse/IMPALA-7448 addresses many of these issues and does so by also keeping track of accesses from each coordinator (catalog only operations don't track when tables are used at the coordinators). [~twang], have a look to see if this can simplify things and take into account coordinator cached accesses. Assigning it your way and lowering the priority. > Consider using Guava LoadingCache to cache metadata objects opposed to a > ConcurrentHashMap > -- > > Key: IMPALA-6354 > URL: https://issues.apache.org/jira/browse/IMPALA-6354 > Project: IMPALA > Issue Type: Improvement > Components: Catalog >Reporter: Mostafa Mokhtar >Assignee: Vuk Ercegovac >Priority: Critical > Labels: catalog, catalog-server, scalability > > Look into replacing the ConcurrentHashMap(s) used in the Catalog service to > cache metadata with > [LoadingCache|https://google.github.io/guava/releases/19.0/api/docs/com/google/common/cache/LoadingCache.html]. > Caching metadata using a LoadingCache will allow: > * Better coherency by adding expireAfter and refreshAfter clause to the cache > * Weak reference eviction to respond to garbage collection > * Putting a cap on maximum number of caches entires to avoid OOMs and > excessive GC > * Assigning different weights for each entry (For more efficient eviction) > https://github.com/google/guava/wiki/CachesExplained > https://google.github.io/guava/releases/19.0/api/docs/com/google/common/cache/CacheBuilder.html > https://google.github.io/guava/releases/19.0/api/docs/com/google/common/cache/LoadingCache.html -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-6354) Consider using Guava LoadingCache to cache metadata objects opposed to a ConcurrentHashMap
[ https://issues.apache.org/jira/browse/IMPALA-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vuk Ercegovac updated IMPALA-6354: -- Priority: Major (was: Critical) > Consider using Guava LoadingCache to cache metadata objects opposed to a > ConcurrentHashMap > -- > > Key: IMPALA-6354 > URL: https://issues.apache.org/jira/browse/IMPALA-6354 > Project: IMPALA > Issue Type: Improvement > Components: Catalog >Reporter: Mostafa Mokhtar >Assignee: Vuk Ercegovac >Priority: Major > Labels: catalog, catalog-server, scalability > > Look into replacing the ConcurrentHashMap(s) used in the Catalog service to > cache metadata with > [LoadingCache|https://google.github.io/guava/releases/19.0/api/docs/com/google/common/cache/LoadingCache.html]. > Caching metadata using a LoadingCache will allow: > * Better coherency by adding expireAfter and refreshAfter clause to the cache > * Weak reference eviction to respond to garbage collection > * Putting a cap on maximum number of caches entires to avoid OOMs and > excessive GC > * Assigning different weights for each entry (For more efficient eviction) > https://github.com/google/guava/wiki/CachesExplained > https://google.github.io/guava/releases/19.0/api/docs/com/google/common/cache/CacheBuilder.html > https://google.github.io/guava/releases/19.0/api/docs/com/google/common/cache/LoadingCache.html -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-6354) Consider using Guava LoadingCache to cache metadata objects opposed to a ConcurrentHashMap
[ https://issues.apache.org/jira/browse/IMPALA-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vuk Ercegovac reassigned IMPALA-6354: - Assignee: Tianyi Wang (was: Vuk Ercegovac) > Consider using Guava LoadingCache to cache metadata objects opposed to a > ConcurrentHashMap > -- > > Key: IMPALA-6354 > URL: https://issues.apache.org/jira/browse/IMPALA-6354 > Project: IMPALA > Issue Type: Improvement > Components: Catalog >Reporter: Mostafa Mokhtar >Assignee: Tianyi Wang >Priority: Major > Labels: catalog, catalog-server, scalability > > Look into replacing the ConcurrentHashMap(s) used in the Catalog service to > cache metadata with > [LoadingCache|https://google.github.io/guava/releases/19.0/api/docs/com/google/common/cache/LoadingCache.html]. > Caching metadata using a LoadingCache will allow: > * Better coherency by adding expireAfter and refreshAfter clause to the cache > * Weak reference eviction to respond to garbage collection > * Putting a cap on maximum number of caches entires to avoid OOMs and > excessive GC > * Assigning different weights for each entry (For more efficient eviction) > https://github.com/google/guava/wiki/CachesExplained > https://google.github.io/guava/releases/19.0/api/docs/com/google/common/cache/CacheBuilder.html > https://google.github.io/guava/releases/19.0/api/docs/com/google/common/cache/LoadingCache.html -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7766) Perform constant folding within an expression
[
https://issues.apache.org/jira/browse/IMPALA-7766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664453#comment-16664453
]
Tim Armstrong commented on IMPALA-7766:
---
I think we should be folding subexpressions - maybe the problem is that id + 1
+ 2 is being parsed as (id + 1) + 2 and there isn't a constant subexpression.
In that case I think the solution is to reassociate somehow so that it's in the
form id + (1 + 2) or similar.
> Perform constant folding within an expression
> -
>
> Key: IMPALA-7766
> URL: https://issues.apache.org/jira/browse/IMPALA-7766
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
>Affects Versions: Impala 3.0
>Reporter: Paul Rogers
>Priority: Minor
>
> Suppose we have the following query:
> {code:sql}
> SELECT id + 1 + 2, 2 + 3 FROM foo;
> {code}
> Then, examine the SELECT after rewrites (using the new {{FullRewriteTest}}.)
> We'd expect content folding. However, constant folding only happens if the
> entire expression is constant, it can't handle sub-expressions. Result:
> {code:sql}
> SELECT id + 1 + 2, 5 FROM foo;
> {code}
> Constant folding within an expression is tricky, and it is not clear what, if
> any, performance gain would be had. Still, it is worth keeping in mind.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7751) Kudu insert statement should push down range partition predicates
[
https://issues.apache.org/jira/browse/IMPALA-7751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664449#comment-16664449
]
Quanlong Huang commented on IMPALA-7751:
[~lv], we've tried this before. Though it can balance the load from Impala
side, it brings extra pressure to Kudu. If we have 50 impalad nodes, using the
"NOSHUFFLE" hint, all of them can insert data into a same kudu partition.
Sometimes we see failures like
{code:java}
Kudu error(s) reported, first error: Timed out: Failed to write batch of 611
ops to tablet 5581db1268e04747befd33a67bf1d0df after 491 attempt(s): Failed to
write to server: (no server available): Write(tablet:
5581db1268e04747befd33a67bf1d0df, num_ops: 611, num_attempts: 491) passed its
deadline: Remote error: Service unavailable: Transaction failed, tablet
5581db1268e04747befd33a67bf1d0df transaction memory consumption (62818976) has
exceeded its limit (67108864) or the limit of an ancestral tracker
{code}
We still want each Kudu partition receive sorted data from a single Impala
insert node. The optimal way is to let Impala partition the data by the
{color:#FF}*pruned*{color} kudu partitions. The predicates in the insert
statement can be used in prunning Kudu partitions. Is it hard to do this?
> Kudu insert statement should push down range partition predicates
> -
>
> Key: IMPALA-7751
> URL: https://issues.apache.org/jira/browse/IMPALA-7751
> Project: IMPALA
> Issue Type: Improvement
>Reporter: Quanlong Huang
>Priority: Major
> Attachments: metrics1.tsv, metrics2.tsv, metrics3.tsv, profile.txt
>
>
> We have a job dumping newly added data in HDFS into Kudu table for good
> performance of point queries. Each day we create a new range partition in
> Kudu for the new data on this day. When we add more and more Kudu range
> partitions, we found performance degradation of this job.
> The root cause is, the insert statement for kudu does not leverage the
> partition predicates for kudu range partition keys, which causes skew on the
> insert nodes.
> How to reveal this:
> Step 1: Launch impala cluster with 3 nodes.
> Step 2: Create an HDFS table with more than 3 underlying files, thus will
> have more than 3 scan ranges
> {code:sql}
> create table default.metrics_tbl (
> source_id string,
> event_timestamp bigint,
> value double
> ) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' STORED AS TEXTFILE;
> {code}
> Upload the three attached tsv files into its directory and refresh this table
> in Impala.
> Step 3: Create a Kudu table with mix partitions containing 3 hash partitions
> and 3 range partitions.
> {code:sql}
> create table default.metrics_kudu_tbl (
> source_id string,
> event_timestamp bigint,
> value double,
> primary key(source_id, event_timestamp)
> ) partition by
> hash (source_id) PARTITIONS 3,
> range (event_timestamp) (
> partition 0 <= values < 1,
> partition 1 <= values < 2,
> partition 2 <= values < 3
> ) stored as kudu;
> {code}
> Step 4: Dump rows in HDFS table into Kudu giving partition predicates.
> {code:sql}
> insert into table metrics_kudu_tbl
> select source_id, event_timestamp, value from metrics_tbl
> where event_timestamp >= 1 and event_timestamp < 2;
> {code}
> Step 5: Looking into the profile, there're three fragment instances
> containing KuduTableSink but only one of them received and generated data.
> {code:java}
> Averaged Fragment F01:
> KuduTableSink:
> - TotalNumRows: 1.00K (1000)
> Fragment F01:
> Instance 6347506799a2966d:6e82f4920004
> KuduTableSink:
>- TotalNumRows: 3.00K (3000)
> Instance 6347506799a2966d:6e82f4920005
> KuduTableSink:
>- TotalNumRows: 0 (0)
> Instance 6347506799a2966d:6e82f4920003
> KuduTableSink:
>- TotalNumRows: 0 (0)
> {code}
> Thus, only one fragment instance of F01 is sorting and ingesting data into
> Impala.
> Generally, if there're N range partitions and all the inserted rows are
> belong to one range (supplied by the partition predicates in WHERE clause),
> only 1/N of the insert fragments are producing data.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7767) Add metrics and profiles for automatic invalidation
Tianyi Wang created IMPALA-7767: --- Summary: Add metrics and profiles for automatic invalidation Key: IMPALA-7767 URL: https://issues.apache.org/jira/browse/IMPALA-7767 Project: IMPALA Issue Type: Bug Affects Versions: Impala 3.2.0 Reporter: Tianyi Wang Assignee: Tianyi Wang The automatic invalidation introduced by IMPALA-7448 lacks supportive metrics and query profiles. They should be added to help users to understand whether it's working and to diagnose possible issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7767) Add metrics and profiles for automatic invalidation
Tianyi Wang created IMPALA-7767: --- Summary: Add metrics and profiles for automatic invalidation Key: IMPALA-7767 URL: https://issues.apache.org/jira/browse/IMPALA-7767 Project: IMPALA Issue Type: Bug Affects Versions: Impala 3.2.0 Reporter: Tianyi Wang Assignee: Tianyi Wang The automatic invalidation introduced by IMPALA-7448 lacks supportive metrics and query profiles. They should be added to help users to understand whether it's working and to diagnose possible issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-7766) Perform constant folding within an expression
Paul Rogers created IMPALA-7766:
---
Summary: Perform constant folding within an expression
Key: IMPALA-7766
URL: https://issues.apache.org/jira/browse/IMPALA-7766
Project: IMPALA
Issue Type: Improvement
Components: Frontend
Affects Versions: Impala 3.0
Reporter: Paul Rogers
Suppose we have the following query:
{code:sql}
SELECT id + 1 + 2, 2 + 3 FROM foo;
{code}
Then, examine the SELECT after rewrites (using the new {{FullRewriteTest}}.)
We'd expect content folding. However, constant folding only happens if the
entire expression is constant, it can't handle sub-expressions. Result:
{code:sql}
SELECT id + 1 + 2, 5 FROM foo;
{code}
Constant folding within an expression is tricky, and it is not clear what, if
any, performance gain would be had. Still, it is worth keeping in mind.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7766) Perform constant folding within an expression
Paul Rogers created IMPALA-7766:
---
Summary: Perform constant folding within an expression
Key: IMPALA-7766
URL: https://issues.apache.org/jira/browse/IMPALA-7766
Project: IMPALA
Issue Type: Improvement
Components: Frontend
Affects Versions: Impala 3.0
Reporter: Paul Rogers
Suppose we have the following query:
{code:sql}
SELECT id + 1 + 2, 2 + 3 FROM foo;
{code}
Then, examine the SELECT after rewrites (using the new {{FullRewriteTest}}.)
We'd expect content folding. However, constant folding only happens if the
entire expression is constant, it can't handle sub-expressions. Result:
{code:sql}
SELECT id + 1 + 2, 5 FROM foo;
{code}
Constant folding within an expression is tricky, and it is not clear what, if
any, performance gain would be had. Still, it is worth keeping in mind.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (IMPALA-7506) Support global INVALIDATE METADATA on fetch-on-demand impalad
[ https://issues.apache.org/jira/browse/IMPALA-7506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adrian Ng reassigned IMPALA-7506: - Assignee: Vuk Ercegovac > Support global INVALIDATE METADATA on fetch-on-demand impalad > - > > Key: IMPALA-7506 > URL: https://issues.apache.org/jira/browse/IMPALA-7506 > Project: IMPALA > Issue Type: Sub-task >Reporter: Todd Lipcon >Assignee: Vuk Ercegovac >Priority: Major > > There is some complexity with how this is implemented in the original code: > it depends on maintaining the minimum version of any object in the impalad's > local cache. We can't determine that in an on-demand impalad, so INVALIDATE > METADATA is not supported currently on "fetch-on-demand". -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7733) TestInsertParquetQueries.test_insert_parquet is flaky in S3 due to rename
[
https://issues.apache.org/jira/browse/IMPALA-7733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664425#comment-16664425
]
Vuk Ercegovac commented on IMPALA-7733:
---
[~tianyiwang], pls have a look. Assigned randomly, so pls re-assign if you
can't get to it.
> TestInsertParquetQueries.test_insert_parquet is flaky in S3 due to rename
> -
>
> Key: IMPALA-7733
> URL: https://issues.apache.org/jira/browse/IMPALA-7733
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.1.0
>Reporter: Vuk Ercegovac
>Assignee: Tianyi Wang
>Priority: Blocker
> Labels: flaky-test
>
> I see two examples in the past two months or so where this test fails due to
> a rename error on S3. The test's stacktrace looks like this:
> {noformat}
> query_test/test_insert_parquet.py:112: in test_insert_parquet
> self.run_test_case('insert_parquet', vector, unique_database,
> multiple_impalad=True)
> common/impala_test_suite.py:408: in run_test_case
> result = self.__execute_query(target_impalad_client, query, user=user)
> common/impala_test_suite.py:625: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:160: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:176: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:350: in __execute_query
> self.wait_for_finished(handle)
> beeswax/impala_beeswax.py:371: in wait_for_finished
> raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> EQuery aborted:Error(s) moving partition files. First error (of 1) was:
> Hdfs op (RENAME
> s3a:///test_insert_parquet_968f37fe.db/orders_insert_table/_impala_insert_staging/4e45cd68bcddd451_3c7156ed/.4e45cd68bcddd451-3c7156ed0002_803672621_dir/4e45cd68bcddd451-3c7156ed0002_448261088_data.0.parq
> TO
> s3a:///test-warehouse/test_insert_parquet_968f37fe.db/orders_insert_table/4e45cd68bcddd451-3c7156ed0002_448261088_data.0.parq)
> failed, error was:
> s3a:///test-warehouse/test_insert_parquet_968f37fe.db/orders_insert_table/_impala_insert_staging/4e45cd68bcddd451_3c7156ed/.4e45cd68bcddd451-3c7156ed0002_803672621_dir/4e45cd68bcddd451-3c7156ed0002_448261088_data.0.parq
> E Error(5): Input/output error{noformat}
> Since we know this happens once in a while, some ideas to deflake it:
> * retry
> * check for this specific issue... if we think its platform flakiness, then
> we should skip it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-7733) TestInsertParquetQueries.test_insert_parquet is flaky in S3 due to rename
[
https://issues.apache.org/jira/browse/IMPALA-7733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vuk Ercegovac reassigned IMPALA-7733:
-
Assignee: Tianyi Wang
> TestInsertParquetQueries.test_insert_parquet is flaky in S3 due to rename
> -
>
> Key: IMPALA-7733
> URL: https://issues.apache.org/jira/browse/IMPALA-7733
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.1.0
>Reporter: Vuk Ercegovac
>Assignee: Tianyi Wang
>Priority: Blocker
> Labels: flaky-test
>
> I see two examples in the past two months or so where this test fails due to
> a rename error on S3. The test's stacktrace looks like this:
> {noformat}
> query_test/test_insert_parquet.py:112: in test_insert_parquet
> self.run_test_case('insert_parquet', vector, unique_database,
> multiple_impalad=True)
> common/impala_test_suite.py:408: in run_test_case
> result = self.__execute_query(target_impalad_client, query, user=user)
> common/impala_test_suite.py:625: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:160: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:176: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:350: in __execute_query
> self.wait_for_finished(handle)
> beeswax/impala_beeswax.py:371: in wait_for_finished
> raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> EQuery aborted:Error(s) moving partition files. First error (of 1) was:
> Hdfs op (RENAME
> s3a:///test_insert_parquet_968f37fe.db/orders_insert_table/_impala_insert_staging/4e45cd68bcddd451_3c7156ed/.4e45cd68bcddd451-3c7156ed0002_803672621_dir/4e45cd68bcddd451-3c7156ed0002_448261088_data.0.parq
> TO
> s3a:///test-warehouse/test_insert_parquet_968f37fe.db/orders_insert_table/4e45cd68bcddd451-3c7156ed0002_448261088_data.0.parq)
> failed, error was:
> s3a:///test-warehouse/test_insert_parquet_968f37fe.db/orders_insert_table/_impala_insert_staging/4e45cd68bcddd451_3c7156ed/.4e45cd68bcddd451-3c7156ed0002_803672621_dir/4e45cd68bcddd451-3c7156ed0002_448261088_data.0.parq
> E Error(5): Input/output error{noformat}
> Since we know this happens once in a while, some ideas to deflake it:
> * retry
> * check for this specific issue... if we think its platform flakiness, then
> we should skip it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-2515) Impala is unable to read a Parquet decimal column if size is larger than needed
[
https://issues.apache.org/jira/browse/IMPALA-2515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Volker updated IMPALA-2515:
Description:
Impala cannot read this:
{code}
{"name": "tmp_1",
"type": "fixed",
"size": 8,
"logicalType": "decimal",
"precision": 10,
"scale": 5}
{code}
However, this can be read:
{code}
{"name": "tmp_1",
"type": "fixed",
"size": 5,
"logicalType": "decimal",
"precision": 10,
"scale": 5}
{code}
Size must be precisely set to this, or Impala is unable to read the decimal
column:
{code}
size = int(math.ceil((math.log(2, 10) + precision) / math.log(256, 10)))
{code}
There is nothing in the Parquet spec that says that Decimal columns must be
sized precisely. Arguably it's a bug in the writer if it's doing it, because
it's just wasting space.
https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#decimal
was:
Impala cannot read this:
{code}
{"name": "tmp_1",
"type": "fixed",
"size": 8,
"logicalType": "decimal",
"precision": 10,
"scale": 5}
{code}
However, this can be read:
{code}
{"name": "tmp_1",
"type": "fixed",
"size": 5,
"logicalType": "decimal",
"precision": 10,
"scale": 5}
{code}
Size must be precisely set to this, or Impala is unable to read the decimal
column:
{code}
size = int(math.ceil((math.log(2, 10) + precision) / math.log(256, 10)))
{code}
There is nothing in the Parquet spec that says that Decimal columns must be
sized precisely. Arguably it's a bug in the writer if it's doing it, because
it's just wasting space.
https://github.com/Parquet/parquet-format/blob/master/LogicalTypes.md#decimal
> Impala is unable to read a Parquet decimal column if size is larger than
> needed
> ---
>
> Key: IMPALA-2515
> URL: https://issues.apache.org/jira/browse/IMPALA-2515
> Project: IMPALA
> Issue Type: Sub-task
> Components: Backend
>Affects Versions: Impala 2.3.0
>Reporter: Taras Bobrovytsky
>Priority: Minor
> Labels: ramp-up
>
> Impala cannot read this:
> {code}
> {"name": "tmp_1",
> "type": "fixed",
> "size": 8,
> "logicalType": "decimal",
> "precision": 10,
> "scale": 5}
> {code}
> However, this can be read:
> {code}
> {"name": "tmp_1",
> "type": "fixed",
> "size": 5,
> "logicalType": "decimal",
> "precision": 10,
> "scale": 5}
> {code}
> Size must be precisely set to this, or Impala is unable to read the decimal
> column:
> {code}
> size = int(math.ceil((math.log(2, 10) + precision) / math.log(256, 10)))
> {code}
> There is nothing in the Parquet spec that says that Decimal columns must be
> sized precisely. Arguably it's a bug in the writer if it's doing it, because
> it's just wasting space.
> https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#decimal
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7351) Add memory estimates for plan nodes and sinks with missing estimates
[ https://issues.apache.org/jira/browse/IMPALA-7351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664412#comment-16664412 ] ASF subversion and git services commented on IMPALA-7351: - Commit 8cbec20ea4f1399fce14018d30050d4b7ee501bc in impala's branch refs/heads/master from [~bikram.sngh91] [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=8cbec20 ] IMPALA-7351: Add estimates to Exchange node Added rough estimates for exchange node and a justification of the method in the in-line comments. Testing: Updated Planner tests. Change-Id: I5b577f9511abc48b992e814d50bba4959f23f7fd Reviewed-on: http://gerrit.cloudera.org:8080/11692 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Add memory estimates for plan nodes and sinks with missing estimates > > > Key: IMPALA-7351 > URL: https://issues.apache.org/jira/browse/IMPALA-7351 > Project: IMPALA > Issue Type: Sub-task > Components: Frontend >Reporter: Tim Armstrong >Assignee: Bikramjeet Vig >Priority: Major > Labels: admission-control, resource-management > > Many plan nodes and sinks, e.g. KuduScanNode, KuduTableSink, ExchangeNode, > etc are missing memory estimates entirely. > We should add a basic estimate for all these cases based on experiments and > data from real workloads. In some cases 0 may be the right estimate (e.g. for > streaming nodes like SelectNode that just pass through data) but we should > remove TODOs and document the reasoning in those cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7765) Impala 3.1 Doc: Docuement MAX_MEM_ESTIMATE_FOR_ADMISSION
Alex Rodoni created IMPALA-7765: --- Summary: Impala 3.1 Doc: Docuement MAX_MEM_ESTIMATE_FOR_ADMISSION Key: IMPALA-7765 URL: https://issues.apache.org/jira/browse/IMPALA-7765 Project: IMPALA Issue Type: Sub-task Components: Docs Reporter: Alex Rodoni Assignee: Alex Rodoni -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-7765) Impala 3.1 Doc: Docuement MAX_MEM_ESTIMATE_FOR_ADMISSION
Alex Rodoni created IMPALA-7765: --- Summary: Impala 3.1 Doc: Docuement MAX_MEM_ESTIMATE_FOR_ADMISSION Key: IMPALA-7765 URL: https://issues.apache.org/jira/browse/IMPALA-7765 Project: IMPALA Issue Type: Sub-task Components: Docs Reporter: Alex Rodoni Assignee: Alex Rodoni -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7753) Invalid logic when rewriting ORDER BY clause expressions
[
https://issues.apache.org/jira/browse/IMPALA-7753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664395#comment-16664395
]
Paul Rogers commented on IMPALA-7753:
-
Checked if an ORDER BY clause can contain an expression (if not, we'd not need
to do rewrites). Turns out it can:
{noformat}
order_by_elements ::=
order_by_element:e
...
order_by_element ::=
expr:e opt_order_param:o opt_nulls_order_param:n
{noformat}
Given that the parser allows an "expr" in the order by clause, applying
expression rewrite rules seems reasonable.
> Invalid logic when rewriting ORDER BY clause expressions
>
>
> Key: IMPALA-7753
> URL: https://issues.apache.org/jira/browse/IMPALA-7753
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 3.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> The select statement represents the ORDER BY clause in two distinct ways.
> First, there is a list of the "original" ordering expressions,
> {{orderByElements_}}. Second, there is an analyzed list in {{sortInfo_}}. The
> explanation is:
> {code:java}
> // create copies, we don't want to modify the original parse node, in
> case
> // we need to print it
> {code}
> Later, we apply rewrite rules to the {{ORDER BY}} expression, but we do so
> using the original version, not the copy:
> {code:java}
> for (OrderByElement orderByElem: orderByElements_) {
> orderByElem.setExpr(rewriteCheckOrdinalResult(rewriter,
> orderByElem.getExpr()));
> }
> {code}
> The result is that we apply rewrite rules to expressions which have not been
> analyzed, triggering the assertion mentioned above. This assertion is telling
> us something: we skipped a step. Here, it turns out we are rewriting the
> wrong set of expressions. Modifying the code to rewrite those in
> {{sortInfo_}} solves the problem. The current behavior is a bug as the
> rewrites currently do nothing, and the expressions we thought we were
> rewriting are never touched.
> The correct code would rewrite the expressions which are actually used when
> analyzing the query:
> {code}if (orderByElements_ != null) {
> List sortExprs = sortInfo_.getSortExprs();
> for (int i = 0; i < sortExprs.size(); i++) {
> sortExprs.set(i, rewriteCheckOrdinalResult(rewriter,
> sortExprs.get(i)));
> }
> {code}
> We can, in addition, ask a more basic question: do we even need to do
> rewrites for {{ORDER BY}} expressions? The only valid expressions are column
> references, aren't they? Or, does Impala allow expressions in the {{ORDER
> BY}} clause?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7762) Add a timeout in Impala when waiting for a catalog update
[ https://issues.apache.org/jira/browse/IMPALA-7762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fredy Wijaya updated IMPALA-7762: - Affects Version/s: Impala 3.1.0 > Add a timeout in Impala when waiting for a catalog update > - > > Key: IMPALA-7762 > URL: https://issues.apache.org/jira/browse/IMPALA-7762 > Project: IMPALA > Issue Type: Improvement > Components: Catalog >Affects Versions: Impala 3.1.0 >Reporter: Fredy Wijaya >Priority: Major > > When something goes wrong, "invalidate metadata" can cause a hang while > waiting for a catalog update that never arrives. There's a TODO > (https://github.com/apache/impala/blob/1d72c7584fd33746b17826c1dc54a1819fce9ec5/be/src/service/impala-server.cc#L1612-L1613) > in the code to set a timeout instead of waiting indefinitely. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7763) Consider locking principal/privilege update and Sentry refresh operations
[ https://issues.apache.org/jira/browse/IMPALA-7763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fredy Wijaya updated IMPALA-7763: - Affects Version/s: Impala 3.1.0 > Consider locking principal/privilege update and Sentry refresh operations > -- > > Key: IMPALA-7763 > URL: https://issues.apache.org/jira/browse/IMPALA-7763 > Project: IMPALA > Issue Type: Improvement > Components: Catalog >Affects Versions: Impala 3.1.0 >Reporter: Fredy Wijaya >Priority: Major > > There's currently no lock between a Sentry refresh and any operations outside > Sentry refresh for updating principal and privileges, which can cause the > catalog to be temporarily inconsistent. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7764) SentryProxy needs unit tests
[ https://issues.apache.org/jira/browse/IMPALA-7764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fredy Wijaya updated IMPALA-7764: - Summary: SentryProxy needs unit tests (was: SentryProxy needs some tests) > SentryProxy needs unit tests > > > Key: IMPALA-7764 > URL: https://issues.apache.org/jira/browse/IMPALA-7764 > Project: IMPALA > Issue Type: Improvement > Components: Infrastructure >Affects Versions: Impala 3.1.0 >Reporter: Fredy Wijaya >Assignee: Fredy Wijaya >Priority: Major > > There are currently no unit tests in SentryProxy, which can lead to a number > of bugs when changing code in SentryProxy. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7764) SentryProxy needs some tests
Fredy Wijaya created IMPALA-7764: Summary: SentryProxy needs some tests Key: IMPALA-7764 URL: https://issues.apache.org/jira/browse/IMPALA-7764 Project: IMPALA Issue Type: Improvement Components: Infrastructure Affects Versions: Impala 3.1.0 Reporter: Fredy Wijaya Assignee: Fredy Wijaya There are currently no unit tests in SentryProxy, which can lead to a number of bugs when changing code in SentryProxy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (IMPALA-6852) Add a section to the query profile covering number of fragments and total bytes scanned per host
[ https://issues.apache.org/jira/browse/IMPALA-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Ho reassigned IMPALA-6852: -- Assignee: Michal Ostrowski > Add a section to the query profile covering number of fragments and total > bytes scanned per host > > > Key: IMPALA-6852 > URL: https://issues.apache.org/jira/browse/IMPALA-6852 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 2.13.0 >Reporter: Mostafa Mokhtar >Assignee: Michal Ostrowski >Priority: Major > Labels: supportability > > Root causing performance issues usually comes down to skew in terms of work > done across hosts. Un-even assignment of fragments or bytes scanned per host > is a common. > Making this information readily available in the query profile will help > speedup RCA. > Proposal is to add two tables to the query profile that cover > * Number of fragments per host > * Number of bytes scanned per host -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-5904) Enable ThreadSanitizer for Impala
[ https://issues.apache.org/jira/browse/IMPALA-5904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Ho reassigned IMPALA-5904: -- Assignee: Michal Ostrowski > Enable ThreadSanitizer for Impala > - > > Key: IMPALA-5904 > URL: https://issues.apache.org/jira/browse/IMPALA-5904 > Project: IMPALA > Issue Type: Improvement > Components: Infrastructure >Reporter: Tim Armstrong >Assignee: Michal Ostrowski >Priority: Major > Attachments: impalad.ERROR > > > It would be great to be able to automatically detect data races in Impala > using ThreadSanitizer to avoid tricky-to-reproduce bugs. This issue tracks > enabling ThreadSanitizer, fixing bugs and adding suppressions to get to the > point where Impala runs cleanly with the sanitizer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-6501) Optimize count(*) for Kudu scans
[ https://issues.apache.org/jira/browse/IMPALA-6501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Ho reassigned IMPALA-6501: -- Assignee: (was: Michal Ostrowski) > Optimize count(*) for Kudu scans > > > Key: IMPALA-6501 > URL: https://issues.apache.org/jira/browse/IMPALA-6501 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 2.11.0 >Reporter: Tim Armstrong >Priority: Major > Labels: kudu > > IMPALA-5036 added an optimisation for count(*) in Parquet scans that avoids > materialising dummy rows. We should do something similar for Kudu. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-4337) Wrap long lines in explain plans
[ https://issues.apache.org/jira/browse/IMPALA-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Ho reassigned IMPALA-4337: -- Assignee: Michal Ostrowski > Wrap long lines in explain plans > > > Key: IMPALA-4337 > URL: https://issues.apache.org/jira/browse/IMPALA-4337 > Project: IMPALA > Issue Type: Improvement > Components: Frontend >Affects Versions: Impala 2.8.0 >Reporter: Henry Robinson >Assignee: Michal Ostrowski >Priority: Minor > Labels: newbie > > Explain plans can have very long lines, particularly when printing lists of > expressions. It should be possible to wrap, and still correctly indent, those > lines. > This is trickier than it sounds because they have to be wrapped in the > context of their place in the plan (i.e. with appropriate prefixes etc). It's > a good opportunity to split out explain plan generation from presentation, > centralizing the logic so that this kind of change is easy to make. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7763) Consider locking principal/privilege update and Sentry refresh operations
Fredy Wijaya created IMPALA-7763: Summary: Consider locking principal/privilege update and Sentry refresh operations Key: IMPALA-7763 URL: https://issues.apache.org/jira/browse/IMPALA-7763 Project: IMPALA Issue Type: Improvement Components: Catalog Reporter: Fredy Wijaya There's currently no lock between a Sentry refresh and any operations outside Sentry refresh for updating principal and privileges, which can cause the catalog to be temporarily inconsistent. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7763) Consider locking principal/privilege update and Sentry refresh operations
Fredy Wijaya created IMPALA-7763: Summary: Consider locking principal/privilege update and Sentry refresh operations Key: IMPALA-7763 URL: https://issues.apache.org/jira/browse/IMPALA-7763 Project: IMPALA Issue Type: Improvement Components: Catalog Reporter: Fredy Wijaya There's currently no lock between a Sentry refresh and any operations outside Sentry refresh for updating principal and privileges, which can cause the catalog to be temporarily inconsistent. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (IMPALA-6501) Optimize count(*) for Kudu scans
[ https://issues.apache.org/jira/browse/IMPALA-6501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Ho reassigned IMPALA-6501: -- Assignee: Michal Ostrowski > Optimize count(*) for Kudu scans > > > Key: IMPALA-6501 > URL: https://issues.apache.org/jira/browse/IMPALA-6501 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 2.11.0 >Reporter: Tim Armstrong >Assignee: Michal Ostrowski >Priority: Major > Labels: kudu > > IMPALA-5036 added an optimisation for count(*) in Parquet scans that avoids > materialising dummy rows. We should do something similar for Kudu. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7762) Add a timeout in Impala when waiting for a catalog update
[ https://issues.apache.org/jira/browse/IMPALA-7762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fredy Wijaya updated IMPALA-7762: - Issue Type: Improvement (was: Bug) > Add a timeout in Impala when waiting for a catalog update > - > > Key: IMPALA-7762 > URL: https://issues.apache.org/jira/browse/IMPALA-7762 > Project: IMPALA > Issue Type: Improvement > Components: Catalog >Reporter: Fredy Wijaya >Priority: Major > > When something goes wrong, "invalidate metadata" can cause a hang while > waiting for a catalog update that never arrives. There's a TODO > (https://github.com/apache/impala/blob/1d72c7584fd33746b17826c1dc54a1819fce9ec5/be/src/service/impala-server.cc#L1612-L1613) > in the code to set a timeout instead of waiting indefinitely. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7762) Add a timeout in Impala when waiting for a catalog update
Fredy Wijaya created IMPALA-7762: Summary: Add a timeout in Impala when waiting for a catalog update Key: IMPALA-7762 URL: https://issues.apache.org/jira/browse/IMPALA-7762 Project: IMPALA Issue Type: Bug Components: Catalog Reporter: Fredy Wijaya When something goes wrong, "invalidate metadata" can cause a hang while waiting for a catalog update that never arrives. There's a TODO (https://github.com/apache/impala/blob/1d72c7584fd33746b17826c1dc54a1819fce9ec5/be/src/service/impala-server.cc#L1612-L1620) in the code to set a timeout instead of waiting indefinitely. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (IMPALA-7762) Add a timeout in Impala when waiting for a catalog update
[ https://issues.apache.org/jira/browse/IMPALA-7762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fredy Wijaya updated IMPALA-7762: - Description: When something goes wrong, "invalidate metadata" can cause a hang while waiting for a catalog update that never arrives. There's a TODO (https://github.com/apache/impala/blob/1d72c7584fd33746b17826c1dc54a1819fce9ec5/be/src/service/impala-server.cc#L1612-L1613) in the code to set a timeout instead of waiting indefinitely. (was: When something goes wrong, "invalidate metadata" can cause a hang while waiting for a catalog update that never arrives. There's a TODO (https://github.com/apache/impala/blob/1d72c7584fd33746b17826c1dc54a1819fce9ec5/be/src/service/impala-server.cc#L1612-L1620) in the code to set a timeout instead of waiting indefinitely.) > Add a timeout in Impala when waiting for a catalog update > - > > Key: IMPALA-7762 > URL: https://issues.apache.org/jira/browse/IMPALA-7762 > Project: IMPALA > Issue Type: Bug > Components: Catalog >Reporter: Fredy Wijaya >Priority: Major > > When something goes wrong, "invalidate metadata" can cause a hang while > waiting for a catalog update that never arrives. There's a TODO > (https://github.com/apache/impala/blob/1d72c7584fd33746b17826c1dc54a1819fce9ec5/be/src/service/impala-server.cc#L1612-L1613) > in the code to set a timeout instead of waiting indefinitely. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7761) Add multiple count distinct to targeted stress and targeted perf
Thomas Tauber-Marshall created IMPALA-7761: -- Summary: Add multiple count distinct to targeted stress and targeted perf Key: IMPALA-7761 URL: https://issues.apache.org/jira/browse/IMPALA-7761 Project: IMPALA Issue Type: Improvement Components: Infrastructure Affects Versions: Impala 3.1.0 Reporter: Thomas Tauber-Marshall Assignee: Thomas Tauber-Marshall With IMPALA-110 in, we should add queries with multiple count distinct to targeted stress and targeted perf -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-7761) Add multiple count distinct to targeted stress and targeted perf
Thomas Tauber-Marshall created IMPALA-7761: -- Summary: Add multiple count distinct to targeted stress and targeted perf Key: IMPALA-7761 URL: https://issues.apache.org/jira/browse/IMPALA-7761 Project: IMPALA Issue Type: Improvement Components: Infrastructure Affects Versions: Impala 3.1.0 Reporter: Thomas Tauber-Marshall Assignee: Thomas Tauber-Marshall With IMPALA-110 in, we should add queries with multiple count distinct to targeted stress and targeted perf -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7748) Remove the appx_count_distinct query option
[ https://issues.apache.org/jira/browse/IMPALA-7748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Tauber-Marshall resolved IMPALA-7748. Resolution: Not A Problem After further discussion, we've decided t leave appx_count_distinct as is > Remove the appx_count_distinct query option > --- > > Key: IMPALA-7748 > URL: https://issues.apache.org/jira/browse/IMPALA-7748 > Project: IMPALA > Issue Type: Improvement >Affects Versions: Impala 3.1.0 >Reporter: Thomas Tauber-Marshall >Priority: Minor > > With IMPALA-110, we can now support multiple count distinct directly, and the > appx_count_distinct query option is no longer needed. Users who want the perf > improvement from it can always just use the ndv() function directly in their > sql. > We'll mark this option as deprecated in the docs starting from 3.1. Removing > it can be targeted for 4.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7748) Remove the appx_count_distinct query option
[ https://issues.apache.org/jira/browse/IMPALA-7748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Tauber-Marshall resolved IMPALA-7748. Resolution: Not A Problem After further discussion, we've decided t leave appx_count_distinct as is > Remove the appx_count_distinct query option > --- > > Key: IMPALA-7748 > URL: https://issues.apache.org/jira/browse/IMPALA-7748 > Project: IMPALA > Issue Type: Improvement >Affects Versions: Impala 3.1.0 >Reporter: Thomas Tauber-Marshall >Priority: Minor > > With IMPALA-110, we can now support multiple count distinct directly, and the > appx_count_distinct query option is no longer needed. Users who want the perf > improvement from it can always just use the ndv() function directly in their > sql. > We'll mark this option as deprecated in the docs starting from 3.1. Removing > it can be targeted for 4.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (IMPALA-6352) TestTableSample took too long in recent tests
[
https://issues.apache.org/jira/browse/IMPALA-6352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-6352:
--
Target Version: Impala 3.2.0, Impala 2.13.0 (was: Impala 2.13.0)
> TestTableSample took too long in recent tests
> -
>
> Key: IMPALA-6352
> URL: https://issues.apache.org/jira/browse/IMPALA-6352
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 2.12.0
>Reporter: Vuk Ercegovac
>Assignee: Bikramjeet Vig
>Priority: Critical
> Labels: broken-build
> Attachments: impala6352pstacks.tar.gz
>
>
> TestTableSample test took ~8 hours in recent (12/21) exhaustive rhel tests.
> That caused the overall test to be aborted:
> ...
> 01:53:10 [gw2] PASSED
> query_test/test_tablesample.py::TestTableSample::test_tablesample[repeatable:
> True | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> seq/gzip/block]
> 01:53:10
> query_test/test_tablesample.py::TestTableSample::test_tablesample[repeatable:
> False | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': True,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> seq/gzip/block]
> 10:03:51 [gw2] PASSED
> query_test/test_tablesample.py::TestTableSample::test_tablesample[repeatable:
> False | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': True,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> seq/gzip/block] Build timed out (after 1,440 minutes). Marking the build as
> aborted.
> 10:03:51 Build was aborted
> ...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-7485) test_spilling_naaj hung on jenkins
[
https://issues.apache.org/jira/browse/IMPALA-7485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong reassigned IMPALA-7485:
-
Assignee: Tim Armstrong
> test_spilling_naaj hung on jenkins
> --
>
> Key: IMPALA-7485
> URL: https://issues.apache.org/jira/browse/IMPALA-7485
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Reporter: Csaba Ringhofer
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: broken-build, flaky, flaky-test
> Attachments: resolved_stacks.zip
>
>
> {code}
> query_test/test_spilling.py::TestSpillingDebugActionDimensions::test_spilling_naaj[exec_option:
> {'debug_action': None, 'default_spillable_buffer_size': '256k'} |
> table_format: parquet/none]
> {code}
> seemed hung (was running for more than 4 hours), see
> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/3055/console
> Core dumps and stack traces of impalad were created and the impalad was
> killed. The tests continued without failures after impalad was restarted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7353) Fix bogus too-high memory estimates
[
https://issues.apache.org/jira/browse/IMPALA-7353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664331#comment-16664331
]
Tim Armstrong commented on IMPALA-7353:
---
DataSourceScanNode is the only one left per Bikram.
> Fix bogus too-high memory estimates
> ---
>
> Key: IMPALA-7353
> URL: https://issues.apache.org/jira/browse/IMPALA-7353
> Project: IMPALA
> Issue Type: Sub-task
> Components: Frontend
>Reporter: Tim Armstrong
>Assignee: Bikramjeet Vig
>Priority: Major
> Labels: resource-management
>
> Some operators have bogus memory estimates that are probably way too high. We
> should adjust them to be a little more realistic.
> E.g. in HBaseScanNode:
> {code}
> @Override
> public void computeNodeResourceProfile(TQueryOptions queryOptions) {
> // TODO: What's a good estimate of memory consumption?
> nodeResourceProfile_ = ResourceProfile.noReservation(1024L * 1024L *
> 1024L);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7351) Add memory estimates for plan nodes and sinks with missing estimates
[ https://issues.apache.org/jira/browse/IMPALA-7351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong updated IMPALA-7351: -- Labels: admission-control resource-management (was: resource-management) > Add memory estimates for plan nodes and sinks with missing estimates > > > Key: IMPALA-7351 > URL: https://issues.apache.org/jira/browse/IMPALA-7351 > Project: IMPALA > Issue Type: Sub-task > Components: Frontend >Reporter: Tim Armstrong >Assignee: Bikramjeet Vig >Priority: Major > Labels: admission-control, resource-management > > Many plan nodes and sinks, e.g. KuduScanNode, KuduTableSink, ExchangeNode, > etc are missing memory estimates entirely. > We should add a basic estimate for all these cases based on experiments and > data from real workloads. In some cases 0 may be the right estimate (e.g. for > streaming nodes like SelectNode that just pass through data) but we should > remove TODOs and document the reasoning in those cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7351) Add memory estimates for plan nodes and sinks with missing estimates
[ https://issues.apache.org/jira/browse/IMPALA-7351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong updated IMPALA-7351: -- Target Version: Impala 3.2.0 (was: Impala 3.1.0) > Add memory estimates for plan nodes and sinks with missing estimates > > > Key: IMPALA-7351 > URL: https://issues.apache.org/jira/browse/IMPALA-7351 > Project: IMPALA > Issue Type: Sub-task > Components: Frontend >Reporter: Tim Armstrong >Assignee: Bikramjeet Vig >Priority: Major > Labels: admission-control, resource-management > > Many plan nodes and sinks, e.g. KuduScanNode, KuduTableSink, ExchangeNode, > etc are missing memory estimates entirely. > We should add a basic estimate for all these cases based on experiments and > data from real workloads. In some cases 0 may be the right estimate (e.g. for > streaming nodes like SelectNode that just pass through data) but we should > remove TODOs and document the reasoning in those cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-7759) Add Levenshtein edit distance built-in function
[ https://issues.apache.org/jira/browse/IMPALA-7759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-7759 started by Greg Rahn. - > Add Levenshtein edit distance built-in function > --- > > Key: IMPALA-7759 > URL: https://issues.apache.org/jira/browse/IMPALA-7759 > Project: IMPALA > Issue Type: New Feature >Reporter: Greg Rahn >Assignee: Greg Rahn >Priority: Major > Labels: built-in-function > > References: > * [Netezza - > le_dst()|https://www.ibm.com/support/knowledgecenter/en/SSULQD_7.2.1/com.ibm.nz.dbu.doc/r_dbuser_functions_expressions_fuzzy_funcs.html] > * [Postgres - > levenshtein()|https://www.postgresql.org/docs/current/static/fuzzystrmatch.html#id-1.11.7.24.6] > One notable difference: > * Netezza: if either value is NULL, returns the length of non-NULL value > * Postgres: if either value is NULL, returns NULL > Preference is to implement Postgres version due to ease of cross-system > testing. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-7467) Port ExecQueryFInstances() to KPRC
[ https://issues.apache.org/jira/browse/IMPALA-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Ho reassigned IMPALA-7467: -- Assignee: Thomas Tauber-Marshall > Port ExecQueryFInstances() to KPRC > -- > > Key: IMPALA-7467 > URL: https://issues.apache.org/jira/browse/IMPALA-7467 > Project: IMPALA > Issue Type: Sub-task > Components: Distributed Exec >Affects Versions: Impala 3.1.0 >Reporter: Michael Ho >Assignee: Thomas Tauber-Marshall >Priority: Major > > Port ExecQueryFInstances() from Thrift to KRPC. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7759) Add Levenshtein edit distance built-in function
[ https://issues.apache.org/jira/browse/IMPALA-7759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Rahn updated IMPALA-7759: -- Summary: Add Levenshtein edit distance built-in function (was: Add Levenshtein edit distance function) > Add Levenshtein edit distance built-in function > --- > > Key: IMPALA-7759 > URL: https://issues.apache.org/jira/browse/IMPALA-7759 > Project: IMPALA > Issue Type: New Feature >Reporter: Greg Rahn >Assignee: Greg Rahn >Priority: Major > Labels: built-in-function > > References: > * [Netezza - > le_dst()|https://www.ibm.com/support/knowledgecenter/en/SSULQD_7.2.1/com.ibm.nz.dbu.doc/r_dbuser_functions_expressions_fuzzy_funcs.html] > * [Postgres - > levenshtein()|https://www.postgresql.org/docs/current/static/fuzzystrmatch.html#id-1.11.7.24.6] > One notable difference: > * Netezza: if either value is NULL, returns the length of non-NULL value > * Postgres: if either value is NULL, returns NULL > Preference is to implement Postgres version due to ease of cross-system > testing. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7357) Consider lowering statestore update frequency further or adding jitter to admission controller
[ https://issues.apache.org/jira/browse/IMPALA-7357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong updated IMPALA-7357: -- Target Version: Impala 3.2.0 (was: Impala 3.1.0) > Consider lowering statestore update frequency further or adding jitter to > admission controller > -- > > Key: IMPALA-7357 > URL: https://issues.apache.org/jira/browse/IMPALA-7357 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Reporter: Tim Armstrong >Priority: Major > Labels: admission-control > > We could consider reducing the update frequency further (maybe just for > coordinators??) and possibly adding some jitter to the admission process to > prevent overadmission when multiple queries are submitted at the same time to > different coordinators. > The problem is that the coordinators independently make admission decisions > before the change in state from previous decisions is propagated to other > coordinators. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7735) Expose admission control status in impala-shell
[ https://issues.apache.org/jira/browse/IMPALA-7735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong updated IMPALA-7735: -- Target Version: Impala 3.2.0 (was: Impala 3.1.0) > Expose admission control status in impala-shell > --- > > Key: IMPALA-7735 > URL: https://issues.apache.org/jira/browse/IMPALA-7735 > Project: IMPALA > Issue Type: Improvement > Components: Clients >Affects Versions: Impala 3.1.0 >Reporter: Tim Armstrong >Assignee: Bikramjeet Vig >Priority: Critical > Labels: admission-control > Attachments: Screenshot1.png > > > Following on from IMPALA-7545 we should also expose this in impala-shell. I > left some notes on that JIRA. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7760) Privilege version inconsistency causes a hang when running invalidate metadata
Fredy Wijaya created IMPALA-7760: Summary: Privilege version inconsistency causes a hang when running invalidate metadata Key: IMPALA-7760 URL: https://issues.apache.org/jira/browse/IMPALA-7760 Project: IMPALA Issue Type: Bug Components: Catalog Affects Versions: Impala 3.1.0 Reporter: Fredy Wijaya Assignee: Fredy Wijaya 1. Add a sleep statement before getting all Sentry roles in SentryProxy. 2. Remove role any existing role via Sentry CLI. 3. Run invalidate metadata, this will cause Impala to hang. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-7760) Privilege version inconsistency causes a hang when running invalidate metadata
Fredy Wijaya created IMPALA-7760: Summary: Privilege version inconsistency causes a hang when running invalidate metadata Key: IMPALA-7760 URL: https://issues.apache.org/jira/browse/IMPALA-7760 Project: IMPALA Issue Type: Bug Components: Catalog Affects Versions: Impala 3.1.0 Reporter: Fredy Wijaya Assignee: Fredy Wijaya 1. Add a sleep statement before getting all Sentry roles in SentryProxy. 2. Remove role any existing role via Sentry CLI. 3. Run invalidate metadata, this will cause Impala to hang. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-7760) Privilege version inconsistency causes a hang when running invalidate metadata
[ https://issues.apache.org/jira/browse/IMPALA-7760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-7760 started by Fredy Wijaya. > Privilege version inconsistency causes a hang when running invalidate metadata > -- > > Key: IMPALA-7760 > URL: https://issues.apache.org/jira/browse/IMPALA-7760 > Project: IMPALA > Issue Type: Bug > Components: Catalog >Affects Versions: Impala 3.1.0 >Reporter: Fredy Wijaya >Assignee: Fredy Wijaya >Priority: Critical > > 1. Add a sleep statement before getting all Sentry roles in SentryProxy. > 2. Remove role any existing role via Sentry CLI. > 3. Run invalidate metadata, this will cause Impala to hang. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7523) Planner Test failing with "Failed to assign regions to servers after 60000 millis."
[
https://issues.apache.org/jira/browse/IMPALA-7523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664302#comment-16664302
]
Tim Armstrong commented on IMPALA-7523:
---
Happened in GVO:
https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/3427/testReport/junit/org.apache.impala.planner/S3PlannerTest/org_apache_impala_planner_S3PlannerTest/
> Planner Test failing with "Failed to assign regions to servers after 6
> millis."
> ---
>
> Key: IMPALA-7523
> URL: https://issues.apache.org/jira/browse/IMPALA-7523
> Project: IMPALA
> Issue Type: Task
> Components: Frontend
>Reporter: Philip Zeyliger
>Priority: Critical
> Labels: broken-build, flaky
>
> I've seen
> {{org.apache.impala.planner.PlannerTest.org.apache.impala.planner.PlannerTest}}
> fail with the following trace:
> {code}
> java.lang.IllegalStateException: Failed to assign regions to servers after
> 6 millis.
> at
> org.apache.impala.datagenerator.HBaseTestDataRegionAssignment.performAssignment(HBaseTestDataRegionAssignment.java:153)
> at
> org.apache.impala.planner.PlannerTestBase.setUp(PlannerTestBase.java:120)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> {code}
> I think we've seen it before as indicated in IMPALA-7061.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7759) Add Levenshtein edit distance function
[ https://issues.apache.org/jira/browse/IMPALA-7759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Rahn updated IMPALA-7759: -- Description: References: * [Netezza - le_dst()|https://www.ibm.com/support/knowledgecenter/en/SSULQD_7.2.1/com.ibm.nz.dbu.doc/r_dbuser_functions_expressions_fuzzy_funcs.html] * [Postgres - levenshtein()|https://www.postgresql.org/docs/current/static/fuzzystrmatch.html#id-1.11.7.24.6] One notable difference: * Netezza: if either value is NULL, returns the length of non-NULL value * Postgres: if either value is NULL, returns NULL Preference is to implement Postgres version due to ease of cross-system testing. was: References: * [Netezza - (le_dst())|https://www.ibm.com/support/knowledgecenter/en/SSULQD_7.2.1/com.ibm.nz.dbu.doc/r_dbuser_functions_expressions_fuzzy_funcs.html] * [Postgres - levenshtein()|https://www.postgresql.org/docs/current/static/fuzzystrmatch.html#id-1.11.7.24.6] One notable difference: * Netezza: if either value is NULL, returns the length of non-NULL value * Postgres: if either value is NULL, returns NULL Preference is to implement Postgres version due to ease of cross-system testing. > Add Levenshtein edit distance function > -- > > Key: IMPALA-7759 > URL: https://issues.apache.org/jira/browse/IMPALA-7759 > Project: IMPALA > Issue Type: New Feature >Reporter: Greg Rahn >Assignee: Greg Rahn >Priority: Major > Labels: built-in-function > > References: > * [Netezza - > le_dst()|https://www.ibm.com/support/knowledgecenter/en/SSULQD_7.2.1/com.ibm.nz.dbu.doc/r_dbuser_functions_expressions_fuzzy_funcs.html] > * [Postgres - > levenshtein()|https://www.postgresql.org/docs/current/static/fuzzystrmatch.html#id-1.11.7.24.6] > One notable difference: > * Netezza: if either value is NULL, returns the length of non-NULL value > * Postgres: if either value is NULL, returns NULL > Preference is to implement Postgres version due to ease of cross-system > testing. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7759) Add Levenshtein edit distance function
Greg Rahn created IMPALA-7759: - Summary: Add Levenshtein edit distance function Key: IMPALA-7759 URL: https://issues.apache.org/jira/browse/IMPALA-7759 Project: IMPALA Issue Type: New Feature Reporter: Greg Rahn Assignee: Greg Rahn References: * [Netezza - (le_dst())|https://www.ibm.com/support/knowledgecenter/en/SSULQD_7.2.1/com.ibm.nz.dbu.doc/r_dbuser_functions_expressions_fuzzy_funcs.html] * [Postgres - levenshtein()|https://www.postgresql.org/docs/current/static/fuzzystrmatch.html#id-1.11.7.24.6] One notable difference: * Netezza: if either value is NULL, returns the length of non-NULL value * Postgres: if either value is NULL, returns NULL Preference is to implement Postgres version due to ease of cross-system testing. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7759) Add Levenshtein edit distance function
[ https://issues.apache.org/jira/browse/IMPALA-7759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Rahn updated IMPALA-7759: -- Labels: built-in-function (was: ) > Add Levenshtein edit distance function > -- > > Key: IMPALA-7759 > URL: https://issues.apache.org/jira/browse/IMPALA-7759 > Project: IMPALA > Issue Type: New Feature >Reporter: Greg Rahn >Assignee: Greg Rahn >Priority: Major > Labels: built-in-function > > References: > * [Netezza - > (le_dst())|https://www.ibm.com/support/knowledgecenter/en/SSULQD_7.2.1/com.ibm.nz.dbu.doc/r_dbuser_functions_expressions_fuzzy_funcs.html] > * [Postgres - > levenshtein()|https://www.postgresql.org/docs/current/static/fuzzystrmatch.html#id-1.11.7.24.6] > One notable difference: > * Netezza: if either value is NULL, returns the length of non-NULL value > * Postgres: if either value is NULL, returns NULL > Preference is to implement Postgres version due to ease of cross-system > testing. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7759) Add Levenshtein edit distance function
Greg Rahn created IMPALA-7759: - Summary: Add Levenshtein edit distance function Key: IMPALA-7759 URL: https://issues.apache.org/jira/browse/IMPALA-7759 Project: IMPALA Issue Type: New Feature Reporter: Greg Rahn Assignee: Greg Rahn References: * [Netezza - (le_dst())|https://www.ibm.com/support/knowledgecenter/en/SSULQD_7.2.1/com.ibm.nz.dbu.doc/r_dbuser_functions_expressions_fuzzy_funcs.html] * [Postgres - levenshtein()|https://www.postgresql.org/docs/current/static/fuzzystrmatch.html#id-1.11.7.24.6] One notable difference: * Netezza: if either value is NULL, returns the length of non-NULL value * Postgres: if either value is NULL, returns NULL Preference is to implement Postgres version due to ease of cross-system testing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (IMPALA-2607) S3: need to determine the right default for fs.s3a.connection.maximum
[ https://issues.apache.org/jira/browse/IMPALA-2607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong reassigned IMPALA-2607: - Assignee: (was: Sailesh Mukil) > S3: need to determine the right default for fs.s3a.connection.maximum > - > > Key: IMPALA-2607 > URL: https://issues.apache.org/jira/browse/IMPALA-2607 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 2.3.0 >Reporter: Dan Hecht >Priority: Major > Labels: s3, usability > > When Impala is scanning parquet files off S3, it appears we can get into a > deadlock if fs.s3a.connection.maximum is not set high enough, because we can > spread the connection use across files but unless we have a connection for > all selected columns of a file, we can't make progress scanning that file. > So, we need to figure out the right default for fs.s3a.connection.maximum for > Impala, and also determine whether we need to put an upper bound on scanner > concurrency given the number of connections. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-4060) Timeout in stress test TPC-H queries when run against Kudu
[
https://issues.apache.org/jira/browse/IMPALA-4060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Tauber-Marshall resolved IMPALA-4060.
Resolution: Cannot Reproduce
> Timeout in stress test TPC-H queries when run against Kudu
> --
>
> Key: IMPALA-4060
> URL: https://issues.apache.org/jira/browse/IMPALA-4060
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Affects Versions: Impala 2.7.0
>Reporter: Dimitris Tsirogiannis
>Assignee: Thomas Tauber-Marshall
>Priority: Major
> Labels: kudu, stress
>
> Running the stress test against Kudu results in a large number of queries
> timing out with the following error:
> {noformat}
> I0831 22:57:36.206172 24679 jni-util.cc:169] java.lang.RuntimeException:
> Loading Kudu Table failed
> at
> com.cloudera.impala.planner.KuduScanNode.computeScanRangeLocations(KuduScanNode.java:155)
> at
> com.cloudera.impala.planner.KuduScanNode.init(KuduScanNode.java:104)
> at
> com.cloudera.impala.planner.SingleNodePlanner.createScanNode(SingleNodePlanner.java:1257)
> at
> com.cloudera.impala.planner.SingleNodePlanner.createTableRefNode(SingleNodePlanner.java:1470)
> at
> com.cloudera.impala.planner.SingleNodePlanner.createTableRefsPlan(SingleNodePlanner.java:745)
> at
> com.cloudera.impala.planner.SingleNodePlanner.createSelectPlan(SingleNodePlanner.java:585)
> at
> com.cloudera.impala.planner.SingleNodePlanner.createQueryPlan(SingleNodePlanner.java:236)
> at
> com.cloudera.impala.planner.SingleNodePlanner.createSingleNodePlan(SingleNodePlanner.java:144)
> at
> com.cloudera.impala.planner.Planner.createPlan(Planner.java:62)
> at
> com.cloudera.impala.service.Frontend.createExecRequest(Frontend.java:975)
> at
> com.cloudera.impala.service.JniFrontend.createExecRequest(JniFrontend.java:150)
> Caused by: com.stumbleupon.async.TimeoutException: Timed out after 1ms
> when joining Deferred@182109(state=PENDING, result=null, callback=get
> tablet locations from the master for table Kudu Master -> release master
> lookup permit -> retry RPC -> org.kududb.client.AsyncKuduClient$4@8e968ff ->
> wakeup thread Thread-55, errback=passthrough -> release master lookup permit
> -> retry RPC after error -> passthrough -> wakeup thread Thread-55)
> at com.stumbleupon.async.Deferred.doJoin(Deferred.java:1161)
> at com.stumbleupon.async.Deferred.join(Deferred.java:1029)
> at org.kududb.client.KuduClient.openTable(KuduClient.java:181)
> at
> com.cloudera.impala.planner.KuduScanNode.computeScanRangeLocations(KuduScanNode.java:119)
> ... 10 more
> {noformat}
> Another entry in the log that seems quite relevant is related to a long
> thread creation time:
> {noformat}
> W0831 22:52:59.616951 8714 thread.cc:502] negotiator [worker] (thread pool)
> Time spent creating pthread: real 37.363suser 0.000s sys 0.000s
> {noformat}
> The stress test was run in an EC2 8 node cluster with CDH 5.9.1 installed.
> The latest Kudu was installed using packages. OS version is Ubuntu 14.04. The
> stress test run TPC-H queries in scale factor 10.
> Filing this JIRA for Impala now until the necessary changes in the stress
> test generator are checked in.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-4462) Kudu stress test causes Timed out errors
[
https://issues.apache.org/jira/browse/IMPALA-4462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Tauber-Marshall resolved IMPALA-4462.
Resolution: Cannot Reproduce
> Kudu stress test causes Timed out errors
>
>
> Key: IMPALA-4462
> URL: https://issues.apache.org/jira/browse/IMPALA-4462
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.8.0
>Reporter: Taras Bobrovytsky
>Assignee: Thomas Tauber-Marshall
>Priority: Major
> Labels: kudu, stress
>
> When the stress test runs with high concurrency on the EC2 cluster, sometimes
> queries fail with the following error message. This was observed in cases
> where we run only select queries and in cases where we also run CRUD queries.
> {code}
> status.cc:114] Unable to open scanner: Timed out: Client connection
> negotiation failed: client connection to 172.28.195.167:7050: Timeout
> exceeded waiting to connect: Timed out: Client connection negotiation failed:
> client connection to 172.28.194.188:7050: Timeout exceeded waiting to connect
> @ 0x11b3ba1 (unknown)
> @ 0x17d5de7 (unknown)
> @ 0x1771308 (unknown)
> @ 0x1771726 (unknown)
> @ 0x1773646 (unknown)
> @ 0x177350c (unknown)
> @ 0x177308d (unknown)
> @ 0x1772ec3 (unknown)
> @ 0x132d4f4 (unknown)
> @ 0x15d6273 (unknown)
> @ 0x15dd24c (unknown)
> @ 0x15dd18f (unknown)
> @ 0x15dd0ea (unknown)
> @ 0x1a2518a (unknown)
> @ 0x7faf323299d1 (unknown)
> @ 0x7faf320768fd (unknown)
> runtime-state.cc:208] Error from query 554b52e24918e84c:9be6b9c9:
> Unable to open scanner: Timed out: Client connection negotiation failed:
> client connection to 172.28.195.167:7050: Timeout exceeded waiting to
> connect: Timed out: Client connection negotiation failed: client connection
> to 172.28.194.188:7050: Timeout exceeded waiting to connect
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (IMPALA-4060) Timeout in stress test TPC-H queries when run against Kudu
[
https://issues.apache.org/jira/browse/IMPALA-4060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Tauber-Marshall resolved IMPALA-4060.
Resolution: Cannot Reproduce
> Timeout in stress test TPC-H queries when run against Kudu
> --
>
> Key: IMPALA-4060
> URL: https://issues.apache.org/jira/browse/IMPALA-4060
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Affects Versions: Impala 2.7.0
>Reporter: Dimitris Tsirogiannis
>Assignee: Thomas Tauber-Marshall
>Priority: Major
> Labels: kudu, stress
>
> Running the stress test against Kudu results in a large number of queries
> timing out with the following error:
> {noformat}
> I0831 22:57:36.206172 24679 jni-util.cc:169] java.lang.RuntimeException:
> Loading Kudu Table failed
> at
> com.cloudera.impala.planner.KuduScanNode.computeScanRangeLocations(KuduScanNode.java:155)
> at
> com.cloudera.impala.planner.KuduScanNode.init(KuduScanNode.java:104)
> at
> com.cloudera.impala.planner.SingleNodePlanner.createScanNode(SingleNodePlanner.java:1257)
> at
> com.cloudera.impala.planner.SingleNodePlanner.createTableRefNode(SingleNodePlanner.java:1470)
> at
> com.cloudera.impala.planner.SingleNodePlanner.createTableRefsPlan(SingleNodePlanner.java:745)
> at
> com.cloudera.impala.planner.SingleNodePlanner.createSelectPlan(SingleNodePlanner.java:585)
> at
> com.cloudera.impala.planner.SingleNodePlanner.createQueryPlan(SingleNodePlanner.java:236)
> at
> com.cloudera.impala.planner.SingleNodePlanner.createSingleNodePlan(SingleNodePlanner.java:144)
> at
> com.cloudera.impala.planner.Planner.createPlan(Planner.java:62)
> at
> com.cloudera.impala.service.Frontend.createExecRequest(Frontend.java:975)
> at
> com.cloudera.impala.service.JniFrontend.createExecRequest(JniFrontend.java:150)
> Caused by: com.stumbleupon.async.TimeoutException: Timed out after 1ms
> when joining Deferred@182109(state=PENDING, result=null, callback=get
> tablet locations from the master for table Kudu Master -> release master
> lookup permit -> retry RPC -> org.kududb.client.AsyncKuduClient$4@8e968ff ->
> wakeup thread Thread-55, errback=passthrough -> release master lookup permit
> -> retry RPC after error -> passthrough -> wakeup thread Thread-55)
> at com.stumbleupon.async.Deferred.doJoin(Deferred.java:1161)
> at com.stumbleupon.async.Deferred.join(Deferred.java:1029)
> at org.kududb.client.KuduClient.openTable(KuduClient.java:181)
> at
> com.cloudera.impala.planner.KuduScanNode.computeScanRangeLocations(KuduScanNode.java:119)
> ... 10 more
> {noformat}
> Another entry in the log that seems quite relevant is related to a long
> thread creation time:
> {noformat}
> W0831 22:52:59.616951 8714 thread.cc:502] negotiator [worker] (thread pool)
> Time spent creating pthread: real 37.363suser 0.000s sys 0.000s
> {noformat}
> The stress test was run in an EC2 8 node cluster with CDH 5.9.1 installed.
> The latest Kudu was installed using packages. OS version is Ubuntu 14.04. The
> stress test run TPC-H queries in scale factor 10.
> Filing this JIRA for Impala now until the necessary changes in the stress
> test generator are checked in.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (IMPALA-4462) Kudu stress test causes Timed out errors
[
https://issues.apache.org/jira/browse/IMPALA-4462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Tauber-Marshall resolved IMPALA-4462.
Resolution: Cannot Reproduce
> Kudu stress test causes Timed out errors
>
>
> Key: IMPALA-4462
> URL: https://issues.apache.org/jira/browse/IMPALA-4462
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.8.0
>Reporter: Taras Bobrovytsky
>Assignee: Thomas Tauber-Marshall
>Priority: Major
> Labels: kudu, stress
>
> When the stress test runs with high concurrency on the EC2 cluster, sometimes
> queries fail with the following error message. This was observed in cases
> where we run only select queries and in cases where we also run CRUD queries.
> {code}
> status.cc:114] Unable to open scanner: Timed out: Client connection
> negotiation failed: client connection to 172.28.195.167:7050: Timeout
> exceeded waiting to connect: Timed out: Client connection negotiation failed:
> client connection to 172.28.194.188:7050: Timeout exceeded waiting to connect
> @ 0x11b3ba1 (unknown)
> @ 0x17d5de7 (unknown)
> @ 0x1771308 (unknown)
> @ 0x1771726 (unknown)
> @ 0x1773646 (unknown)
> @ 0x177350c (unknown)
> @ 0x177308d (unknown)
> @ 0x1772ec3 (unknown)
> @ 0x132d4f4 (unknown)
> @ 0x15d6273 (unknown)
> @ 0x15dd24c (unknown)
> @ 0x15dd18f (unknown)
> @ 0x15dd0ea (unknown)
> @ 0x1a2518a (unknown)
> @ 0x7faf323299d1 (unknown)
> @ 0x7faf320768fd (unknown)
> runtime-state.cc:208] Error from query 554b52e24918e84c:9be6b9c9:
> Unable to open scanner: Timed out: Client connection negotiation failed:
> client connection to 172.28.195.167:7050: Timeout exceeded waiting to
> connect: Timed out: Client connection negotiation failed: client connection
> to 172.28.194.188:7050: Timeout exceeded waiting to connect
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-6445) Whitespace should be stripped or detected in kudu master addresses metadata
[ https://issues.apache.org/jira/browse/IMPALA-6445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Tauber-Marshall reassigned IMPALA-6445: -- Assignee: (was: Thomas Tauber-Marshall) > Whitespace should be stripped or detected in kudu master addresses metadata > --- > > Key: IMPALA-6445 > URL: https://issues.apache.org/jira/browse/IMPALA-6445 > Project: IMPALA > Issue Type: Improvement > Components: Frontend >Reporter: Todd Lipcon >Priority: Major > > Currently the kudu master list metadata is split on ',' and directly fed > through to Kudu. This means that if a user specifies a list such as "m1, m2, > m3" with spaces after the commas, it will pass those hosts on to Kudu as > "m1", " m2", and " m3". Two of those three hostnames are of course invalid > and Kudu will only be able to connect when m1 is the active master. > We should either strip those spaces or detect this case and throw an error on > the bad metadata. (I prefer stripping) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6433) Add read support for PageHeaderV2 to the parquet scanner
[ https://issues.apache.org/jira/browse/IMPALA-6433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664252#comment-16664252 ] Tim Armstrong commented on IMPALA-6433: --- [~boroknagyz] do you know if this is a dependency for page skipping? I.e. are you likely to pick this up? > Add read support for PageHeaderV2 to the parquet scanner > > > Key: IMPALA-6433 > URL: https://issues.apache.org/jira/browse/IMPALA-6433 > Project: IMPALA > Issue Type: New Feature > Components: Backend >Affects Versions: Impala 2.11.0 >Reporter: Lars Volker >Priority: Major > Labels: parquet > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-6653) Unicode support for Kudu table names
[
https://issues.apache.org/jira/browse/IMPALA-6653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Tauber-Marshall reassigned IMPALA-6653:
--
Assignee: (was: Thomas Tauber-Marshall)
> Unicode support for Kudu table names
>
>
> Key: IMPALA-6653
> URL: https://issues.apache.org/jira/browse/IMPALA-6653
> Project: IMPALA
> Issue Type: Bug
>Reporter: Jim Halfpenny
>Priority: Major
>
> It is possible to create a Kudu table containing unicode characters in its in
> Impala by specifying the kudu.table_name attribute. When trying to select
> from this table you receive an error that the underlying table does not exist.
> The example below shows a table being created successfully, but failing on a
> select * statement.
> {{[jh-kafka-2:21000] > create table test2( a int primary key) stored as kudu
> TBLPROPERTIES('kudu.table_name' = 'impala::kudutest.');}}
> {{Query: create table test2( a int primary key) stored as kudu
> TBLPROPERTIES('kudu.table_name' = 'impala::kudutest.')}}
> {{WARNINGS: Unpartitioned Kudu tables are inefficient for large data
> sizes.}}{{Fetched 0 row(s) in 0.64s}}
> {{[jh-kafka-2:21000] > select * from test2;}}
> {{Query: select * from test2}}
> {{Query submitted at: 2018-03-13 08:23:29 (Coordinator:
> https://jh-kafka-2:25000)}}
> {{ERROR: AnalysisException: Failed to load metadata for table: 'test2'}}
> {{CAUSED BY: TableLoadingException: Error loading metadata for Kudu table
> impala::kudutest.}}
> {{CAUSED BY: ImpalaRuntimeException: Error opening Kudu table
> 'impala::kudutest.', Kudu error: The table does not exist: table_name:
> "impala::kudutest."}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-5505) Partition and sort Kudu UPDATE and DELETE
[ https://issues.apache.org/jira/browse/IMPALA-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Tauber-Marshall reassigned IMPALA-5505: -- Assignee: (was: Thomas Tauber-Marshall) > Partition and sort Kudu UPDATE and DELETE > - > > Key: IMPALA-5505 > URL: https://issues.apache.org/jira/browse/IMPALA-5505 > Project: IMPALA > Issue Type: Improvement > Components: Frontend >Affects Versions: Impala 2.10.0 >Reporter: Thomas Tauber-Marshall >Priority: Minor > Labels: kudu, performance > > A recent change (IMPALA-3742) added partitioning and sorting for Kudu > INSERTs. We should extend this to also cover UPDATE and DELETE. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-5505) Partition and sort Kudu UPDATE and DELETE
[ https://issues.apache.org/jira/browse/IMPALA-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Tauber-Marshall updated IMPALA-5505: --- Priority: Minor (was: Major) > Partition and sort Kudu UPDATE and DELETE > - > > Key: IMPALA-5505 > URL: https://issues.apache.org/jira/browse/IMPALA-5505 > Project: IMPALA > Issue Type: Improvement > Components: Frontend >Affects Versions: Impala 2.10.0 >Reporter: Thomas Tauber-Marshall >Assignee: Thomas Tauber-Marshall >Priority: Minor > Labels: kudu, performance > > A recent change (IMPALA-3742) added partitioning and sorting for Kudu > INSERTs. We should extend this to also cover UPDATE and DELETE. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-7655) Codegen output for conditional functions (if,isnull, coalesce) is very suboptimal
[
https://issues.apache.org/jira/browse/IMPALA-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong reassigned IMPALA-7655:
-
Assignee: Paul Rogers
> Codegen output for conditional functions (if,isnull, coalesce) is very
> suboptimal
> -
>
> Key: IMPALA-7655
> URL: https://issues.apache.org/jira/browse/IMPALA-7655
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Reporter: Tim Armstrong
>Assignee: Paul Rogers
>Priority: Major
> Labels: codegen, perf, performance
>
> https://gerrit.cloudera.org/#/c/11565/ provided a clue that an aggregation
> involving an if() function was very slow, 10x slower than the equivalent
> version using a case:
> {noformat}
> [localhost:21000] default> set num_nodes=1; set mt_dop=1; select count(case
> when l_orderkey is NULL then 1 else NULL end) from
> tpch10_parquet.lineitem;summary;
> NUM_NODES set to 1
> MT_DOP set to 1
> Query: select count(case when l_orderkey is NULL then 1 else NULL end) from
> tpch10_parquet.lineitem
> Query submitted at: 2018-10-04 11:17:31 (Coordinator:
> http://tarmstrong-box:25000)
> Query progress can be monitored at:
> http://tarmstrong-box:25000/query_plan?query_id=274b2a6f35cefe31:95a19642
> +--+
> | count(case when l_orderkey is null then 1 else null end) |
> +--+
> | 0|
> +--+
> Fetched 1 row(s) in 0.51s
> +--++--+--+++--+---+-+
> | Operator | #Hosts | Avg Time | Max Time | #Rows | Est. #Rows | Peak
> Mem | Est. Peak Mem | Detail |
> +--++--+--+++--+---+-+
> | 01:AGGREGATE | 1 | 44.03ms | 44.03ms | 1 | 1 | 25.00
> KB | 10.00 MB | FINALIZE|
> | 00:SCAN HDFS | 1 | 411.57ms | 411.57ms | 59.99M | -1 | 16.61
> MB | 88.00 MB | tpch10_parquet.lineitem |
> +--++--+--+++--+---+-+
> [localhost:21000] default> set num_nodes=1; set mt_dop=1; select
> count(if(l_orderkey is NULL, 1, NULL)) from tpch10_parquet.lineitem;summary;
> NUM_NODES set to 1
> MT_DOP set to 1
> Query: select count(if(l_orderkey is NULL, 1, NULL)) from
> tpch10_parquet.lineitem
> Query submitted at: 2018-10-04 11:23:07 (Coordinator:
> http://tarmstrong-box:25000)
> Query progress can be monitored at:
> http://tarmstrong-box:25000/query_plan?query_id=8e46ab1b84c4dbff:2786ca26
> ++
> | count(if(l_orderkey is null, 1, null)) |
> ++
> | 0 |
> ++
> Fetched 1 row(s) in 1.01s
> +--++--+--+++--+---+-+
> | Operator | #Hosts | Avg Time | Max Time | #Rows | Est. #Rows | Peak
> Mem | Est. Peak Mem | Detail |
> +--++--+--+++--+---+-+
> | 01:AGGREGATE | 1 | 422.07ms | 422.07ms | 1 | 1 | 25.00
> KB | 10.00 MB | FINALIZE|
> | 00:SCAN HDFS | 1 | 511.13ms | 511.13ms | 59.99M | -1 | 16.61
> MB | 88.00 MB | tpch10_parquet.lineitem |
> +--++--+--+++--+---+-+
> {noformat}
> It turns out that this is because we don't have good codegen support for
> ConditionalFunction, and just fall back to emitting a call to the interpreted
> path:
> https://github.com/apache/impala/blob/master/be/src/exprs/conditional-functions.cc#L28
> See CaseExpr for an example of much better codegen support:
> https://github.com/apache/impala/blob/master/be/src/exprs/case-expr.cc#L178
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7743) Impala 3.1 Doc: Document the option to load incremental statistics from catalog
[ https://issues.apache.org/jira/browse/IMPALA-7743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni updated IMPALA-7743: Description: https://gerrit.cloudera.org/#/c/11790/ > Impala 3.1 Doc: Document the option to load incremental statistics from > catalog > --- > > Key: IMPALA-7743 > URL: https://issues.apache.org/jira/browse/IMPALA-7743 > Project: IMPALA > Issue Type: Sub-task >Reporter: Alex Rodoni >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc > > https://gerrit.cloudera.org/#/c/11790/ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work stopped] (IMPALA-7356) Stress test for memory-based admission control
[ https://issues.apache.org/jira/browse/IMPALA-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-7356 stopped by Tim Armstrong. - > Stress test for memory-based admission control > -- > > Key: IMPALA-7356 > URL: https://issues.apache.org/jira/browse/IMPALA-7356 > Project: IMPALA > Issue Type: Test > Components: Infrastructure >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > Labels: admission-control > > We should extend the existing stress test to have a new mode designed to test > memory-based admission control, where the stress test framework does not try > to throttle memory consumption but instead relies on Impala doing so. > The required changes would be: > * A mode to disable throttling > * Options for stricter pass conditions - queries should not fail with OOM > even if the stress test tries to submit way too many queries. > * However AC queue timeouts may be ok. > * Investigation into the logic for choosing which query to run next and when > - does that need to change? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work stopped] (IMPALA-7648) Add tests for all cases where OOM is expected
[ https://issues.apache.org/jira/browse/IMPALA-7648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-7648 stopped by Tim Armstrong. - > Add tests for all cases where OOM is expected > - > > Key: IMPALA-7648 > URL: https://issues.apache.org/jira/browse/IMPALA-7648 > Project: IMPALA > Issue Type: Test > Components: Infrastructure >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > Labels: resource-management > > We should add tests for most or all of the subtasks of IMPALA-4834 that > exercise the expected OOM code path. I'll add some bullet points here to > track what the coverage is and what's missing. > * -Aggregations with large var-len string expressions- > * -Top-N with large window- > * -Kudu scans- : covered by kudu-scan-mem-usage.test > * -Nested loop join- : covered by single-node-nlj-exhaustive.test > * -Many duplicate keys on build side of hash join- > * -Large number of NULLS on build side of NAAJ- : covered by > spilling-naaj-no-deny-reservation.test > * -HDFS table partitioned insert- : covered insert-mem-limit.test > * -Large analytic window can't be spilled- > * Queries processing large strings (may need multiple tests to cover > different places). large_strings.test has some coverage > * Parquet files with large pages > * -Exchange uses a lot of memory- -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-5031) UBSAN clean and method for testing UBSAN cleanliness
[ https://issues.apache.org/jira/browse/IMPALA-5031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664125#comment-16664125 ] ASF subversion and git services commented on IMPALA-5031: - Commit 1d72c7584fd33746b17826c1dc54a1819fce9ec5 in impala's branch refs/heads/master from [~jbapple] [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=1d72c75 ] IMPALA-5031: Increase backend test timeout under UBSAN_FULL When codegen is run with ubsan, backend tests slow down dramatically. This patch increases the timeout to four hours in when UBSAN_FULL is the build type. This limit is approached by expr-test, which takes almost three hours under UBSAN_FULL. Change-Id: I3eee4c2b3affc9d65d86c043fcc382d7248adf3e Reviewed-on: http://gerrit.cloudera.org:8080/11764 Reviewed-by: Jim Apple Tested-by: Impala Public Jenkins > UBSAN clean and method for testing UBSAN cleanliness > > > Key: IMPALA-5031 > URL: https://issues.apache.org/jira/browse/IMPALA-5031 > Project: IMPALA > Issue Type: Task > Components: Backend, Infrastructure >Affects Versions: Impala 2.9.0 >Reporter: Jim Apple >Assignee: Jim Apple >Priority: Minor > > http://releases.llvm.org/3.8.0/tools/clang/docs/UndefinedBehaviorSanitizer.html > builds are supported after https://gerrit.cloudera.org/#/c/6186/, but > Impala's test suite triggers many errors under UBSAN. Those errors should be > fixed and then there should be a way to run the test suite under UBSAN and > fail if there were any errors detected. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Comment Edited] (IMPALA-7751) Kudu insert statement should push down range partition predicates
[
https://issues.apache.org/jira/browse/IMPALA-7751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664107#comment-16664107
]
Lars Volker edited comment on IMPALA-7751 at 10/25/18 6:09 PM:
---
[~stiga-huang] - Impala partitions the data by the partition columns of the
target table. You can use the "NOSHUFFLE" hint to prevent that:
https://impala.apache.org/docs/build/html/topics/impala_hints.html
was (Author: lv):
[~stiga-huang] - Impala partitions the data by the partition columns of the
target table. You can use the "NOSHUFFLE" hint to prevent that:
https://impala.apache.org/docs/build/html/topics/impala_hints.html
CC [~twmarshall]
> Kudu insert statement should push down range partition predicates
> -
>
> Key: IMPALA-7751
> URL: https://issues.apache.org/jira/browse/IMPALA-7751
> Project: IMPALA
> Issue Type: Improvement
>Reporter: Quanlong Huang
>Priority: Major
> Attachments: metrics1.tsv, metrics2.tsv, metrics3.tsv, profile.txt
>
>
> We have a job dumping newly added data in HDFS into Kudu table for good
> performance of point queries. Each day we create a new range partition in
> Kudu for the new data on this day. When we add more and more Kudu range
> partitions, we found performance degradation of this job.
> The root cause is, the insert statement for kudu does not leverage the
> partition predicates for kudu range partition keys, which causes skew on the
> insert nodes.
> How to reveal this:
> Step 1: Launch impala cluster with 3 nodes.
> Step 2: Create an HDFS table with more than 3 underlying files, thus will
> have more than 3 scan ranges
> {code:sql}
> create table default.metrics_tbl (
> source_id string,
> event_timestamp bigint,
> value double
> ) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' STORED AS TEXTFILE;
> {code}
> Upload the three attached tsv files into its directory and refresh this table
> in Impala.
> Step 3: Create a Kudu table with mix partitions containing 3 hash partitions
> and 3 range partitions.
> {code:sql}
> create table default.metrics_kudu_tbl (
> source_id string,
> event_timestamp bigint,
> value double,
> primary key(source_id, event_timestamp)
> ) partition by
> hash (source_id) PARTITIONS 3,
> range (event_timestamp) (
> partition 0 <= values < 1,
> partition 1 <= values < 2,
> partition 2 <= values < 3
> ) stored as kudu;
> {code}
> Step 4: Dump rows in HDFS table into Kudu giving partition predicates.
> {code:sql}
> insert into table metrics_kudu_tbl
> select source_id, event_timestamp, value from metrics_tbl
> where event_timestamp >= 1 and event_timestamp < 2;
> {code}
> Step 5: Looking into the profile, there're three fragment instances
> containing KuduTableSink but only one of them received and generated data.
> {code:java}
> Averaged Fragment F01:
> KuduTableSink:
> - TotalNumRows: 1.00K (1000)
> Fragment F01:
> Instance 6347506799a2966d:6e82f4920004
> KuduTableSink:
>- TotalNumRows: 3.00K (3000)
> Instance 6347506799a2966d:6e82f4920005
> KuduTableSink:
>- TotalNumRows: 0 (0)
> Instance 6347506799a2966d:6e82f4920003
> KuduTableSink:
>- TotalNumRows: 0 (0)
> {code}
> Thus, only one fragment instance of F01 is sorting and ingesting data into
> Impala.
> Generally, if there're N range partitions and all the inserted rows are
> belong to one range (supplied by the partition predicates in WHERE clause),
> only 1/N of the insert fragments are producing data.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
