[jira] [Created] (IMPALA-7874) The submitted SQL is stuck and cannot be quickly entered into the execution state.
nilone created IMPALA-7874:
--
Summary: The submitted SQL is stuck and cannot be quickly entered
into the execution state.
Key: IMPALA-7874
URL: https://issues.apache.org/jira/browse/IMPALA-7874
Project: IMPALA
Issue Type: Bug
Components: Backend, Catalog
Affects Versions: Impala 2.12.0, Impala 2.10.0
Reporter: nilone
Attachments: KB)~_BMDT1FM4{~U]YD7N[I.png, ZK$_]9Q8B)~JNZJXZL1CAUC.png,
Z}SUTV0%(O(I3GW$K5$LBF8.png, `[TL{)]P0R0)C]2D~D~4FQQ.png,
}`W}LD7]`IG$T4]O~MMREAE.png
Due to the recent tough problems encountered in impala, we may decide to
abandon its use in production, This is a great pity. !!! We have used impala
for several years and have tried to solve some problems, but this time we have
tried a lot of trials and it has no effect.
The system applies impala to ETL data processing. Currently, the main
problem is that the SQL task startup delay submitted by the client is in the
CREATED state on the Coordinator 25000 web, not the Running state. These tasks
cannot be quickly queried in the CM. As you can see from the list, it is
possible to wait for a while to run past, or it may stay stuck and die for a
long time. The situation may seem to be related to metadata loading, related to
the catalogd service. We have more than 300,000 tables and 8 million
partitions, and have tried to reduce some useless tables, but in the end did
not capture slow queries on the Mysql metabase. we've tried to restart Hive
,Namenode.
Later, we found the problem was similar to that described by IMPALA-5058.
We tried to upgrade the version of impala to 2.12.0 using overwritten file
mode. The service log became more detailed, but it was still not solved.
Tracking the log of catalogd found the following rule. Once "Remaining items in
queue: 0, Loads in progress: 1" appears in some worker threads, the problem
will appear. We tried to analyze the jstack information of catalogd and found
that a large number of threads are in Waiting state.
This is our cluster environment:
CDH 5.13.1 impala-2.10.0 & 2.12.0
300 datanodes, 80 impalad
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7871) Don't load Hive builtin jars for dataload
[ https://issues.apache.org/jira/browse/IMPALA-7871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe McDonnell resolved IMPALA-7871. --- Resolution: Fixed Fix Version/s: Impala 3.2.0 > Don't load Hive builtin jars for dataload > - > > Key: IMPALA-7871 > URL: https://issues.apache.org/jira/browse/IMPALA-7871 > Project: IMPALA > Issue Type: Improvement > Components: Infrastructure >Affects Versions: Impala 3.1.0 >Reporter: Joe McDonnell >Assignee: Joe McDonnell >Priority: Major > Fix For: Impala 3.2.0 > > > One step in dataload is "Loading Hive Builtins", which copies a large number > of jars into HDFS (or whatever storage). This step takes a couple minutes on > HDFS dataload and 8 minutes on S3. Despite its name, I can't find any > indication that Hive or anything else uses these jars. Dataload and core > tests run fine without it. S3 can load data without it. There's no indication > that this is needed. > Unless we find something using these jars, we should remove this step. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7873) TestExchangeMemUsage.test_exchange_mem_usage_scaling doesn't hit the memory limit
[
https://issues.apache.org/jira/browse/IMPALA-7873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692521#comment-16692521
]
Pooja Nilangekar commented on IMPALA-7873:
--
Yes, I hit the issue while testing IMPALA-7367. I think the limit I set may not
be accurate. I will take a look at this.
> TestExchangeMemUsage.test_exchange_mem_usage_scaling doesn't hit the memory
> limit
> -
>
> Key: IMPALA-7873
> URL: https://issues.apache.org/jira/browse/IMPALA-7873
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Joe McDonnell
>Priority: Blocker
> Labels: broken-build
>
> This is failing to hit a memory exceeded on the last two core runs:
> {noformat}
> query_test/test_mem_usage_scaling.py:386: in test_exchange_mem_usage_scaling
> self.run_test_case('QueryTest/exchange-mem-scaling', vector)
> common/impala_test_suite.py:482: in run_test_case
> assert False, "Expected exception: %s" % expected_str
> E AssertionError: Expected exception: Memory limit exceeded{noformat}
> It might be that the limit needs to be adjusted.
> There were two changes since the last successful run: IMPALA-7367
> (2a4835cfba7597362cc1e72e21315868c5c75d0a) and IMPALA-5031
> (53ce6bb571cd9ae07ba5255197d35aa852a6f97c)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-7873) TestExchangeMemUsage.test_exchange_mem_usage_scaling doesn't hit the memory limit
[
https://issues.apache.org/jira/browse/IMPALA-7873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pooja Nilangekar reassigned IMPALA-7873:
Assignee: Pooja Nilangekar
> TestExchangeMemUsage.test_exchange_mem_usage_scaling doesn't hit the memory
> limit
> -
>
> Key: IMPALA-7873
> URL: https://issues.apache.org/jira/browse/IMPALA-7873
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Joe McDonnell
>Assignee: Pooja Nilangekar
>Priority: Blocker
> Labels: broken-build
>
> This is failing to hit a memory exceeded on the last two core runs:
> {noformat}
> query_test/test_mem_usage_scaling.py:386: in test_exchange_mem_usage_scaling
> self.run_test_case('QueryTest/exchange-mem-scaling', vector)
> common/impala_test_suite.py:482: in run_test_case
> assert False, "Expected exception: %s" % expected_str
> E AssertionError: Expected exception: Memory limit exceeded{noformat}
> It might be that the limit needs to be adjusted.
> There were two changes since the last successful run: IMPALA-7367
> (2a4835cfba7597362cc1e72e21315868c5c75d0a) and IMPALA-5031
> (53ce6bb571cd9ae07ba5255197d35aa852a6f97c)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Closed] (IMPALA-7861) ABFS docs: SSL is now enabled by default regardless of URI scheme
[ https://issues.apache.org/jira/browse/IMPALA-7861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni closed IMPALA-7861. --- Resolution: Fixed Fix Version/s: Impala 3.1.0 > ABFS docs: SSL is now enabled by default regardless of URI scheme > - > > Key: IMPALA-7861 > URL: https://issues.apache.org/jira/browse/IMPALA-7861 > Project: IMPALA > Issue Type: Bug >Reporter: Sean Mackrory >Assignee: Alex Rodoni >Priority: Trivial > Labels: future_release_doc > Fix For: Impala 3.1.0 > > > When it was originally submitted to Hadoop, the ABFS driver disabled TLS when > using the URI scheme "abfs", and enabled TLS when using the URI scheme > "abfss". This was reflected in the documentation originally submitted for > ABFS: > https://github.com/apache/impala/commit/5baa289fd039d339c63d5c475645e69fe0c6b8df. > We should update that to reflect that TLS is enabled with either URI scheme > unless you disable TLS in configuration by setting > fs.azure.always.use.https=false in the configuration. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7873) TestExchangeMemUsage.test_exchange_mem_usage_scaling doesn't hit the memory limit
[
https://issues.apache.org/jira/browse/IMPALA-7873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692519#comment-16692519
]
Joe McDonnell commented on IMPALA-7873:
---
[~poojanilangekar], can you see if this is related to IMPALA-7367? It might
have changed the memory requirements of this statement.
> TestExchangeMemUsage.test_exchange_mem_usage_scaling doesn't hit the memory
> limit
> -
>
> Key: IMPALA-7873
> URL: https://issues.apache.org/jira/browse/IMPALA-7873
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Joe McDonnell
>Priority: Blocker
> Labels: broken-build
>
> This is failing to hit a memory exceeded on the last two core runs:
> {noformat}
> query_test/test_mem_usage_scaling.py:386: in test_exchange_mem_usage_scaling
> self.run_test_case('QueryTest/exchange-mem-scaling', vector)
> common/impala_test_suite.py:482: in run_test_case
> assert False, "Expected exception: %s" % expected_str
> E AssertionError: Expected exception: Memory limit exceeded{noformat}
> It might be that the limit needs to be adjusted.
> There were two changes since the last successful run: IMPALA-7367
> (2a4835cfba7597362cc1e72e21315868c5c75d0a) and IMPALA-5031
> (53ce6bb571cd9ae07ba5255197d35aa852a6f97c)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7873) TestExchangeMemUsage.test_exchange_mem_usage_scaling doesn't hit the memory limit
[
https://issues.apache.org/jira/browse/IMPALA-7873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell updated IMPALA-7873:
--
Description:
This is failing to hit a memory exceeded on the last two core runs:
{noformat}
query_test/test_mem_usage_scaling.py:386: in test_exchange_mem_usage_scaling
self.run_test_case('QueryTest/exchange-mem-scaling', vector)
common/impala_test_suite.py:482: in run_test_case
assert False, "Expected exception: %s" % expected_str
E AssertionError: Expected exception: Memory limit exceeded{noformat}
It might be that the limit needs to be adjusted.
There were two changes since the last successful run: IMPALA-7367
(2a4835cfba7597362cc1e72e21315868c5c75d0a) and IMPALA-5031
(53ce6bb571cd9ae07ba5255197d35aa852a6f97c)
was:
This is failing to hit a memory exceeded on the last two core runs:
{noformat}
query_test/test_mem_usage_scaling.py:386: in test_exchange_mem_usage_scaling
self.run_test_case('QueryTest/exchange-mem-scaling', vector)
common/impala_test_suite.py:482: in run_test_case
assert False, "Expected exception: %s" % expected_str
E AssertionError: Expected exception: Memory limit exceeded{noformat}
It might be that the limit needs to be adjusted.
There were two changes since the last successful run: IMPALA-7367 and
IMPALA-5031
> TestExchangeMemUsage.test_exchange_mem_usage_scaling doesn't hit the memory
> limit
> -
>
> Key: IMPALA-7873
> URL: https://issues.apache.org/jira/browse/IMPALA-7873
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Joe McDonnell
>Priority: Blocker
> Labels: broken-build
>
> This is failing to hit a memory exceeded on the last two core runs:
> {noformat}
> query_test/test_mem_usage_scaling.py:386: in test_exchange_mem_usage_scaling
> self.run_test_case('QueryTest/exchange-mem-scaling', vector)
> common/impala_test_suite.py:482: in run_test_case
> assert False, "Expected exception: %s" % expected_str
> E AssertionError: Expected exception: Memory limit exceeded{noformat}
> It might be that the limit needs to be adjusted.
> There were two changes since the last successful run: IMPALA-7367
> (2a4835cfba7597362cc1e72e21315868c5c75d0a) and IMPALA-5031
> (53ce6bb571cd9ae07ba5255197d35aa852a6f97c)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7870) TestAutomaticCatalogInvalidation.test_v1_catalog intermittently fails
[
https://issues.apache.org/jira/browse/IMPALA-7870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692517#comment-16692517
]
Joe McDonnell commented on IMPALA-7870:
---
[~tianyiwang] Since you've seen this before, assigning to you.
> TestAutomaticCatalogInvalidation.test_v1_catalog intermittently fails
> -
>
> Key: IMPALA-7870
> URL: https://issues.apache.org/jira/browse/IMPALA-7870
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 3.1.0
>Reporter: Joe McDonnell
>Assignee: Tianyi Wang
>Priority: Critical
> Labels: broken-build
>
> A couple test runs have seen
> custom_cluster.test_automatic_invalidation.TestAutomaticCatalogInvalidation.test_v1_catalog()
> take slightly longer than the timeout:
> {noformat}
> custom_cluster/test_automatic_invalidation.py:69: in test_v1_catalog
> self._run_test(cursor)
> custom_cluster/test_automatic_invalidation.py:64: in _run_test
> assert time.time() < max_wait_time
> E assert 1542544633.922359 < 1542544633.68588
> E+ where 1542544633.922359 = ()
> E+where = time.time{noformat}
> and
> {noformat}
> custom_cluster/test_automatic_invalidation.py:69: in test_v1_catalog
> self._run_test(cursor)
> custom_cluster/test_automatic_invalidation.py:64: in _run_test
> assert time.time() < max_wait_time
> E assert 1542372031.541345 < 1542372031.303814
> E+ where 1542372031.541345 = ()
> E+where = time.time{noformat}
> This has been seen twice on the core job.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-7870) TestAutomaticCatalogInvalidation.test_v1_catalog intermittently fails
[
https://issues.apache.org/jira/browse/IMPALA-7870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell reassigned IMPALA-7870:
-
Assignee: Tianyi Wang
> TestAutomaticCatalogInvalidation.test_v1_catalog intermittently fails
> -
>
> Key: IMPALA-7870
> URL: https://issues.apache.org/jira/browse/IMPALA-7870
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 3.1.0
>Reporter: Joe McDonnell
>Assignee: Tianyi Wang
>Priority: Critical
> Labels: broken-build
>
> A couple test runs have seen
> custom_cluster.test_automatic_invalidation.TestAutomaticCatalogInvalidation.test_v1_catalog()
> take slightly longer than the timeout:
> {noformat}
> custom_cluster/test_automatic_invalidation.py:69: in test_v1_catalog
> self._run_test(cursor)
> custom_cluster/test_automatic_invalidation.py:64: in _run_test
> assert time.time() < max_wait_time
> E assert 1542544633.922359 < 1542544633.68588
> E+ where 1542544633.922359 = ()
> E+where = time.time{noformat}
> and
> {noformat}
> custom_cluster/test_automatic_invalidation.py:69: in test_v1_catalog
> self._run_test(cursor)
> custom_cluster/test_automatic_invalidation.py:64: in _run_test
> assert time.time() < max_wait_time
> E assert 1542372031.541345 < 1542372031.303814
> E+ where 1542372031.541345 = ()
> E+where = time.time{noformat}
> This has been seen twice on the core job.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7873) TestExchangeMemUsage.test_exchange_mem_usage_scaling doesn't hit the memory limit
Joe McDonnell created IMPALA-7873:
-
Summary: TestExchangeMemUsage.test_exchange_mem_usage_scaling
doesn't hit the memory limit
Key: IMPALA-7873
URL: https://issues.apache.org/jira/browse/IMPALA-7873
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 3.2.0
Reporter: Joe McDonnell
This is failing to hit a memory exceeded on the last two core runs:
{noformat}
query_test/test_mem_usage_scaling.py:386: in test_exchange_mem_usage_scaling
self.run_test_case('QueryTest/exchange-mem-scaling', vector)
common/impala_test_suite.py:482: in run_test_case
assert False, "Expected exception: %s" % expected_str
E AssertionError: Expected exception: Memory limit exceeded{noformat}
It might be that the limit needs to be adjusted.
There were two changes since the last successful run: IMPALA-7367 and
IMPALA-5031
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7808) Refactor SelectStmt analyzer for easier debugging
[
https://issues.apache.org/jira/browse/IMPALA-7808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Rogers updated IMPALA-7808:
Summary: Refactor SelectStmt analyzer for easier debugging (was: Refactor
Analyzer for easier debugging)
> Refactor SelectStmt analyzer for easier debugging
> -
>
> Key: IMPALA-7808
> URL: https://issues.apache.org/jira/browse/IMPALA-7808
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
>Affects Versions: Impala 3.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> The analysis steps in {{SelectStmt}} and {{AnalysisContext}} are large and
> cumbersome. There is ample evidence in the literature that simpler, smaller
> functions are easier to understand and debug than larger, more complex
> functions. This ticket requests breaking up the large functions in these two
> cases into smaller, easier-understood units in preparation for tracking down
> issues related to missing rewrites of the {{WHERE}} and {{GROUP BY}} clauses.
> One might argue that large functions perform better by eliminating
> unnecessary function calls. However, the planner is not performance
> sensitive, and the dozen extra calls that this change introduce will not
> change performance given the thousands of calls already made.
> Experience has shown that the JIT compiler in the JVM actually does a better
> job optimizing smaller functions, and gives up when functions get to large.
> So, by creating smaller functions, we may actually allow the JIT compiler to
> generate better code.
> And, this refactoring is in support of a possible outcome that the planner
> can handle rewrites without making multiple passes through the analyzer: that
> savings will far outweigh the few extra calls this change introduces.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-539) Impala should gather final runtime profile from fragments for aborted/cancelled query
[
https://issues.apache.org/jira/browse/IMPALA-539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692475#comment-16692475
]
Michael Ho commented on IMPALA-539:
---
Ideally, we should always wait for all backend completion even in the case of
cancellation below. Of course, this may be problematic if a backend is
unresponsive for whatever reason so we can wait with a timeout once the
cancellation RPCs have been issued.
{noformat}
void Coordinator::HandleExecStateTransition(
const ExecState old_state, const ExecState new_state) {
static const unordered_map exec_state_to_event{
{ExecState::EXECUTING,"Executing"},
{ExecState::RETURNED_RESULTS, "Last row fetched"},
{ExecState::CANCELLED,"Execution cancelled"},
{ExecState::ERROR,"Execution error"}};
if (old_state == new_state) return;
// Once we enter a terminal state, we stay there, guaranteeing this code runs
only once.
DCHECK_EQ(old_state, ExecState::EXECUTING);
// Should never transition to the initial state.
DCHECK_NE(new_state, ExecState::EXECUTING);
// Can't transition until the exec RPCs are no longer in progress. Otherwise,
a
// cancel RPC could be missed, and resources freed before a backend has had a
chance
// to take a resource reference.
DCHECK(exec_rpcs_complete_barrier_ != nullptr &&
exec_rpcs_complete_barrier_->pending() <= 0) << "exec rpcs not completed";
query_events_->MarkEvent(exec_state_to_event.at(new_state));
// This thread won the race to transitioning into a terminal state - terminate
// execution and release resources.
ReleaseExecResources();
if (new_state == ExecState::RETURNED_RESULTS) {
// TODO: IMPALA-6984: cancel all backends in this case too.
WaitForBackends(); <<-
} else {
CancelBackends(); <<- should wait for backend completion too in this
case.
}
ReleaseAdmissionControlResources();
// Can compute summary only after we stop accepting reports from the
backends. Both
// WaitForBackends() and CancelBackends() ensures that.
// TODO: should move this off of the query execution path?
ComputeQuerySummary();
{noformat}
> Impala should gather final runtime profile from fragments for
> aborted/cancelled query
> -
>
> Key: IMPALA-539
> URL: https://issues.apache.org/jira/browse/IMPALA-539
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Affects Versions: Impala 1.1
>Reporter: Alan Choi
>Priority: Minor
> Labels: query-lifecycle
>
> For cancelled/aborted queries, the final runtime profile from plan fragments
> are not recorded and reported by the coordinator. For short running queries,
> we won't see any profile at all because no intermediate runtime profile has
> been sent.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7551) Inaccurate timeline for "Rows Available"
[ https://issues.apache.org/jira/browse/IMPALA-7551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong updated IMPALA-7551: -- Labels: observability query-lifecycle (was: query-lifecycle) > Inaccurate timeline for "Rows Available" > - > > Key: IMPALA-7551 > URL: https://issues.apache.org/jira/browse/IMPALA-7551 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 3.1.0 >Reporter: Pooja Nilangekar >Priority: Major > Labels: observability, query-lifecycle > > While debugging IMPALA-6932, it was noticed that the "Rows Available" metric > in the query profile was a short duration (~ 1 second) for a long running > limit 1 query (~ 1 hour). > Currently, it tracks when Open() from the top-most node in the plan returns, > not when the first row is actually produced. This can be misleading. A better > timeline would be to return true when the first non-empty batch was added to > the PlanRootSink. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-5031) UBSAN clean and method for testing UBSAN cleanliness
[ https://issues.apache.org/jira/browse/IMPALA-5031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692453#comment-16692453 ] ASF subversion and git services commented on IMPALA-5031: - Commit 48d1d2d7421d668f8dd5142b05395e2b163f621b in impala's branch refs/heads/master from [~jbapple] [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=48d1d2d ] IMPALA-5031: method calls on NULL are not UBSAN-clean According to [expr.post] in the C++14 standard, a call to a member function like a->b() is interpreted as (a->b)(). In other words, the dereferencing is done separately from the call. This makes calling member functions on nullptr undefined behavior, since the dereference invokes undefined behavior. Change-Id: I8b38cb1ebba02fc163534ffcc95e4ebe41cbb115 Reviewed-on: http://gerrit.cloudera.org:8080/11950 Reviewed-by: Jim Apple Tested-by: Impala Public Jenkins > UBSAN clean and method for testing UBSAN cleanliness > > > Key: IMPALA-5031 > URL: https://issues.apache.org/jira/browse/IMPALA-5031 > Project: IMPALA > Issue Type: Task > Components: Backend, Infrastructure >Affects Versions: Impala 2.9.0 >Reporter: Jim Apple >Assignee: Jim Apple >Priority: Minor > > http://releases.llvm.org/3.8.0/tools/clang/docs/UndefinedBehaviorSanitizer.html > builds are supported after https://gerrit.cloudera.org/#/c/6186/, but > Impala's test suite triggers many errors under UBSAN. Those errors should be > fixed and then there should be a way to run the test suite under UBSAN and > fail if there were any errors detected. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-5031) UBSAN clean and method for testing UBSAN cleanliness
[
https://issues.apache.org/jira/browse/IMPALA-5031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692455#comment-16692455
]
ASF subversion and git services commented on IMPALA-5031:
-
Commit 8f151d0e63aa5f3548e283fadcaf801456cb3354 in impala's branch
refs/heads/master from [~jbapple]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=8f151d0 ]
IMPALA-5031: Zero-length arrays are undefined behavior
This patch prevents the creation of a zero-length array. This is
illegal under paragraph 5 of §6.7.5.2 ("Array declarators") of the C99
standard, which reads:
If the size is an expression that is not an integer constant
expression: if it occurs in a declaration at function prototype
scope, it is treated as if it were replaced by *; otherwise, each
time it is evaluated it shall have a value greater than zero.
Variable-length arrays are not part of C++14, but they are a common
compiler extension and are available in both clang++ and g++. That the
semantics of missing features are the same is implied by [intro.scope]
in the C++14 standard, which reads:
C++ is a general purpose programming language based on the C
programming language as described in ISO/IEC 9899:1999 Programming
languages — C (hereinafter referred to as the C standard).
Change-Id: I93f7d0b0506e4b6a2ff3303477d887a428431f96
Reviewed-on: http://gerrit.cloudera.org:8080/11811
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> UBSAN clean and method for testing UBSAN cleanliness
>
>
> Key: IMPALA-5031
> URL: https://issues.apache.org/jira/browse/IMPALA-5031
> Project: IMPALA
> Issue Type: Task
> Components: Backend, Infrastructure
>Affects Versions: Impala 2.9.0
>Reporter: Jim Apple
>Assignee: Jim Apple
>Priority: Minor
>
> http://releases.llvm.org/3.8.0/tools/clang/docs/UndefinedBehaviorSanitizer.html
> builds are supported after https://gerrit.cloudera.org/#/c/6186/, but
> Impala's test suite triggers many errors under UBSAN. Those errors should be
> fixed and then there should be a way to run the test suite under UBSAN and
> fail if there were any errors detected.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7804) Various scanner tests intermittently failing on S3 on different runs
[
https://issues.apache.org/jira/browse/IMPALA-7804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692449#comment-16692449
]
Joe McDonnell commented on IMPALA-7804:
---
Here's a theory about IMPALA-6709:
Before the change, this is the order:
# Create table
# Copy file to table's directory in HDFS
# Query it
After the change, this is the order:
# Copy file to the database's directory in HDFS (not the table's dir)
# Create table
# LOAD DATA INPATH moves the file from the database directory to the table
directory
# Query it
The difference has to do with metadata. Before IMPALA-6709, we get the metadata
when we query the table (#3) after we've copied the file (#2). This gives the
filesystem time to get consistent, because metadata takes a couple seconds.
After IMPALA-6709, we get the metadata in the LOAD step before we copy the
file. This means that when we immediately query it, we don't need to load
metadata. So, very little time passes after we copy the file before we query
it, and the filesystem has less time to get consistent.
> Various scanner tests intermittently failing on S3 on different runs
>
>
> Key: IMPALA-7804
> URL: https://issues.apache.org/jira/browse/IMPALA-7804
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: David Knupp
>Priority: Blocker
> Labels: S3
>
> The failures have to do with getting AWS client credentials.
> *query_test/test_scanners.py:696: in test_decimal_encodings*
> _Stacktrace_
> {noformat}
> query_test/test_scanners.py:696: in test_decimal_encodings
> self.run_test_case('QueryTest/parquet-decimal-formats', vector,
> unique_database)
> common/impala_test_suite.py:496: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:358: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:438: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:260: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E -255.00,-255.00,-255.00 == -255.00,-255.00,-255.00
> E -255.00,-255.00,-255.00 != -65535.00,-65535.00,-65535.00
> E -65535.00,-65535.00,-65535.00 != -999.99,-999.99,-999.99
> E -65535.00,-65535.00,-65535.00 !=
> 0.00,-.99,-.99
> E -999.99,-999.99,-999.99 != 0.00,0.00,0.00
> E -999.99,-999.99,-999.99 !=
> 0.00,.99,.99
> E 0.00,-.99,-.99 !=
> 255.00,255.00,255.00
> E 0.00,-.99,-.99 !=
> 65535.00,65535.00,65535.00
> E 0.00,0.00,0.00 != 999.99,999.99,999.99
> E 0.00,0.00,0.00 != None
> E 0.00,.99,.99 != None
> E 0.00,.99,.99 != None
> E 255.00,255.00,255.00 != None
> E 255.00,255.00,255.00 != None
> E 65535.00,65535.00,65535.00 != None
> E 65535.00,65535.00,65535.00 != None
> E 999.99,999.99,999.99 != None
> E 999.99,999.99,999.99 != None
> E Number of rows returned (expected vs actual): 18 != 9
> {noformat}
> _Standard Error_
> {noformat}
> SET sync_ddl=False;
> -- executing against localhost:21000
> DROP DATABASE IF EXISTS `test_huge_num_rows_76a09ef1` CASCADE;
> -- 2018-11-01 09:42:41,140 INFO MainThread: Started query
> 4c4bc0e7b69d7641:130ffe73
> SET sync_ddl=False;
> -- executing against localhost:21000
> CREATE DATABASE `test_huge_num_rows_76a09ef1`;
> -- 2018-11-01 09:42:42,402 INFO MainThread: Started query
> e34d714d6a62cba1:2a8544d0
> -- 2018-11-01 09:42:42,405 INFO MainThread: Created database
> "test_huge_num_rows_76a09ef1" for test ID
> "query_test/test_scanners.py::TestParquet::()::test_huge_num_rows[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': True,
> 'abort_on_error': 1, 'debug_action':
> '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0',
> 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]"
> 18/11/01 09:42:43 DEBUG s3a.S3AFileSystem: Initializing S3AFileSystem for
> impala-test-uswest2-1
> 18/11/01 09:42:43 DEBUG s3a.S3AUtils: Propagating entries under
> fs.s3a.bucket.impala-test-uswest2-1.
> 18/11/01 09:42:43 WARN impl.MetricsConfig: Cannot locate configuration: tried
>
[jira] [Resolved] (IMPALA-7857) Log more information about statestore failure detector
[ https://issues.apache.org/jira/browse/IMPALA-7857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong resolved IMPALA-7857. --- Resolution: Fixed Fix Version/s: Impala 3.2.0 > Log more information about statestore failure detector > -- > > Key: IMPALA-7857 > URL: https://issues.apache.org/jira/browse/IMPALA-7857 > Project: IMPALA > Issue Type: Bug > Components: Distributed Exec >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > Labels: statestore, supportability > Fix For: Impala 3.2.0 > > > For debugging heartbeat failures (or non-failures) it would be useful to log > enough information to infer the current state of the failure detector from > logs. Specifically: > * Upon a failure, we should log the number of consecutive failures according > to the failure detector. And also maybe how many failures remain until it's > considered to be failed. > * We should log when the failure count is reset to 0 by a successful > heartbeat. > Currently if there are occasional failures it's hard to tell with certainty > whether it was reset correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7857) Log more information about statestore failure detector
[ https://issues.apache.org/jira/browse/IMPALA-7857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692401#comment-16692401 ] ASF subversion and git services commented on IMPALA-7857: - Commit 93a0ce857f181f2fe4248252428fc2adfdf1bdb7 in impala's branch refs/heads/master from [~tarmstr...@cloudera.com] [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=93a0ce8 ] IMPALA-7857: log more information about statestore failure detection This adds a couple of log messages for state transitions in the statestore's failure detector. Testing: Ran test_statestore.py and checked for presence of new log messages. Added a new tests to test_statestore that exercises handling of intermittent heartbeat failures (required to produce one of the new log messages). Change-Id: Ie6ff85bee117000e4434dcffd3d1680a79905f14 Reviewed-on: http://gerrit.cloudera.org:8080/11937 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Log more information about statestore failure detector > -- > > Key: IMPALA-7857 > URL: https://issues.apache.org/jira/browse/IMPALA-7857 > Project: IMPALA > Issue Type: Bug > Components: Distributed Exec >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > Labels: statestore, supportability > > For debugging heartbeat failures (or non-failures) it would be useful to log > enough information to infer the current state of the failure detector from > logs. Specifically: > * Upon a failure, we should log the number of consecutive failures according > to the failure detector. And also maybe how many failures remain until it's > considered to be failed. > * We should log when the failure count is reset to 0 by a successful > heartbeat. > Currently if there are occasional failures it's hard to tell with certainty > whether it was reset correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Comment Edited] (IMPALA-6984) Coordinator should cancel backends when returning EOS
[
https://issues.apache.org/jira/browse/IMPALA-6984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692357#comment-16692357
]
Michael Ho edited comment on IMPALA-6984 at 11/19/18 10:19 PM:
---
Now that IMPALA-4063 is fixed, we should have a guarantee that the a final
profile will be sent even when the fragment instances are cancelled. This is
guaranteed as cancellation won't have any effect until all fragment instances
on a backend have finished calling {{FIS:Prepare()}} so the query state thread
will definitely send one report upon completion of all fragment instances.
However, the logic in {{Coordinator::BackendState::Done()}} is preventing the
final profile from being applied as {{status_}} is already set to
{{CANCELLED}}. It may need to be fixed up in order for the coordinator to
cleanly cancel the fragment instances on EOS.
{noformat}
inline bool Coordinator::BackendState::IsDone() const {
return num_remaining_instances_ == 0 || !status_.ok();
}
{noformat}
was (Author: kwho):
Now that IMPALA-4063 is fixed, we should have a guarantee that the a final
profile will be sent even when the fragment instances are cancelled. This is
guaranteed as cancellation won't have any effect until the FIS on a backend has
passed the preparation stage so the query state thread will definitely send one
report upon completion of all fragment instancse.
However, the logic in {{Coordinator::BackendState::Done()}} is preventing the
final profile from being applied as {{status_}} is already set to
{{CANCELLED}}. It may need to be fixed up in order for the coordinator to
cleanly cancel the fragment instances on EOS.
{noformat}
inline bool Coordinator::BackendState::IsDone() const {
return num_remaining_instances_ == 0 || !status_.ok();
}
{noformat}
> Coordinator should cancel backends when returning EOS
> -
>
> Key: IMPALA-6984
> URL: https://issues.apache.org/jira/browse/IMPALA-6984
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Affects Versions: Impala 3.0
>Reporter: Dan Hecht
>Priority: Major
> Labels: query-lifecycle
>
> Currently, the Coordinator waits for backends rather than proactively
> cancelling them in the case of hitting EOS. There's a tangled mess that makes
> it tricky to proactively cancel the backends related to how
> {{Coordinator::ComputeQuerySummary()}} works – we can't update the summary
> until the profiles are no longer changing (which also makes sense given that
> we want the exec summary to be consistent with the final profile). But we
> current tie together the FIS status and the profile, and cancellation of
> backends causes the FIS to return CANCELLED, which then means that the
> remaining FIS on that backend won't produce a final profile.
> With the rework of the protocol for IMPALA-2990 we should make it possible to
> sort this out such that a final profile can be requested regardless of how a
> FIS ends execution.
> This also relates to IMPALA-5783.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7606) Time based auto invalidation not working
[ https://issues.apache.org/jira/browse/IMPALA-7606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tianyi Wang resolved IMPALA-7606. - Resolution: Fixed Fix Version/s: Impala 3.1.0 IMPALA-7606: Fix IllegalStateException in CatalogTableInvalidator CatalogdTableInvalidator detects if a table is in a normal state using Table.isLoaded() function. This is wrong because if there is an error during the loading of a table, isLoaded() returns true. This patch checks if the table is an IncompleteTable instead. Also fixed a bug in tryInstallGcListener(). A test is added to test the memory-based invalidation. Change-Id: If938a40434b00af516445152f88832ef55d0d0ce Reviewed-on: http://gerrit.cloudera.org:8080/11512 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Time based auto invalidation not working > > > Key: IMPALA-7606 > URL: https://issues.apache.org/jira/browse/IMPALA-7606 > Project: IMPALA > Issue Type: Bug >Affects Versions: Impala 3.1.0 >Reporter: Rituparna Agrawal >Assignee: Tianyi Wang >Priority: Major > Fix For: Impala 3.1.0 > > Attachments: catalogd.INFO > > > Set the invalidate_tables_timeout_s to 60 on both catalogd and all impalads. > Have tables populated. Catalogd is keeps throwing this exception. Tables are > not getting invalidated. > No other logs seen on impalad from CatalogTableInvalidator > [10:50 AM] Parna Agrawal: > [10:50 AM] Parna Agrawal: W0921 10:50:19.465893 13428 > CatalogdTableInvalidator.java:284] Unexpected exception thrown while > attempting to automatically invalidate tables. Will retry in 5 seconds. > Java exception follows: > java.lang.IllegalStateException > at com.google.common.base.Preconditions.checkState(Preconditions.java:129) > at org.apache.impala.catalog.Table.getLastUsedTime(Table.java:174) > at > org.apache.impala.catalog.CatalogdTableInvalidator.invalidateOlderThan(CatalogdTableInvalidator.java:229) > at > org.apache.impala.catalog.CatalogdTableInvalidator.access$900(CatalogdTableInvalidator.java:51) > at > org.apache.impala.catalog.CatalogdTableInvalidator$DaemonThread.run(CatalogdTableInvalidator.java:275) > at java.lang.Thread.run(Thread.java:748) > W0921 10:50:24.467514 13428 CatalogdTableInvalidator.java:284] Unexpected > exception thrown while attempting to automatically invalidate tables. Will > retry in 5 seconds. > Java exception follows: > java.lang.IllegalStateException > at com.google.common.base.Preconditions.checkState(Preconditions.java:129) > at org.apache.impala.catalog.Table.getLastUsedTime(Table.java:174) > at > org.apache.impala.catalog.CatalogdTableInvalidator.invalidateOlderThan(CatalogdTableInvalidator.java:229) > at > org.apache.impala.catalog.CatalogdTableInvalidator.access$900(CatalogdTableInvalidator.java:51) > at > org.apache.impala.catalog.CatalogdTableInvalidator$DaemonThread.run(CatalogdTableInvalidator.java:275) > at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6984) Coordinator should cancel backends when returning EOS
[
https://issues.apache.org/jira/browse/IMPALA-6984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692357#comment-16692357
]
Michael Ho commented on IMPALA-6984:
Now that IMPALA-4063 is fixed, we should have a guarantee that the a final
profile will be sent even when the fragment instances are cancelled. This is
guaranteed as cancellation won't have any effect until the FIS on a backend has
passed the preparation stage so the query state thread will definitely send one
report upon completion of all fragment instancse.
However, the logic in {{Coordinator::BackendState::Done()}} is preventing the
final profile from being applied as {{status_}} is already set to
{{CANCELLED}}. It may need to be fixed up in order for the coordinator to
cleanly cancel the fragment instances on EOS.
{noformat}
inline bool Coordinator::BackendState::IsDone() const {
return num_remaining_instances_ == 0 || !status_.ok();
}
{noformat}
> Coordinator should cancel backends when returning EOS
> -
>
> Key: IMPALA-6984
> URL: https://issues.apache.org/jira/browse/IMPALA-6984
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Affects Versions: Impala 3.0
>Reporter: Dan Hecht
>Priority: Major
> Labels: query-lifecycle
>
> Currently, the Coordinator waits for backends rather than proactively
> cancelling them in the case of hitting EOS. There's a tangled mess that makes
> it tricky to proactively cancel the backends related to how
> {{Coordinator::ComputeQuerySummary()}} works – we can't update the summary
> until the profiles are no longer changing (which also makes sense given that
> we want the exec summary to be consistent with the final profile). But we
> current tie together the FIS status and the profile, and cancellation of
> backends causes the FIS to return CANCELLED, which then means that the
> remaining FIS on that backend won't produce a final profile.
> With the rework of the protocol for IMPALA-2990 we should make it possible to
> sort this out such that a final profile can be requested regardless of how a
> FIS ends execution.
> This also relates to IMPALA-5783.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-3040) test_caching_ddl failed with unexpected get_num_cache_requests
[
https://issues.apache.org/jira/browse/IMPALA-3040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyi Wang resolved IMPALA-3040.
-
Resolution: Fixed
Fix Version/s: Impala 3.1.0
I haven't seen this for months. Closing it.
> test_caching_ddl failed with unexpected get_num_cache_requests
> --
>
> Key: IMPALA-3040
> URL: https://issues.apache.org/jira/browse/IMPALA-3040
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 2.5.0
>Reporter: Tim Armstrong
>Assignee: Tianyi Wang
>Priority: Major
> Labels: broken-build, flaky
> Fix For: Impala 3.1.0
>
>
> This test failed on the integration job.
> http://sandbox.jenkins.cloudera.com/view/Impala/view/Builds%20-%202.5.0%20release/job/impala-cdh5.7.0-integration/3/testReport/junit/query_test.test_hdfs_caching/TestHdfsCachingDdl/test_caching_ddl_exec_optiondisable_codegen___False___abort_on_error___1___exec_single_node_rows_threshold___0___batch_size___0___num_nodes___0table_format__text_none_/
> {code}
> query_test.test_hdfs_caching.TestHdfsCachingDdl.test_caching_ddl[exec_option:
> {'disable_codegen': False, 'abort_on_error': 1,
> 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} |
> table_format: text/none] (from pytest)
> Failing for the past 1 build (Since Failed#3 )
> Took 47 sec.
> add description
> Error Message
> assert 9 == 10 + where 10 = get_num_cache_requests()
> Stacktrace
> query_test/test_hdfs_caching.py:132: in test_caching_ddl
> assert num_entries_pre == get_num_cache_requests()
> E assert 9 == 10
> E+ where 10 = get_num_cache_requests()
> Standard Error
> -- connecting to: localhost:21000
> -- executing against localhost:21000
> use default;
> SET sync_ddl=1;
> -- executing against localhost:21000
> drop database if exists `cachedb` cascade;
> -- executing against localhost:21000
> create database cachedb;
> -- executing against localhost:21000
> use functional;
> SET disable_codegen=False;
> SET abort_on_error=1;
> SET exec_single_node_rows_threshold=0;
> SET batch_size=0;
> SET num_nodes=0;
> -- executing against localhost:21000
> use cachedb;
> -- executing against localhost:21000
> create table cached_tbl_nopart (i int) cached in 'testPool';
> -- executing against localhost:21000
> insert into cached_tbl_nopart select 1;
> -- executing against localhost:21000
> select * from cached_tbl_nopart;
> -- executing against localhost:21000
> create table like_cached_tbl_nopart like cached_tbl_nopart;
> -- executing against localhost:21000
> show table stats like_cached_tbl_nopart;
> -- executing against localhost:21000
> show table stats cached_tbl_nopart;
> -- executing against localhost:21000
> alter table cached_tbl_nopart set uncached;
> -- executing against localhost:21000
> show table stats cached_tbl_nopart;
> -- executing against localhost:21000
> drop table if exists cached_tbl_part;
> -- executing against localhost:21000
> create table cached_tbl_part (i int) partitioned by (j int) cached in
> 'testPool' with replication = 9;
> -- executing against localhost:21000
> alter table cached_tbl_part add partition (j=0);
> -- executing against localhost:21000
> alter table cached_tbl_part add partition (j=1) uncached;
> -- executing against localhost:21000
> alter table cached_tbl_part add partition (j=2) cached in 'testPool';
> -- executing against localhost:21000
> show partitions cached_tbl_part;
> -- executing against localhost:21000
> drop table if exists cached_tbl_part;
> -- executing against localhost:21000
> create table cached_tbl_part (i int) partitioned by (j int) cached in
> 'testPool';
> -- executing against localhost:21000
> alter table cached_tbl_part add partition (j=0);
> -- executing against localhost:21000
> alter table cached_tbl_part add partition (j=1) uncached;
> -- executing against localhost:21000
> alter table cached_tbl_part add partition (j=2) cached in 'testPool';
> -- executing against localhost:21000
> show partitions cached_tbl_part;
> -- executing against localhost:21000
> alter table cached_tbl_part partition (j=2) set uncached;
> -- executing against localhost:21000
> show partitions cached_tbl_part;
> -- executing against localhost:21000
> alter table cached_tbl_part partition (j=2) set uncached;
> -- executing against localhost:21000
> show partitions cached_tbl_part;
> -- executing against localhost:21000
> alter table cached_tbl_part partition (j=1) set cached in 'testPool';
> -- executing against localhost:21000
> show partitions cached_tbl_part;
> -- executing against localhost:21000
> insert into cached_tbl_part partition(j) values(3, 3), (4, 4);
> -- executing against localhost:21000
> alter table cached_tbl_part partition (j=3) set cached in 'testPool' with
>
[jira] [Resolved] (IMPALA-7448) Periodically evict recently unused table from catalogd
[ https://issues.apache.org/jira/browse/IMPALA-7448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tianyi Wang resolved IMPALA-7448. - Resolution: Fixed Fix Version/s: Impala 3.1.0 > Periodically evict recently unused table from catalogd > -- > > Key: IMPALA-7448 > URL: https://issues.apache.org/jira/browse/IMPALA-7448 > Project: IMPALA > Issue Type: Improvement > Components: Catalog >Affects Versions: Impala 3.1.0 >Reporter: Tianyi Wang >Assignee: Tianyi Wang >Priority: Major > Fix For: Impala 3.1.0 > > > To limit the memory consumption of catalog, we should experiment with a > mechanism automatically evicting recently unused tables from catalogd. > Initial design: > - impalad to report periodically/asynchronously the set of catalog objects > that were accessed > - catalogd to record some kind of last access time > - catalogd to have some facility to scan over all catalog objects, collect > some number of not-recently-used ones (eg to reach a target amount of evicted > memory), and issue invalidate commands to itself > - no need to have exact LRU behavior -- to simplify, we probably shouldn't > try to do a classical LRU linked list between all catalog objects. > - initial patch probably just triggered manually. Discussed either running > this on a schedule or running this based on JMX GC notifications if we see > that the catalogd finished an old-gen GC and the old gen is more than some > target percentage full. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7675) The result of UpdateTableUsage() RPC is not correctly handled.
[
https://issues.apache.org/jira/browse/IMPALA-7675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyi Wang resolved IMPALA-7675.
-
Resolution: Fixed
Fix Version/s: Impala 3.1.0
IMPALA-7675: Fix the error handling of UpdateTableUsage() RPC
UpdateTableUsage() is logically a one-way RPC and the status object in
TUpdateTableUsageResponse is set only if there is an error at RPC
layer. This patch fixes the incorrect error handling that leads to
NullPointerException in ImpaladTableUsageTracer.
Change-Id: Iccba4c6f4696ef08bc8a614ae13f62b5e445917b
Reviewed-on: http://gerrit.cloudera.org:8080/11603
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> The result of UpdateTableUsage() RPC is not correctly handled.
> --
>
> Key: IMPALA-7675
> URL: https://issues.apache.org/jira/browse/IMPALA-7675
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 3.1.0
>Reporter: Tianyi Wang
>Assignee: Tianyi Wang
>Priority: Major
> Fix For: Impala 3.1.0
>
>
> ImpalaTableUsageTracker.report() doesn't handle the result of
> UpdateTableUsage() RPC correctly and triggers NullpointerException:
> {noformat}
> W1003 11:07:39.252918 6910 ImpaladTableUsageTracker.java:116] Unable to
> report table usage information to catalog server.
> Java exception follows:
> java.lang.NullPointerException
> at
> org.apache.impala.catalog.ImpaladTableUsageTracker.report(ImpaladTableUsageTracker.java:110)
> at
> org.apache.impala.catalog.ImpaladTableUsageTracker.access$000(ImpaladTableUsageTracker.java:44)
> at
> org.apache.impala.catalog.ImpaladTableUsageTracker$1.run(ImpaladTableUsageTracker.java:56)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7859) Nessus Scan find CGI Generic SQL Injection.
[ https://issues.apache.org/jira/browse/IMPALA-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692304#comment-16692304 ] Jim Apple commented on IMPALA-7859: --- For security-related issues, please discuss on private@, making sure to CC anyone who is not a PMC member. I'll be deleting this issue to avoid any leaks. > Nessus Scan find CGI Generic SQL Injection. > --- > > Key: IMPALA-7859 > URL: https://issues.apache.org/jira/browse/IMPALA-7859 > Project: IMPALA > Issue Type: Bug >Reporter: Donghui Xu >Priority: Major > > The nessus scan report shows that the 25000 port and the 25020 port contain > the risk of SQL injection, as follows: > + The following resources may be vulnerable to blind SQL injection : > + The 'object_type' parameter of the /catalog_object CGI : > /catalog_object?object_name=_impala_builtins_type=DATABASEzz_impa > la_builtins_type=DATABASEyy > How can I solve this problem? Thanks. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Deleted] (IMPALA-7859) Nessus Scan find CGI Generic SQL Injection.
[ https://issues.apache.org/jira/browse/IMPALA-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jim Apple deleted IMPALA-7859: -- > Nessus Scan find CGI Generic SQL Injection. > --- > > Key: IMPALA-7859 > URL: https://issues.apache.org/jira/browse/IMPALA-7859 > Project: IMPALA > Issue Type: Bug >Reporter: Donghui Xu >Priority: Major > > The nessus scan report shows that the 25000 port and the 25020 port contain > the risk of SQL injection, as follows: > + The following resources may be vulnerable to blind SQL injection : > + The 'object_type' parameter of the /catalog_object CGI : > /catalog_object?object_name=_impala_builtins_type=DATABASEzz_impa > la_builtins_type=DATABASEyy > How can I solve this problem? Thanks. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7087) Impala is unable to read Parquet decimal columns with lower precision/scale than table metadata
[
https://issues.apache.org/jira/browse/IMPALA-7087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692242#comment-16692242
]
Sahil Takiar commented on IMPALA-7087:
--
For text files with extra scale, Impala throws an error:
{code}
[localhost:21000] default> create table dec_test_high_scale (col decimal(10,2))
stored as textfile;
[localhost:21000] default> insert into table dec_test_high_scale values (1),
(1.1), (1.11);
[localhost:21000] default> select * from dec_test_high_scale;
+--+
| col |
+--+
| 1.00 |
| 1.10 |
| 1.11 |
+--+
[localhost:21000] default> create table dec_test_low_scale (col decimal(10,1))
stored as textfile location
'hdfs://localhost:20500/test-warehouse/dec_test_high_scale';
[localhost:21000] default> select * from dec_test_low_scale;
+--+
| col |
+--+
| NULL |
| NULL |
| NULL |
+--+
WARNINGS: Error converting column: 0 to DECIMAL(10,1)
Error parsing row: file:
hdfs://localhost:20500/test-warehouse/dec_test_high_scale/44b64756e1edb4d-aa75eb24_381290695_data.0.,
before offset: 15
Error converting column: 0 to DECIMAL(10,1)
Error parsing row: file:
hdfs://localhost:20500/test-warehouse/dec_test_high_scale/44b64756e1edb4d-aa75eb24_381290695_data.0.,
before offset: 15
Error converting column: 0 to DECIMAL(10,1)
Error parsing row: file:
hdfs://localhost:20500/test-warehouse/dec_test_high_scale/44b64756e1edb4d-aa75eb24_381290695_data.0.,
before offset: 15
Fetched 3 row(s) in 4.25s
{code}
MySQL and Postgres both support the following:
{code}
create table dec_test (dec_col decimal(10,2));
insert into dec_test values (1.1), (1.11), (1.111);
select * from dec_test;
dec_col
1.10
1.11
1.11
{code}
I don't have access to the SQL standard, so I'm not sure if it has anything to
say about this behavior.
> Impala is unable to read Parquet decimal columns with lower precision/scale
> than table metadata
> ---
>
> Key: IMPALA-7087
> URL: https://issues.apache.org/jira/browse/IMPALA-7087
> Project: IMPALA
> Issue Type: Sub-task
> Components: Backend
>Reporter: Tim Armstrong
>Assignee: Sahil Takiar
>Priority: Major
> Labels: decimal, parquet
>
> This is similar to IMPALA-2515, except relates to a different precision/scale
> in the file metadata rather than just a mismatch in the bytes used to store
> the data. In a lot of cases we should be able to convert the decimal type on
> the fly to the higher-precision type.
> {noformat}
> ERROR: File '/hdfs/path/00_0_x_2' column 'alterd_decimal' has an invalid
> type length. Expecting: 11 len in file: 8
> {noformat}
> It would be convenient to allow reading parquet files where the
> precision/scale in the file can be converted to the precision/scale in the
> table metadata without loss of precision.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Comment Edited] (IMPALA-2385) Update test scripts to support repeated runs of end-to-end tests without restarting Impala
[ https://issues.apache.org/jira/browse/IMPALA-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692197#comment-16692197 ] Joe McDonnell edited comment on IMPALA-2385 at 11/19/18 8:11 PM: - There are two distinct use cases here. Endurance tests: Run Impala for a long time with a broad selection of tests and verify overall durability and behavior. For example, verify that memory usage does not keep increasing. Verify that things don't hang. Verify that resources don't leak. Etc. Targeted iteration of tests: Run specific tests many times and observe failures (or verify that it doesn't fail). It is useful to be able to run only a small set of tests (i.e. specific end to end tests, etc). It is useful to be able to run until the first failure or to record multiple failures. Some effort may go into preserving logs for all test runs. Sometimes we want to do this for custom cluster tests (which is incompatible with keeping Impala running). This is well-suited to having its own script that specifies a limited set of tests and allows certain modes of repeated execution. A simple first step is a script that runs a set of end to end tests and can repeat it a) until it fails b) a specific number of times c) for a specific time period. It should retain results for each run (see the --junit-xml and --result-log args to impala-py.test, which we use in [tests/run-tests.py|[https://github.com/apache/impala/blob/master/tests/run-tests.py#L57-L58]] We should support both. However, there is no such thing as endurance tests for frontend tests, backend tests, or custom cluster tests. That leaves mainly end to end tests. So, from a practical perspective, the script that would work for the targeted case would also solve the endurance case. Let's focus on the script for the targeted iteration and then work in the endurance aspect later. was (Author: joemcdonnell): There are two distinct use cases here. Endurance tests: Run Impala for a long time with a broad selection of tests and verify overall durability and behavior. For example, verify that memory usage does not keep increasing. Verify that things don't hang. Verify that resources don't leak. Etc. Targeted iteration of tests: Run specific tests many times and observe failures (or verify that it doesn't fail). It is useful to be able to run only a small set of tests (i.e. specific end to end tests, etc). It is useful to be able to run until the first failure or to record multiple failures. Some effort may go into preserving logs for all test runs. Sometimes we want to do this for custom cluster tests (which is incompatible with keeping Impala running). This is well-suited to having its own script that specifies a limited set of tests and allows certain modes of repeated execution. A simple first step is a script that runs a set of end to end tests and can repeat it a) until it fails b) a specific number of times c) for a specific time period. It should retain results for each run (see the --junit-xml and --result-log args to impala-py.test, which we use [in tests/run-tests.py|[https://github.com/apache/impala/blob/master/tests/run-tests.py#L57-L58]] We should support both. However, there is no such thing as endurance tests for frontend tests, backend tests, or custom cluster tests. That leaves mainly end to end tests. So, from a practical perspective, the script that would work for the targeted case would also solve the endurance case. Let's focus on the script for the targeted iteration and then work in the endurance aspect later. > Update test scripts to support repeated runs of end-to-end tests without > restarting Impala > -- > > Key: IMPALA-2385 > URL: https://issues.apache.org/jira/browse/IMPALA-2385 > Project: IMPALA > Issue Type: Task > Components: Infrastructure >Affects Versions: Impala 2.2.4 >Reporter: Harrison Sheinblatt >Priority: Major > Labels: test-infra > > The intent of the job is to find crashes due to resource leaks or other > stability issues by running the tests we have (which run in parallel) for an > extended period of time. > Currently, the job runs all tests, which include cluster tests that restart > Impala, and it runs a loop in the driving bash script that forces a restart > each iteration. These restarts must be removed. > Further, it needs to be verified that no other test restarts Impala. Ideally > the testing could also verify this automatically, but at least one manual > check that there are no restarts must be made. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail:
[jira] [Commented] (IMPALA-2385) Update test scripts to support repeated runs of end-to-end tests without restarting Impala
[ https://issues.apache.org/jira/browse/IMPALA-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692197#comment-16692197 ] Joe McDonnell commented on IMPALA-2385: --- There are two distinct use cases here. Endurance tests: Run Impala for a long time with a broad selection of tests and verify overall durability and behavior. For example, verify that memory usage does not keep increasing. Verify that things don't hang. Verify that resources don't leak. Etc. Targeted iteration of tests: Run specific tests many times and observe failures (or verify that it doesn't fail). It is useful to be able to run only a small set of tests (i.e. specific end to end tests, etc). It is useful to be able to run until the first failure or to record multiple failures. Some effort may go into preserving logs for all test runs. Sometimes we want to do this for custom cluster tests (which is incompatible with keeping Impala running). This is well-suited to having its own script that specifies a limited set of tests and allows certain modes of repeated execution. A simple first step is a script that runs a set of end to end tests and can repeat it a) until it fails b) a specific number of times c) for a specific time period. It should retain results for each run (see the --junit-xml and --result-log args to impala-py.test, which we use [in tests/run-tests.py|[https://github.com/apache/impala/blob/master/tests/run-tests.py#L57-L58].] We should support both. However, there is no such thing as endurance tests for frontend tests, backend tests, or custom cluster tests. That leaves mainly end to end tests. So, from a practical perspective, the script that would work for the targeted case would also solve the endurance case. Let's focus on the script for the targeted iteration and then work in the endurance aspect later. > Update test scripts to support repeated runs of end-to-end tests without > restarting Impala > -- > > Key: IMPALA-2385 > URL: https://issues.apache.org/jira/browse/IMPALA-2385 > Project: IMPALA > Issue Type: Task > Components: Infrastructure >Affects Versions: Impala 2.2.4 >Reporter: Harrison Sheinblatt >Priority: Major > Labels: test-infra > > The intent of the job is to find crashes due to resource leaks or other > stability issues by running the tests we have (which run in parallel) for an > extended period of time. > Currently, the job runs all tests, which include cluster tests that restart > Impala, and it runs a loop in the driving bash script that forces a restart > each iteration. These restarts must be removed. > Further, it needs to be verified that no other test restarts Impala. Ideally > the testing could also verify this automatically, but at least one manual > check that there are no restarts must be made. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Comment Edited] (IMPALA-2385) Update test scripts to support repeated runs of end-to-end tests without restarting Impala
[ https://issues.apache.org/jira/browse/IMPALA-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692197#comment-16692197 ] Joe McDonnell edited comment on IMPALA-2385 at 11/19/18 8:10 PM: - There are two distinct use cases here. Endurance tests: Run Impala for a long time with a broad selection of tests and verify overall durability and behavior. For example, verify that memory usage does not keep increasing. Verify that things don't hang. Verify that resources don't leak. Etc. Targeted iteration of tests: Run specific tests many times and observe failures (or verify that it doesn't fail). It is useful to be able to run only a small set of tests (i.e. specific end to end tests, etc). It is useful to be able to run until the first failure or to record multiple failures. Some effort may go into preserving logs for all test runs. Sometimes we want to do this for custom cluster tests (which is incompatible with keeping Impala running). This is well-suited to having its own script that specifies a limited set of tests and allows certain modes of repeated execution. A simple first step is a script that runs a set of end to end tests and can repeat it a) until it fails b) a specific number of times c) for a specific time period. It should retain results for each run (see the --junit-xml and --result-log args to impala-py.test, which we use [in tests/run-tests.py|[https://github.com/apache/impala/blob/master/tests/run-tests.py#L57-L58]] We should support both. However, there is no such thing as endurance tests for frontend tests, backend tests, or custom cluster tests. That leaves mainly end to end tests. So, from a practical perspective, the script that would work for the targeted case would also solve the endurance case. Let's focus on the script for the targeted iteration and then work in the endurance aspect later. was (Author: joemcdonnell): There are two distinct use cases here. Endurance tests: Run Impala for a long time with a broad selection of tests and verify overall durability and behavior. For example, verify that memory usage does not keep increasing. Verify that things don't hang. Verify that resources don't leak. Etc. Targeted iteration of tests: Run specific tests many times and observe failures (or verify that it doesn't fail). It is useful to be able to run only a small set of tests (i.e. specific end to end tests, etc). It is useful to be able to run until the first failure or to record multiple failures. Some effort may go into preserving logs for all test runs. Sometimes we want to do this for custom cluster tests (which is incompatible with keeping Impala running). This is well-suited to having its own script that specifies a limited set of tests and allows certain modes of repeated execution. A simple first step is a script that runs a set of end to end tests and can repeat it a) until it fails b) a specific number of times c) for a specific time period. It should retain results for each run (see the --junit-xml and --result-log args to impala-py.test, which we use [in tests/run-tests.py|[https://github.com/apache/impala/blob/master/tests/run-tests.py#L57-L58].] We should support both. However, there is no such thing as endurance tests for frontend tests, backend tests, or custom cluster tests. That leaves mainly end to end tests. So, from a practical perspective, the script that would work for the targeted case would also solve the endurance case. Let's focus on the script for the targeted iteration and then work in the endurance aspect later. > Update test scripts to support repeated runs of end-to-end tests without > restarting Impala > -- > > Key: IMPALA-2385 > URL: https://issues.apache.org/jira/browse/IMPALA-2385 > Project: IMPALA > Issue Type: Task > Components: Infrastructure >Affects Versions: Impala 2.2.4 >Reporter: Harrison Sheinblatt >Priority: Major > Labels: test-infra > > The intent of the job is to find crashes due to resource leaks or other > stability issues by running the tests we have (which run in parallel) for an > extended period of time. > Currently, the job runs all tests, which include cluster tests that restart > Impala, and it runs a loop in the driving bash script that forces a restart > each iteration. These restarts must be removed. > Further, it needs to be verified that no other test restarts Impala. Ideally > the testing could also verify this automatically, but at least one manual > check that there are no restarts must be made. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail:
[jira] [Commented] (IMPALA-7859) Nessus Scan find CGI Generic SQL Injection.
[ https://issues.apache.org/jira/browse/IMPALA-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692154#comment-16692154 ] bharath v commented on IMPALA-7859: --- [~davidxdh] We don't construct a SQL query using the params from this endpoint. So a SQL injection kinda attack is not possible AFAICT. However, it is interesting that nessus scan flagged only this endpoint (among others) and we'd like to understand why. Does the report include any diagnostic information on what test URL was constructed and what was expected/actual output etc? > Nessus Scan find CGI Generic SQL Injection. > --- > > Key: IMPALA-7859 > URL: https://issues.apache.org/jira/browse/IMPALA-7859 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 2.10.0 >Reporter: Donghui Xu >Priority: Major > > The nessus scan report shows that the 25000 port and the 25020 port contain > the risk of SQL injection, as follows: > + The following resources may be vulnerable to blind SQL injection : > + The 'object_type' parameter of the /catalog_object CGI : > /catalog_object?object_name=_impala_builtins_type=DATABASEzz_impa > la_builtins_type=DATABASEyy > How can I solve this problem? Thanks. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7872) Extended health checks to mark node as down
Tim Armstrong created IMPALA-7872: - Summary: Extended health checks to mark node as down Key: IMPALA-7872 URL: https://issues.apache.org/jira/browse/IMPALA-7872 Project: IMPALA Issue Type: Improvement Components: Distributed Exec Reporter: Tim Armstrong This is an umbrella JIRA to improve handling of complex failure modes aside from fail-stop. The current statestore heartbeat mechanism assumes that an Impala daemon that responds to heartbeats is healthy and can be scheduled on. Memory-based admission control provides a bit more robustness here by not admitting queries on daemons where memory would be oversubscribed. Examples of failure modes of interest are: * Hangs, where a particular node can't make progress (the JVM hangs in IMPALA-7483 are a good example) on some or all queries. * Repeated fragment instance startup failures. E.g. where coordinators can't successfully start fragments on an impala daemon, because of communication errors or other issues. We can't automatically handle all failure modes, but we could improve handling of some common ones, particularly repeated fragment startup failures or hangs. The goal would be to degrade more gracefully to avoid repeated failures causing a cluster-wide outage. The goal isn't to prevent all failures, just to recover to a healthy state automatically in more scenarios. IMPALA-1760 (graceful shutdown) may give us some better options here, since if a node notices that it is somehow unhealthy, it could gracefully remove itself from scheduling and restart itself. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7804) Various scanner tests intermittently failing on S3 on different runs
[
https://issues.apache.org/jira/browse/IMPALA-7804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692121#comment-16692121
]
Joe McDonnell commented on IMPALA-7804:
---
New failures in test_scanners.TestParquet.test_zero_rows() and
test_scanners.TestParquet.test_corrupt_rle_counts(). Since a bunch of these
involve test_scanners.py, IMPALA-6709 comes to mind:
[https://github.com/apache/impala/commit/e27954a5aa585db23fe3c97726aa89305efa306d#diff-c0076d575312e4fb9c8d8fad0191224e]
This changed how we load files for several of these tests. It may have
unforeseen s3 implications.
Separately, Amazon has this blurb in their s3 introduction about consistency:
{noformat}
Amazon S3 provides read-after-write consistency for PUTS of new objects in your
S3 bucket in all regions with one caveat. The caveat is that if you make a HEAD
or GET request to the key name (to find if the object exists) before creating
the object, Amazon S3 provides eventual consistency for
read-after-write.{noformat}
If I understand this correctly, whenever we check for the existence of a file,
that puts us into the eventual consistency case when we write it. If the file
doesn't exist and we do a direct put, it is consistent. In quite a few of our
tests, we know the file won't already exist and we don't care about checking
for existence. Based on the s3 output, we are definitely checking for existence
before writing the file (in this case, renaming to that location, which is a
copy):
{noformat}
18/11/18 05:42:51 DEBUG s3a.S3AFileSystem: Getting path status for
s3a://impala-test-uswest2-1/test-warehouse/test_corrupt_rle_counts_44c21800.db/bad_rle_repeat_count.parquet
(test-warehouse/test_corrupt_rle_counts_44c21800.db/bad_rle_repeat_count.parquet)
18/11/18 05:42:51 DEBUG s3a.S3AStorageStatistics: object_metadata_requests += 1
-> 12
18/11/18 05:42:51 DEBUG s3a.S3AStorageStatistics: object_metadata_requests += 1
-> 13
18/11/18 05:42:51 DEBUG s3a.S3AStorageStatistics: object_list_requests += 1 ->
6
18/11/18 05:42:52 DEBUG s3a.S3AFileSystem: Not Found:
s3a://impala-test-uswest2-1/test-warehouse/test_corrupt_rle_counts_44c21800.db/bad_rle_repeat_count.parquet
18/11/18 05:42:52 DEBUG s3a.S3AFileSystem: rename: destination path
s3a://impala-test-uswest2-1/test-warehouse/test_corrupt_rle_counts_44c21800.db/bad_rle_repeat_count.parquet
not found
...
18/11/18 05:42:52 DEBUG s3a.S3AFileSystem: rename: renaming file
s3a://impala-test-uswest2-1/test-warehouse/test_corrupt_rle_counts_44c21800.db/bad_rle_repeat_count.parquet._COPYING_
to
s3a://impala-test-uswest2-1/test-warehouse/test_corrupt_rle_counts_44c21800.db/bad_rle_repeat_count.parquet
18/11/18 05:42:52 DEBUG s3a.S3AFileSystem: copyFile
test-warehouse/test_corrupt_rle_counts_44c21800.db/bad_rle_repeat_count.parquet._COPYING_
->
test-warehouse/test_corrupt_rle_counts_44c21800.db/bad_rle_repeat_count.parquet
{noformat}
Impala's LOAD DATA statement checks for existence in
[FileSystemUtil::relocateAllVisibleFiles()|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java#L143],
and it seems like the hadoop commandline also does so. Our s3 filesystem
utility based on Boto3 (tests/util/s3_util.py) has an overwrite mode that
doesn't seem to do an existence check. Switching to use this filesystem util
might impact our s3 consistency.
> Various scanner tests intermittently failing on S3 on different runs
>
>
> Key: IMPALA-7804
> URL: https://issues.apache.org/jira/browse/IMPALA-7804
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: David Knupp
>Priority: Blocker
> Labels: S3
>
> The failures have to do with getting AWS client credentials.
> *query_test/test_scanners.py:696: in test_decimal_encodings*
> _Stacktrace_
> {noformat}
> query_test/test_scanners.py:696: in test_decimal_encodings
> self.run_test_case('QueryTest/parquet-decimal-formats', vector,
> unique_database)
> common/impala_test_suite.py:496: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:358: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:438: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:260: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E -255.00,-255.00,-255.00 == -255.00,-255.00,-255.00
> E -255.00,-255.00,-255.00 != -65535.00,-65535.00,-65535.00
> E -65535.00,-65535.00,-65535.00 != -999.99,-999.99,-999.99
> E -65535.00,-65535.00,-65535.00 !=
[jira] [Commented] (IMPALA-7585) Always set user credentials after creating a KRPC proxy
[
https://issues.apache.org/jira/browse/IMPALA-7585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692110#comment-16692110
]
ASF subversion and git services commented on IMPALA-7585:
-
Commit e1c9cbd0771cd01776c2db79cf753bf73d786e5a in impala's branch
refs/heads/master from [~jbapple]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=e1c9cbd ]
IMPALA-7585: support LDAP in run-workload.py
This patch just threads through the user, password, and ssl settings
all the way back to the ImpalaBeeswaxClient.
Change-Id: Ibfa987d8a027f50bc1ba3db5aa355331442a74ba
Reviewed-on: http://gerrit.cloudera.org:8080/11938
Tested-by: Impala Public Jenkins
Reviewed-by: David Knupp
> Always set user credentials after creating a KRPC proxy
> ---
>
> Key: IMPALA-7585
> URL: https://issues.apache.org/jira/browse/IMPALA-7585
> Project: IMPALA
> Issue Type: Bug
> Components: Distributed Exec
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Michael Ho
>Assignee: Michael Ho
>Priority: Major
> Fix For: Impala 3.1.0
>
>
> {{kudu::rpc::Proxy}} ctor may fail in {{GetLoggedInUser()}} for various
> reason:
> {noformat}
> Error calling getpwuid_r(): No such file or directory (error 2).
> {noformat}
> This resulted in an empty user name being used in
> {{kudu::rpc::ConnectionId}}. With plaintext SASL (e.g. in an insecure Impala
> cluster), this may result in the following error:
> {noformat}
> Not authorized: Client connection negotiation failed: client connection to
> 127.0.0.1:27000: SASL(-1): generic failure: All-whitespace username.
> {noformat}
> While one can argue that Kudu should fall back to some default username (e.g.
> "cpp-client") when {{GetLoggedInUserName()}} fails, it may have non-trivial
> consequence (e.g. generating an authn token with some random username on one
> machine while using the real user name on another machine). Therefore, it's
> best for Impala to explicitly set the user credentials
> (impala/) after creating the proxy.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7871) Don't load Hive builtin jars for dataload
Joe McDonnell created IMPALA-7871: - Summary: Don't load Hive builtin jars for dataload Key: IMPALA-7871 URL: https://issues.apache.org/jira/browse/IMPALA-7871 Project: IMPALA Issue Type: Improvement Components: Infrastructure Affects Versions: Impala 3.1.0 Reporter: Joe McDonnell Assignee: Joe McDonnell One step in dataload is "Loading Hive Builtins", which copies a large number of jars into HDFS (or whatever storage). This step takes a couple minutes on HDFS dataload and 8 minutes on S3. Despite its name, I can't find any indication that Hive or anything else uses these jars. Dataload and core tests run fine without it. S3 can load data without it. There's no indication that this is needed. Unless we find something using these jars, we should remove this step. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7870) TestAutomaticCatalogInvalidation.test_v1_catalog intermittently fails
[
https://issues.apache.org/jira/browse/IMPALA-7870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell updated IMPALA-7870:
--
Description:
A couple test runs have seen
custom_cluster.test_automatic_invalidation.TestAutomaticCatalogInvalidation.test_v1_catalog()
take slightly longer than the timeout:
{noformat}
custom_cluster/test_automatic_invalidation.py:69: in test_v1_catalog
self._run_test(cursor)
custom_cluster/test_automatic_invalidation.py:64: in _run_test
assert time.time() < max_wait_time
E assert 1542544633.922359 < 1542544633.68588
E+ where 1542544633.922359 = ()
E+where = time.time{noformat}
and
{noformat}
custom_cluster/test_automatic_invalidation.py:69: in test_v1_catalog
self._run_test(cursor)
custom_cluster/test_automatic_invalidation.py:64: in _run_test
assert time.time() < max_wait_time
E assert 1542372031.541345 < 1542372031.303814
E+ where 1542372031.541345 = ()
E+where = time.time{noformat}
This has been seen twice on the core job.
was:
A couple test runs have seen
custom_cluster.test_automatic_invalidation.TestAutomaticCatalogInvalidation.test_v1_catalog()
take slightly longer than the timeout:
{noformat}
custom_cluster/test_automatic_invalidation.py:69: in test_v1_catalog
self._run_test(cursor)
custom_cluster/test_automatic_invalidation.py:64: in _run_test
assert time.time() < max_wait_time
E assert 1542544633.922359 < 1542544633.68588
E+ where 1542544633.922359 = ()
E+where = time.time{noformat}
and
{noformat}
custom_cluster/test_automatic_invalidation.py:69: in test_v1_catalog
self._run_test(cursor)
custom_cluster/test_automatic_invalidation.py:64: in _run_test
assert time.time() < max_wait_time
E assert 1542372031.541345 < 1542372031.303814
E+ where 1542372031.541345 = ()
E+where = time.time{noformat}
> TestAutomaticCatalogInvalidation.test_v1_catalog intermittently fails
> -
>
> Key: IMPALA-7870
> URL: https://issues.apache.org/jira/browse/IMPALA-7870
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 3.1.0
>Reporter: Joe McDonnell
>Priority: Critical
> Labels: broken-build
>
> A couple test runs have seen
> custom_cluster.test_automatic_invalidation.TestAutomaticCatalogInvalidation.test_v1_catalog()
> take slightly longer than the timeout:
> {noformat}
> custom_cluster/test_automatic_invalidation.py:69: in test_v1_catalog
> self._run_test(cursor)
> custom_cluster/test_automatic_invalidation.py:64: in _run_test
> assert time.time() < max_wait_time
> E assert 1542544633.922359 < 1542544633.68588
> E+ where 1542544633.922359 = ()
> E+where = time.time{noformat}
> and
> {noformat}
> custom_cluster/test_automatic_invalidation.py:69: in test_v1_catalog
> self._run_test(cursor)
> custom_cluster/test_automatic_invalidation.py:64: in _run_test
> assert time.time() < max_wait_time
> E assert 1542372031.541345 < 1542372031.303814
> E+ where 1542372031.541345 = ()
> E+where = time.time{noformat}
> This has been seen twice on the core job.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7870) TestAutomaticCatalogInvalidation.test_v1_catalog intermittently fails
Joe McDonnell created IMPALA-7870:
-
Summary: TestAutomaticCatalogInvalidation.test_v1_catalog
intermittently fails
Key: IMPALA-7870
URL: https://issues.apache.org/jira/browse/IMPALA-7870
Project: IMPALA
Issue Type: Bug
Components: Catalog
Affects Versions: Impala 3.1.0
Reporter: Joe McDonnell
A couple test runs have seen
custom_cluster.test_automatic_invalidation.TestAutomaticCatalogInvalidation.test_v1_catalog()
take slightly longer than the timeout:
{noformat}
custom_cluster/test_automatic_invalidation.py:69: in test_v1_catalog
self._run_test(cursor)
custom_cluster/test_automatic_invalidation.py:64: in _run_test
assert time.time() < max_wait_time
E assert 1542544633.922359 < 1542544633.68588
E+ where 1542544633.922359 = ()
E+where = time.time{noformat}
and
{noformat}
custom_cluster/test_automatic_invalidation.py:69: in test_v1_catalog
self._run_test(cursor)
custom_cluster/test_automatic_invalidation.py:64: in _run_test
assert time.time() < max_wait_time
E assert 1542372031.541345 < 1542372031.303814
E+ where 1542372031.541345 = ()
E+where = time.time{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7367) Pack StringValue, CollectionValue and TimestampValue slots
[
https://issues.apache.org/jira/browse/IMPALA-7367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pooja Nilangekar resolved IMPALA-7367.
--
Resolution: Fixed
Fix Version/s: Impala 3.2.0
> Pack StringValue, CollectionValue and TimestampValue slots
> --
>
> Key: IMPALA-7367
> URL: https://issues.apache.org/jira/browse/IMPALA-7367
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Reporter: Tim Armstrong
>Assignee: Pooja Nilangekar
>Priority: Major
> Labels: perfomance
> Fix For: Impala 3.2.0
>
> Attachments: 0001-WIP.patch
>
>
> This is a follow-on to finish up the work from IMPALA-2789. IMPALA-2789
> didn't actually fully pack the memory layout because StringValue,
> TimestampValue and CollectionValue still occupy 16 bytes but only have 12
> bytes of actual data. This results in a higher memory footprint, which leads
> to higher memory requirements and worse performance. We don't get any benefit
> from the padding since the majority of tuples are not actually aligned in
> memory anyway.
> I did a quick version of the change for StringValue only which improves TPC-H
> performance.
> {noformat}
> Report Generated on 2018-07-30
> Run Description: "b5608264b4552e44eb73ded1e232a8775c3dba6b vs
> f1e401505ac20c0400eec819b9196f7f506fb927"
> Cluster Name: UNKNOWN
> Lab Run Info: UNKNOWN
> Impala Version: impalad version 3.1.0-SNAPSHOT RELEASE ()
> Baseline Impala Version: impalad version 3.1.0-SNAPSHOT RELEASE (2018-07-27)
> +--+---+-++++
> | Workload | File Format | Avg (s) | Delta(Avg) | GeoMean(s) |
> Delta(GeoMean) |
> +--+---+-++++
> | TPCH(10) | parquet / none / none | 2.69| -4.78% | 2.09 |
> -3.11% |
> +--+---+-++++
> +--+--+---++-++++-+---+
> | Workload | Query| File Format | Avg(s) | Base Avg(s) |
> Delta(Avg) | StdDev(%) | Base StdDev(%) | Num Clients | Iters |
> +--+--+---++-++++-+---+
> | TPCH(10) | TPCH-Q22 | parquet / none / none | 0.94 | 0.93|
> +0.75% | 3.37%| 2.84%| 1 | 30|
> | TPCH(10) | TPCH-Q13 | parquet / none / none | 3.32 | 3.32|
> +0.13% | 1.74%| 2.09%| 1 | 30|
> | TPCH(10) | TPCH-Q11 | parquet / none / none | 0.99 | 0.99|
> -0.02% | 3.74%| 3.16%| 1 | 30|
> | TPCH(10) | TPCH-Q5 | parquet / none / none | 2.30 | 2.33|
> -0.96% | 2.15%| 2.45%| 1 | 30|
> | TPCH(10) | TPCH-Q2 | parquet / none / none | 1.55 | 1.57|
> -1.45% | 1.65%| 1.49%| 1 | 30|
> | TPCH(10) | TPCH-Q8 | parquet / none / none | 2.89 | 2.93|
> -1.51% | 2.69%| 1.34%| 1 | 30|
> | TPCH(10) | TPCH-Q9 | parquet / none / none | 5.96 | 6.06|
> -1.63% | 1.34%| 1.82%| 1 | 30|
> | TPCH(10) | TPCH-Q20 | parquet / none / none | 1.58 | 1.61|
> -1.85% | 2.28%| 2.16%| 1 | 30|
> | TPCH(10) | TPCH-Q16 | parquet / none / none | 1.18 | 1.21|
> -2.11% | 3.68%| 4.72%| 1 | 30|
> | TPCH(10) | TPCH-Q3 | parquet / none / none | 2.13 | 2.18|
> -2.31% | 2.09%| 1.92%| 1 | 30|
> | TPCH(10) | TPCH-Q15 | parquet / none / none | 1.86 | 1.90|
> -2.52% | 2.06%| 2.22%| 1 | 30|
> | TPCH(10) | TPCH-Q17 | parquet / none / none | 1.85 | 1.90|
> -2.86% | 10.00% | 8.02%| 1 | 30|
> | TPCH(10) | TPCH-Q10 | parquet / none / none | 2.58 | 2.66|
> -2.93% | 1.68%| 6.49%| 1 | 30|
> | TPCH(10) | TPCH-Q14 | parquet / none / none | 1.37 | 1.42|
> -3.22% | 3.35%| 6.24%| 1 | 30|
> | TPCH(10) | TPCH-Q18 | parquet / none / none | 4.99 | 5.17|
> -3.38% | 1.75%| 3.82%| 1 | 30|
> | TPCH(10) | TPCH-Q6 | parquet / none / none | 0.66 | 0.69|
> -3.73% | 5.04%| 4.12%| 1 | 30|
> | TPCH(10) | TPCH-Q4 | parquet / none / none | 1.07 | 1.12|
> -3.97% | 1.79%| 2.85%| 1
[jira] [Commented] (IMPALA-5031) UBSAN clean and method for testing UBSAN cleanliness
[ https://issues.apache.org/jira/browse/IMPALA-5031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692033#comment-16692033 ] ASF subversion and git services commented on IMPALA-5031: - Commit 53ce6bb571cd9ae07ba5255197d35aa852a6f97c in impala's branch refs/heads/master from [~jbapple] [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=53ce6bb ] IMPALA-5031: signed overflow is undefined behavior This patch fixes a signed overflow in the backend test DecimalTest.Overflow. The interesting part of the backtrace is: runtime/decimal-value.inline.h:254:17: runtime error: signed integer overflow: 0x4b3b4ca85a86c47a098a223f + 0x4b3b4ca85a86c47a098a223f cannot be represented in type '__int128' #0 detail::AddLarge(__int128, int, __int128, int, int, bool, bool*) runtime/decimal-value.inline.h:254:17 #1 DecimalValue<__int128> DecimalValue<__int128>::Add<__int128>( int, DecimalValue<__int128> const&, int, int, int, bool, bool*) const runtime/decimal-value.inline.h:371:14 #2 DecimalTest_Overflow_Test::TestBody() runtime/decimal-test.cc:540:9 Change-Id: I146bcf35d34cc0e14be0633427d3e4bd0e5a261e Reviewed-on: http://gerrit.cloudera.org:8080/11917 Reviewed-by: Jim Apple Tested-by: Impala Public Jenkins > UBSAN clean and method for testing UBSAN cleanliness > > > Key: IMPALA-5031 > URL: https://issues.apache.org/jira/browse/IMPALA-5031 > Project: IMPALA > Issue Type: Task > Components: Backend, Infrastructure >Affects Versions: Impala 2.9.0 >Reporter: Jim Apple >Assignee: Jim Apple >Priority: Minor > > http://releases.llvm.org/3.8.0/tools/clang/docs/UndefinedBehaviorSanitizer.html > builds are supported after https://gerrit.cloudera.org/#/c/6186/, but > Impala's test suite triggers many errors under UBSAN. Those errors should be > fixed and then there should be a way to run the test suite under UBSAN and > fail if there were any errors detected. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7367) Pack StringValue, CollectionValue and TimestampValue slots
[
https://issues.apache.org/jira/browse/IMPALA-7367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692034#comment-16692034
]
ASF subversion and git services commented on IMPALA-7367:
-
Commit 2a4835cfba7597362cc1e72e21315868c5c75d0a in impala's branch
refs/heads/master from poojanilangekar
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=2a4835c ]
IMPALA-7367: Pack StringValue and CollectionValue slots
This change packs StringValue and CollectionValue slots to ensure
they now occupy 12 bytes instead of 16 bytes. This reduces the
memory requirements and improves the performance. Since Kudu
tuples are populated using a memcopy, 4 bytes of padding was
added to StringSlots in Kudu tables.
Testing:
Ran core tests.
Added static asserts to ensure the value sizes are as expected.
Performance tests on TPCH-40 produced 3.96% improvement.
Change-Id: I32f3b06622c087e4aa288e8db1bf4581b10d386a
Reviewed-on: http://gerrit.cloudera.org:8080/11599
Tested-by: Impala Public Jenkins
Reviewed-by: Tim Armstrong
> Pack StringValue, CollectionValue and TimestampValue slots
> --
>
> Key: IMPALA-7367
> URL: https://issues.apache.org/jira/browse/IMPALA-7367
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Reporter: Tim Armstrong
>Assignee: Pooja Nilangekar
>Priority: Major
> Labels: perfomance
> Attachments: 0001-WIP.patch
>
>
> This is a follow-on to finish up the work from IMPALA-2789. IMPALA-2789
> didn't actually fully pack the memory layout because StringValue,
> TimestampValue and CollectionValue still occupy 16 bytes but only have 12
> bytes of actual data. This results in a higher memory footprint, which leads
> to higher memory requirements and worse performance. We don't get any benefit
> from the padding since the majority of tuples are not actually aligned in
> memory anyway.
> I did a quick version of the change for StringValue only which improves TPC-H
> performance.
> {noformat}
> Report Generated on 2018-07-30
> Run Description: "b5608264b4552e44eb73ded1e232a8775c3dba6b vs
> f1e401505ac20c0400eec819b9196f7f506fb927"
> Cluster Name: UNKNOWN
> Lab Run Info: UNKNOWN
> Impala Version: impalad version 3.1.0-SNAPSHOT RELEASE ()
> Baseline Impala Version: impalad version 3.1.0-SNAPSHOT RELEASE (2018-07-27)
> +--+---+-++++
> | Workload | File Format | Avg (s) | Delta(Avg) | GeoMean(s) |
> Delta(GeoMean) |
> +--+---+-++++
> | TPCH(10) | parquet / none / none | 2.69| -4.78% | 2.09 |
> -3.11% |
> +--+---+-++++
> +--+--+---++-++++-+---+
> | Workload | Query| File Format | Avg(s) | Base Avg(s) |
> Delta(Avg) | StdDev(%) | Base StdDev(%) | Num Clients | Iters |
> +--+--+---++-++++-+---+
> | TPCH(10) | TPCH-Q22 | parquet / none / none | 0.94 | 0.93|
> +0.75% | 3.37%| 2.84%| 1 | 30|
> | TPCH(10) | TPCH-Q13 | parquet / none / none | 3.32 | 3.32|
> +0.13% | 1.74%| 2.09%| 1 | 30|
> | TPCH(10) | TPCH-Q11 | parquet / none / none | 0.99 | 0.99|
> -0.02% | 3.74%| 3.16%| 1 | 30|
> | TPCH(10) | TPCH-Q5 | parquet / none / none | 2.30 | 2.33|
> -0.96% | 2.15%| 2.45%| 1 | 30|
> | TPCH(10) | TPCH-Q2 | parquet / none / none | 1.55 | 1.57|
> -1.45% | 1.65%| 1.49%| 1 | 30|
> | TPCH(10) | TPCH-Q8 | parquet / none / none | 2.89 | 2.93|
> -1.51% | 2.69%| 1.34%| 1 | 30|
> | TPCH(10) | TPCH-Q9 | parquet / none / none | 5.96 | 6.06|
> -1.63% | 1.34%| 1.82%| 1 | 30|
> | TPCH(10) | TPCH-Q20 | parquet / none / none | 1.58 | 1.61|
> -1.85% | 2.28%| 2.16%| 1 | 30|
> | TPCH(10) | TPCH-Q16 | parquet / none / none | 1.18 | 1.21|
> -2.11% | 3.68%| 4.72%| 1 | 30|
> | TPCH(10) | TPCH-Q3 | parquet / none / none | 2.13 | 2.18|
> -2.31% | 2.09%| 1.92%| 1 | 30|
> | TPCH(10) | TPCH-Q15 | parquet / none / none | 1.86 | 1.90|
> -2.52% | 2.06%| 2.22%| 1 | 30|
> | TPCH(10) |
[jira] [Created] (IMPALA-7869) Split up parquet-column-readers.cc for readability and compile time
Tim Armstrong created IMPALA-7869: - Summary: Split up parquet-column-readers.cc for readability and compile time Key: IMPALA-7869 URL: https://issues.apache.org/jira/browse/IMPALA-7869 Project: IMPALA Issue Type: Improvement Components: Backend Reporter: Tim Armstrong Assignee: Tim Armstrong [~csringhofer] suggested reorganising the file to be easier to read on https://gerrit.cloudera.org/#/c/8319/ Compile times are also an issue - this file is the longest pole in the Impala compilation at the moment. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7477) Improve QueryResultSet interface to allow appending a batch of rows at a time
[ https://issues.apache.org/jira/browse/IMPALA-7477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691927#comment-16691927 ] ASF subversion and git services commented on IMPALA-7477: - Commit 031d5690f8a7d354da83cb34da3cc96bd0a55e86 in impala's branch refs/heads/branch-3.1.0 from Zoltan Borok-Nagy [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=031d569 ] Revert "IMPALA-7477: Batch-oriented query set construction" This reverts commit 0f63b2c9f9d62b0d22191f454b672a6047206252. > Improve QueryResultSet interface to allow appending a batch of rows at a time > - > > Key: IMPALA-7477 > URL: https://issues.apache.org/jira/browse/IMPALA-7477 > Project: IMPALA > Issue Type: Sub-task > Components: Backend >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > Fix For: Impala 3.2.0 > > > The QueryResultSet interface used from PlanRootSink operates on a row at a > time and is inefficient and inelegant. We can improve code readability by > moving some logic from PlanRootSink to QueryResultSet and improve perf if we > switch to an interface that allows appending a batch. > This will make IMPALA-4268 incrementally easier. > There are some TODOs in the code related to this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7477) Improve QueryResultSet interface to allow appending a batch of rows at a time
[ https://issues.apache.org/jira/browse/IMPALA-7477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691917#comment-16691917 ] ASF subversion and git services commented on IMPALA-7477: - Commit 33d943b2592a76ec616f7bae0664f255ebfa1718 in impala's branch refs/heads/branch-3.1.0 from Zoltan Borok-Nagy [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=33d943b ] Revert "IMPALA-7477: Batch-oriented query set construction" This reverts commit 0f63b2c9f9d62b0d22191f454b672a6047206252. > Improve QueryResultSet interface to allow appending a batch of rows at a time > - > > Key: IMPALA-7477 > URL: https://issues.apache.org/jira/browse/IMPALA-7477 > Project: IMPALA > Issue Type: Sub-task > Components: Backend >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > Fix For: Impala 3.2.0 > > > The QueryResultSet interface used from PlanRootSink operates on a row at a > time and is inefficient and inelegant. We can improve code readability by > moving some logic from PlanRootSink to QueryResultSet and improve perf if we > switch to an interface that allows appending a batch. > This will make IMPALA-4268 incrementally easier. > There are some TODOs in the code related to this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7822) Crash in repeat() constructing strings > 1GB
[
https://issues.apache.org/jira/browse/IMPALA-7822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691542#comment-16691542
]
ASF subversion and git services commented on IMPALA-7822:
-
Commit 1760339a5fb4375a22c3579e5eeb505335bcf8f6 in impala's branch
refs/heads/branch-3.1.0 from [~tarmstr...@cloudera.com]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=1760339 ]
IMPALA-7822: handle overflows in repeat() builtin
We need to carefully check that the intermediate value fits in an
int64_t and the final size fits in an int. If they don't we
raise an error and fail the query.
Testing:
Added a couple of backend tests to exercise the
overflow check code paths.
Change-Id: I872ce77bc2cb29116881c27ca2a5216f722cdb2a
Reviewed-on: http://gerrit.cloudera.org:8080/11889
Reviewed-by: Thomas Marshall
Tested-by: Impala Public Jenkins
> Crash in repeat() constructing strings > 1GB
>
>
> Key: IMPALA-7822
> URL: https://issues.apache.org/jira/browse/IMPALA-7822
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.3.0, Impala 2.5.0, Impala 2.4.0, Impala 2.6.0,
> Impala 2.7.0, Impala 2.8.0, Impala 2.9.0, Impala 2.10.0, Impala 2.11.0,
> Impala 3.0, Impala 3.1.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: crash
> Fix For: Impala 3.1.0
>
>
> {noformat}
> select repeat('x', 1024 * 1024 * 1024 * 1024);
> {noformat}
> {noformat}
> #0 0x04551aa3 in
> google_breakpad::ExceptionHandler::SignalHandler(int, siginfo_t*, void*) ()
> #1 0x7f72cdc0f362 in ?? () from
> /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
> #2 0x7f72cdc138b9 in JVM_handle_linux_signal ()
>from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
> #3 0x7f72cdc06f78 in ?? () from
> /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
> #4
> #5 0x7f72cab8a06c in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> #6 0x030e19da in impala::StringFunctions::Repeat (context=0x721f800,
> str=..., n=...)
> at be/src/exprs/string-functions-ir.cc:100
> #7 0x03145c32 in
> impala::ScalarFnCall::InterpretEval (
> this=0xaf2cb40, eval=0xd26ccc0, row=0x0) at
> be/src/exprs/scalar-fn-call.cc:485
> #8 0x0312bd85 in impala::ScalarFnCall::GetStringVal (this=0xaf2cb40,
> eval=0xd26ccc0,
> row=0x0) at be/src/exprs/scalar-fn-call.cc:599
> #9 0x030da116 in impala::ScalarExprEvaluator::GetValue
> (this=0xd26ccc0, expr=..., row=0x0)
> at be/src/exprs/scalar-expr-evaluator.cc:299
> #10 0x030d9da5 in impala::ScalarExprEvaluator::GetValue
> (this=0xd26ccc0, row=0x0)
> at be/src/exprs/scalar-expr-evaluator.cc:250
> #11 0x01fc59bc in
> Java_org_apache_impala_service_FeSupport_NativeEvalExprsWithoutRow (
> env=0xd25b1e0, caller_class=0x7f724d523280,
> thrift_expr_batch=0x7f724d523298,
> thrift_query_ctx_bytes=0x7f724d523290) at be/src/service/fe-support.cc:237
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-5031) UBSAN clean and method for testing UBSAN cleanliness
[ https://issues.apache.org/jira/browse/IMPALA-5031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691551#comment-16691551 ] ASF subversion and git services commented on IMPALA-5031: - Commit f79f2d4ad0b8ea2f209785a9b2bec61703102881 in impala's branch refs/heads/branch-3.1.0 from [~jbapple] [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=f79f2d4 ] IMPALA-5031: Jenkins jobs should fail when UBSAN fails Before this patch, Jenkins builds do not fail or warn if UBSAN produces an error. This patch introduces two new failure modes, "error" and "death". In "error", a junit error is generated, which causes a build to be labeled "unstable". In "death", a forked process constantly monitors the test logs and kills the test run if a UBSAN error is found. Change-Id: I783243458ac2765a97a1dd7dd40d458cc2e1d80b Reviewed-on: http://gerrit.cloudera.org:8080/11813 Reviewed-by: Jim Apple Tested-by: Impala Public Jenkins > UBSAN clean and method for testing UBSAN cleanliness > > > Key: IMPALA-5031 > URL: https://issues.apache.org/jira/browse/IMPALA-5031 > Project: IMPALA > Issue Type: Task > Components: Backend, Infrastructure >Affects Versions: Impala 2.9.0 >Reporter: Jim Apple >Assignee: Jim Apple >Priority: Minor > > http://releases.llvm.org/3.8.0/tools/clang/docs/UndefinedBehaviorSanitizer.html > builds are supported after https://gerrit.cloudera.org/#/c/6186/, but > Impala's test suite triggers many errors under UBSAN. Those errors should be > fixed and then there should be a way to run the test suite under UBSAN and > fail if there were any errors detected. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7835) Creating a role with the same name as the user name with object ownership enabled can cause INVALIDATE METADATA to hang
[
https://issues.apache.org/jira/browse/IMPALA-7835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691550#comment-16691550
]
ASF subversion and git services commented on IMPALA-7835:
-
Commit 87c2ad0531d66ac7c253f340301b5782f3afe7fd in impala's branch
refs/heads/branch-3.1.0 from [~fredyw]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=87c2ad0 ]
IMPALA-7835: Role and user catalog objects with the same name can cause
INVALIDATE METADATA to hang
Before this patch, enabling object ownership and running INVALIDATE
METADATA for the role and user catalog objects with the same principal
name could cause INVALIDATE METADATA to hang because a principal name
without a type is not guaranteed to be unique. This causes the catalog
delta logic to assume principal catalog objects with the same name but
with different types are considered the same causing a catalog version
inconsistency. This patch makes the principal object key to be unique by
appending the principal type information.
Testing:
- Added E2E authorization test
- Ran all FE tests
- Ran all E2E authorization tests
Change-Id: I516cf72e69e142a1349950cfca91f035c1ed445f
Reviewed-on: http://gerrit.cloudera.org:8080/11910
Reviewed-by: Vuk Ercegovac
Tested-by: Impala Public Jenkins
> Creating a role with the same name as the user name with object ownership
> enabled can cause INVALIDATE METADATA to hang
> ---
>
> Key: IMPALA-7835
> URL: https://issues.apache.org/jira/browse/IMPALA-7835
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 3.1.0
>Reporter: Fredy Wijaya
>Assignee: Fredy Wijaya
>Priority: Critical
> Fix For: Impala 3.1.0
>
>
> Start Impala with object ownership enabled.
> {noformat}
> [localhost:21000] default> create database foo;
> [localhost:21000] default> create role impdev;
> [localhost:21000] default> grant all on server to role impdev;
> [localhost:21000] default> grant role impdev to group impdev;
> [localhost:21000] default> invalidate metadata; -- this will hang
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-3652) Fix resource transfer in subplans with limits
[
https://issues.apache.org/jira/browse/IMPALA-3652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691544#comment-16691544
]
ASF subversion and git services commented on IMPALA-3652:
-
Commit ccabc491e20e566144f0b07739a65092c48a4953 in impala's branch
refs/heads/branch-3.1.0 from [~twmarshall]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=ccabc49 ]
IMPALA-3652: Fix resource transfer in subplans with limits
Impala assumes that when Reset() is called on an ExecNode, all of the
memory returned from that node by GetNext() has been attached to the
output RowBatch. In a query with a LIMIT on the subplan, such that
some nodes don't reach 'eos', this may not be the case.
The solution is to have Reset() take a RowBatch that any such memory
can be attached to. I examined all ExecNodes for resources being
transferred on 'eos' and added transferring of those resources in
Resst().
Testing:
- Added e2e tests that repro the issue for hash and nested loop joins.
Change-Id: I3968a379fcbb5d30fcec304995d3e44933dbbc77
Reviewed-on: http://gerrit.cloudera.org:8080/11852
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> Fix resource transfer in subplans with limits
> -
>
> Key: IMPALA-3652
> URL: https://issues.apache.org/jira/browse/IMPALA-3652
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.6.0
>Reporter: Tim Armstrong
>Assignee: Thomas Tauber-Marshall
>Priority: Major
> Labels: resource-management
> Fix For: Impala 3.1.0
>
>
> There is a tricky corner case in our resource transfer model with subplans
> and limits. The problem is that the limit in the subplan may mean that the
> exec node is reset before it has returned its full output. The resource
> transfer logic generally attaches resources to batches at specific points in
> the output, e.g. end of partition, end of block, so it's possible that
> batches returned before the Reset() may reference resources that have not yet
> been transferred. It's unclear if we test this scenario consistently or if
> it's always handled correctly.
> One example is this query, reported in IMPALA-5456:
> {code}
> select c_custkey, c_mktsegment, o_orderkey, o_orderdate
> from customer c,
> (select o1.o_orderkey, o2.o_orderdate
>from c.c_orders o1, c.c_orders o2
>where o1.o_orderkey = o2.o_orderkey limit 10) v limit 500;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7824) Running INVALIDATE METADATA with authorization enabled can cause a hang if Sentry is unavailable
[ https://issues.apache.org/jira/browse/IMPALA-7824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691547#comment-16691547 ] ASF subversion and git services commented on IMPALA-7824: - Commit 59f3f6b901581d2f6b5084fa739db2b5870f2af0 in impala's branch refs/heads/branch-3.1.0 from [~fredyw] [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=59f3f6b ] IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable Before this patch, running INVALIDATE METADATA when Sentry is unavailable could cause Impala query to hang. PolicyReader thread in SentryProxy is used by two use cases, one as a background thread that periodically refreshes Sentry policy and another one as a synchronous operation for INVALIDATE METADATA. For the background thread, we need to swallow any exception thrown while refreshing the Sentry policy in order to not kill the background thread. For a synchronous reset operation, such as INVALIDATE METADATA, swallowing an exception causes the Impala catalog to wait indefinitely for authorization catalog objects that never get processed due to Sentry being unavailable. The patch updates the code by not swallowing any exception in INVALIDATE METADATA and return the exception to the caller. Testing: - Ran all FE tests - Added a new E2E test - Ran all E2E authorization tests Change-Id: Icff987a6184f62a338faadfdc1a0d349d912fc37 Reviewed-on: http://gerrit.cloudera.org:8080/11897 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Running INVALIDATE METADATA with authorization enabled can cause a hang if > Sentry is unavailable > > > Key: IMPALA-7824 > URL: https://issues.apache.org/jira/browse/IMPALA-7824 > Project: IMPALA > Issue Type: Bug > Components: Catalog >Affects Versions: Impala 3.0, Impala 2.12.0 >Reporter: Fredy Wijaya >Assignee: Fredy Wijaya >Priority: Critical > Fix For: Impala 3.1.0 > > > Steps to reproduce: > 1. Start Impala with authorization > 2. Kill Sentry > 3. Run INVALIDATE METADATA -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7840) test_concurrent_schema_change is missing a possible error message
[ https://issues.apache.org/jira/browse/IMPALA-7840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691549#comment-16691549 ] ASF subversion and git services commented on IMPALA-7840: - Commit 4a10096064b95bff4d25d2af3c393794ec4e328b in impala's branch refs/heads/branch-3.1.0 from [~twmarshall] [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=4a10096 ] IMPALA-7840: add missing error to test_concurrent_schema_change test_concurrent_schema_change runs a series of alters and inserts on the same Kudu table concurrently to ensure that Impala can handle this without crashing. There is a list of expected error messages in the test. One possible legitimate error is missing, causing the test to sometimes be flaky. This patch adds that error message to the test. Change-Id: Ibc01e1244e05be30343c05b5a0f8676f4d4603d2 Reviewed-on: http://gerrit.cloudera.org:8080/11913 Reviewed-by: Thomas Marshall Tested-by: Impala Public Jenkins > test_concurrent_schema_change is missing a possible error message > - > > Key: IMPALA-7840 > URL: https://issues.apache.org/jira/browse/IMPALA-7840 > Project: IMPALA > Issue Type: Bug >Affects Versions: Impala 3.1.0 >Reporter: Thomas Tauber-Marshall >Assignee: Thomas Tauber-Marshall >Priority: Critical > Fix For: Impala 3.1.0 > > > test_concurrent_schema_change runs a series of alters and inserts on the same > Kudu table concurrently to ensure that Impala can handle this without > crashing. > There is a list of expected error messages in the test. One possible > legitimate error is missing, causing the test to sometimes be flaky. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7791) Aggregation Node memory estimates don't account for number of fragment instances
[
https://issues.apache.org/jira/browse/IMPALA-7791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691543#comment-16691543
]
ASF subversion and git services commented on IMPALA-7791:
-
Commit ffdb8a13e4d550f07c8a756625f4df9441b157fc in impala's branch
refs/heads/branch-3.1.0 from poojanilangekar
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=ffdb8a1 ]
IMPALA-7791: Compute AggregationNode's estimated rows using # instances
Previously, the AggregationNode calculated the estimated number
of rows based on input cardinality without accounting for the
division of input data across multiple fragment instances. This
bloated up the memory estimates for the node. After this change,
the AggregationNode accounts for the number of fragment instances
while estimating the number of rows per instance. A skew factor of
1.5 was added to account for data skew among multiple fragment
instances. This number was derived using empirical analysis of
real-world and benchmark (tpch, tpcds) queries.
Testing:
Tested queries with changed estimates to avoid cases of
significant underestimation of memory.
Ran front-end and end-to-end tests affected by this change.
Change-Id: I2cb9746fafa3e5952e28caa952837e285bcc22ac
Reviewed-on: http://gerrit.cloudera.org:8080/11854
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> Aggregation Node memory estimates don't account for number of fragment
> instances
>
>
> Key: IMPALA-7791
> URL: https://issues.apache.org/jira/browse/IMPALA-7791
> Project: IMPALA
> Issue Type: Sub-task
>Affects Versions: Impala 3.1.0
>Reporter: Pooja Nilangekar
>Assignee: Pooja Nilangekar
>Priority: Blocker
> Fix For: Impala 3.1.0
>
>
> AggregationNode's memory estimates are calculated based on the input
> cardinality of the node, without accounting for the division of input data
> across fragment instances. This results in very high memory estimates. In
> reality, the nodes often use only a part of this memory.
> Example query:
> {code:java}
> [localhost:21000] default> select distinct * from tpch.lineitem limit 5;
> {code}
> Summary:
> {code:java}
> +--++--+--+---++---+---+---+
> | Operator | #Hosts | Avg Time | Max Time | #Rows | Est. #Rows | Peak Mem
> | Est. Peak Mem | Detail
>
>
>
>
>|
> +--++--+--+---++---+---+---+
> | 04:EXCHANGE | 1 | 21.24us | 21.24us | 5 | 5 | 48.00 KB
> | 16.00 KB | UNPARTITIONED
>
>
>
>
>|
> | 03:AGGREGATE | 3 | 5.11s| 5.15s| 15| 5 | 576.21
> MB | 1.62 GB | FINALIZE
>
>
>
>
>
[jira] [Commented] (IMPALA-7818) Standardize use of Expr predicates
[
https://issues.apache.org/jira/browse/IMPALA-7818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691545#comment-16691545
]
ASF subversion and git services commented on IMPALA-7818:
-
Commit 8dfbe3c82d4acfae60d58636af041ac13708ef1e in impala's branch
refs/heads/branch-3.1.0 from [~paul-rogers]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=8dfbe3c ]
IMPALA-7818: Standardize use of Expr predicates
The Expr node provide two kinds of queries about node categories:
predicates and isMumble() functions. This change standardizes two
existing methods to use predicates instead.
First, the existing isLiteral() method is replaced by a new
IS_LITERAL predicate.
The key purpose is to clean up the confusing use of the existing
isNullLiteral() method, which actually is a check for either a null
literal or a cast of a null. To make this clearer, replaced this
method with a new IS_NULL_VALUE() predicate.
Added a new IS_NULL_LITERAL predicate for the case when a node must be
exactly the NULL literal, not a cast to NULL. Replaced instances of
foo instanceof NullLiteral with a use of the new IS_NULL_LITERAL
predicate. (This revealed bugs which will be addressed separately.)
Added an IS_NON_NULL_LITERAL to replace the two-method idiom used in
several places.
Tests: No functional change. Reran all tests to ensure nothing changed.
Change-Id: I09a089c0f2484246e5c6444b78990fa9260c036a
Reviewed-on: http://gerrit.cloudera.org:8080/11887
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> Standardize use of Expr predicates
> --
>
> Key: IMPALA-7818
> URL: https://issues.apache.org/jira/browse/IMPALA-7818
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
>Affects Versions: Impala 3.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> The {{Expr}} (expression) class in the frontend has many handy predicates
> such as {{IS_BINARY_PREDICATE}} and so on. (Note the confusing terminology:
> there are subclasses of {{Expr}} which are SQL predicates. The predicates
> discussed here are those based on the Guava {{Predicate}} class.)
> The class also has predicate-like methods: {{isLiteral()}} and
> {{isNullLiteral()}}. As it has evolved, {{isNullLiteral()}} checks not only
> for a null literal, but also a cast of a null to some type. These functions
> are in the base class, but do their work by checking the type of the
> expression.
> It would be cleaner to make these methods into predicates with names that
> follow their function: {{IS_LITERAL}}, and {{IS_NULL_VALUE}}.
> Further, there are a few places in the code that check for a non-null literal
> using an and of conditions, This would be cleaner using a new
> {{IS_NON_NULL_LITERAL}} predicate.
> These changes put us in position to add new predicates as a result of work in
> the pipeline; this refactoring is done as a separate change to keep the other
> commit smaller.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7528) Division by zero when computing cardinalities of many to many joins on NULL columns
[
https://issues.apache.org/jira/browse/IMPALA-7528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691546#comment-16691546
]
ASF subversion and git services commented on IMPALA-7528:
-
Commit d07bc818d6a8af6b2ebc2d1d2672849abc40f51c in impala's branch
refs/heads/branch-3.1.0 from [~bikram.sngh91]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=d07bc81 ]
IMPALA-7528: Fix division by zero when computing cardinalities
This patch fixes a case where there can be a division by zero when
computing cardinalities of many to many joins on NULL columns.
Testing:
Added a planner test case
Change-Id: Ieedd51d3ad6131a4ea2a5883f05104e6a0f2cd14
Reviewed-on: http://gerrit.cloudera.org:8080/11901
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> Division by zero when computing cardinalities of many to many joins on NULL
> columns
> ---
>
> Key: IMPALA-7528
> URL: https://issues.apache.org/jira/browse/IMPALA-7528
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 2.12.0
>Reporter: Balazs Jeszenszky
>Assignee: Bikramjeet Vig
>Priority: Critical
> Labels: planner
> Fix For: Impala 3.1.0
>
>
> The following:
> {code:java}
> | F00:PLAN FRAGMENT [RANDOM] hosts=1 instances=1 |
> | Per-Host Resources: mem-estimate=33.94MB mem-reservation=1.94MB|
> | 02:HASH JOIN [INNER JOIN, BROADCAST] |
> | | hash predicates: b.code = a.code|
> | | fk/pk conjuncts: none |
> | | runtime filters: RF000 <- a.code|
> | | mem-estimate=1.94MB mem-reservation=1.94MB spill-buffer=64.00KB |
> | | tuple-ids=1,0 row-size=163B cardinality=9223372036854775807 |
> < Estimation due to overflow.
> | | |
> | |--03:EXCHANGE [BROADCAST] |
> | | | mem-estimate=0B mem-reservation=0B |
> | | | tuple-ids=0 row-size=82B cardinality=823 |
> | | | |
> | | F01:PLAN FRAGMENT [RANDOM] hosts=1 instances=1 |
> | | Per-Host Resources: mem-estimate=32.00MB mem-reservation=0B |
> | | 00:SCAN HDFS [default.sample_07 a, RANDOM] |
> | | partitions=1/1 files=1 size=44.98KB |
> | | stats-rows=823 extrapolated-rows=disabled|
> | | table stats: rows=823 size=44.98KB |
> | | column stats: all|
> | | mem-estimate=32.00MB mem-reservation=0B |
> | | tuple-ids=0 row-size=82B cardinality=823 |
> | | |
> | 01:SCAN HDFS [default.sample_08 b, RANDOM] |
> |partitions=1/1 files=1 size=44.99KB |
> |runtime filters: RF000 -> b.code|
> |stats-rows=823 extrapolated-rows=disabled |
> |table stats: rows=823 size=44.99KB |
> |column stats: all |
> |mem-estimate=32.00MB mem-reservation=0B |
> |tuple-ids=1 row-size=82B cardinality=823|
> ++
> {code}
> is the result of both join columns having 0 as NDV.
> https://github.com/cloudera/Impala/blob/cdh5-trunk/fe/src/main/java/org/apache/impala/planner/JoinNode.java#L368
> should handle this more gracefully.
> IMPALA-7310 makes it a bit more likely that someone will run into this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
