[jira] [Commented] (IMPALA-7415) Flaky test: TestImpalaShellInteractive.test_multiline_queries_in_history
[
https://issues.apache.org/jira/browse/IMPALA-7415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574069#comment-16574069
]
Tim Armstrong commented on IMPALA-7415:
---
That looks like the immediate cause. I suspect the reason that the buffer only
contains that is that "Fetched 1 row\(s\) in .*s" gobbled up the "[localhos"
part of the prompt.
> Flaky test: TestImpalaShellInteractive.test_multiline_queries_in_history
>
>
> Key: IMPALA-7415
> URL: https://issues.apache.org/jira/browse/IMPALA-7415
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.1.0
>Reporter: Bikramjeet Vig
>Assignee: Bikramjeet Vig
>Priority: Critical
> Labels: broken-build, flaky
>
> This happened on a GVO build on one of my patches:
> {noformat}
> 22:23:51 ] FAIL
> shell/test_shell_interactive.py::TestImpalaShellInteractive::()::test_multiline_queries_in_history
> 22:23:51 ] === FAILURES
> ===
> 22:23:51 ] _
> TestImpalaShellInteractive.test_multiline_queries_in_history _
> 22:23:51 ] [gw5] linux2 -- Python 2.7.12
> /home/ubuntu/Impala/bin/../infra/python/env/bin/python
> 22:23:51 ] shell/test_shell_interactive.py:303: in
> test_multiline_queries_in_history
> 22:23:51 ] child_proc.expect(PROMPT_REGEX)
> 22:23:51 ]
> ../infra/python/env/local/lib/python2.7/site-packages/pexpect/__init__.py:1451:
> in expect
> 22:23:51 ] timeout, searchwindowsize)
> 22:23:51 ]
> ../infra/python/env/local/lib/python2.7/site-packages/pexpect/__init__.py:1466:
> in expect_list
> 22:23:51 ] timeout, searchwindowsize)
> 22:23:51 ]
> ../infra/python/env/local/lib/python2.7/site-packages/pexpect/__init__.py:1568:
> in expect_loop
> 22:23:51 ] raise TIMEOUT(str(err) + '\n' + str(self))
> 22:23:51 ] E TIMEOUT: Timeout exceeded.
> 22:23:51 ] E
> 22:23:51 ] E version: 3.3
> 22:23:51 ] E command: /home/ubuntu/Impala/bin/impala-shell.sh
> 22:23:51 ] E args: ['/home/ubuntu/Impala/bin/impala-shell.sh']
> 22:23:51 ] E searcher:
> 22:23:51 ] E buffer (last 100 chars): 't:21000] default> '
> 22:23:51 ] E before (last 100 chars): 't:21000] default> '
> 22:23:51 ] E after:
> 22:23:51 ] E match: None
> 22:23:51 ] E match_index: None
> 22:23:51 ] E exitstatus: None
> 22:23:51 ] E flag_eof: False
> 22:23:51 ] E pid: 114214
> 22:23:51 ] E child_fd: 18
> 22:23:51 ] E closed: False
> 22:23:51 ] E timeout: 30
> 22:23:51 ] E delimiter:
> 22:23:51 ] E logfile: None
> 22:23:51 ] E logfile_read: None
> 22:23:51 ] E logfile_send: None
> 22:23:51 ] E maxread: 2000
> 22:23:51 ] E ignorecase: False
> 22:23:51 ] E searchwindowsize: None
> 22:23:51 ] E delaybeforesend: 0.05
> 22:23:51 ] E delayafterclose: 0.1
> 22:23:51 ] E delayafterterminate: 0.1
> 22:23:51 ] = 1 failed, 2013 passed, 71 skipped, 45 xfailed, 1 xpassed in
> 2219.10 seconds ==
> {noformat}
> Link to failed GVO:
> [https://jenkins.impala.io/job/gerrit-verify-dryrun/2955/consoleFull]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7415) Flaky test: TestImpalaShellInteractive.test_multiline_queries_in_history
[
https://issues.apache.org/jira/browse/IMPALA-7415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574047#comment-16574047
]
Bikramjeet Vig commented on IMPALA-7415:
[~tarmstr...@cloudera.com] since you have looked at this before, is there a
possibility that that the buffer only contains 't:21000] default> ' which
causes the searcher to fail?
> Flaky test: TestImpalaShellInteractive.test_multiline_queries_in_history
>
>
> Key: IMPALA-7415
> URL: https://issues.apache.org/jira/browse/IMPALA-7415
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.1.0
>Reporter: Bikramjeet Vig
>Assignee: Bikramjeet Vig
>Priority: Critical
> Labels: broken-build, flaky
>
> This happened on a GVO build on one of my patches:
> {noformat}
> 22:23:51 ] FAIL
> shell/test_shell_interactive.py::TestImpalaShellInteractive::()::test_multiline_queries_in_history
> 22:23:51 ] === FAILURES
> ===
> 22:23:51 ] _
> TestImpalaShellInteractive.test_multiline_queries_in_history _
> 22:23:51 ] [gw5] linux2 -- Python 2.7.12
> /home/ubuntu/Impala/bin/../infra/python/env/bin/python
> 22:23:51 ] shell/test_shell_interactive.py:303: in
> test_multiline_queries_in_history
> 22:23:51 ] child_proc.expect(PROMPT_REGEX)
> 22:23:51 ]
> ../infra/python/env/local/lib/python2.7/site-packages/pexpect/__init__.py:1451:
> in expect
> 22:23:51 ] timeout, searchwindowsize)
> 22:23:51 ]
> ../infra/python/env/local/lib/python2.7/site-packages/pexpect/__init__.py:1466:
> in expect_list
> 22:23:51 ] timeout, searchwindowsize)
> 22:23:51 ]
> ../infra/python/env/local/lib/python2.7/site-packages/pexpect/__init__.py:1568:
> in expect_loop
> 22:23:51 ] raise TIMEOUT(str(err) + '\n' + str(self))
> 22:23:51 ] E TIMEOUT: Timeout exceeded.
> 22:23:51 ] E
> 22:23:51 ] E version: 3.3
> 22:23:51 ] E command: /home/ubuntu/Impala/bin/impala-shell.sh
> 22:23:51 ] E args: ['/home/ubuntu/Impala/bin/impala-shell.sh']
> 22:23:51 ] E searcher:
> 22:23:51 ] E buffer (last 100 chars): 't:21000] default> '
> 22:23:51 ] E before (last 100 chars): 't:21000] default> '
> 22:23:51 ] E after:
> 22:23:51 ] E match: None
> 22:23:51 ] E match_index: None
> 22:23:51 ] E exitstatus: None
> 22:23:51 ] E flag_eof: False
> 22:23:51 ] E pid: 114214
> 22:23:51 ] E child_fd: 18
> 22:23:51 ] E closed: False
> 22:23:51 ] E timeout: 30
> 22:23:51 ] E delimiter:
> 22:23:51 ] E logfile: None
> 22:23:51 ] E logfile_read: None
> 22:23:51 ] E logfile_send: None
> 22:23:51 ] E maxread: 2000
> 22:23:51 ] E ignorecase: False
> 22:23:51 ] E searchwindowsize: None
> 22:23:51 ] E delaybeforesend: 0.05
> 22:23:51 ] E delayafterclose: 0.1
> 22:23:51 ] E delayafterterminate: 0.1
> 22:23:51 ] = 1 failed, 2013 passed, 71 skipped, 45 xfailed, 1 xpassed in
> 2219.10 seconds ==
> {noformat}
> Link to failed GVO:
> [https://jenkins.impala.io/job/gerrit-verify-dryrun/2955/consoleFull]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7415) Flaky test: TestImpalaShellInteractive.test_multiline_queries_in_history
Bikramjeet Vig created IMPALA-7415:
--
Summary: Flaky test:
TestImpalaShellInteractive.test_multiline_queries_in_history
Key: IMPALA-7415
URL: https://issues.apache.org/jira/browse/IMPALA-7415
Project: IMPALA
Issue Type: Bug
Affects Versions: Impala 3.1.0
Reporter: Bikramjeet Vig
Assignee: Bikramjeet Vig
This happened on a GVO build on one of my patches:
{noformat}
22:23:51 ] FAIL
shell/test_shell_interactive.py::TestImpalaShellInteractive::()::test_multiline_queries_in_history
22:23:51 ] === FAILURES
===
22:23:51 ] _
TestImpalaShellInteractive.test_multiline_queries_in_history _
22:23:51 ] [gw5] linux2 -- Python 2.7.12
/home/ubuntu/Impala/bin/../infra/python/env/bin/python
22:23:51 ] shell/test_shell_interactive.py:303: in
test_multiline_queries_in_history
22:23:51 ] child_proc.expect(PROMPT_REGEX)
22:23:51 ]
../infra/python/env/local/lib/python2.7/site-packages/pexpect/__init__.py:1451:
in expect
22:23:51 ] timeout, searchwindowsize)
22:23:51 ]
../infra/python/env/local/lib/python2.7/site-packages/pexpect/__init__.py:1466:
in expect_list
22:23:51 ] timeout, searchwindowsize)
22:23:51 ]
../infra/python/env/local/lib/python2.7/site-packages/pexpect/__init__.py:1568:
in expect_loop
22:23:51 ] raise TIMEOUT(str(err) + '\n' + str(self))
22:23:51 ] E TIMEOUT: Timeout exceeded.
22:23:51 ] E
22:23:51 ] E version: 3.3
22:23:51 ] E command: /home/ubuntu/Impala/bin/impala-shell.sh
22:23:51 ] E args: ['/home/ubuntu/Impala/bin/impala-shell.sh']
22:23:51 ] E searcher:
22:23:51 ] E buffer (last 100 chars): 't:21000] default> '
22:23:51 ] E before (last 100 chars): 't:21000] default> '
22:23:51 ] E after:
22:23:51 ] E match: None
22:23:51 ] E match_index: None
22:23:51 ] E exitstatus: None
22:23:51 ] E flag_eof: False
22:23:51 ] E pid: 114214
22:23:51 ] E child_fd: 18
22:23:51 ] E closed: False
22:23:51 ] E timeout: 30
22:23:51 ] E delimiter:
22:23:51 ] E logfile: None
22:23:51 ] E logfile_read: None
22:23:51 ] E logfile_send: None
22:23:51 ] E maxread: 2000
22:23:51 ] E ignorecase: False
22:23:51 ] E searchwindowsize: None
22:23:51 ] E delaybeforesend: 0.05
22:23:51 ] E delayafterclose: 0.1
22:23:51 ] E delayafterterminate: 0.1
22:23:51 ] = 1 failed, 2013 passed, 71 skipped, 45 xfailed, 1 xpassed in
2219.10 seconds ==
{noformat}
Link to failed GVO:
[https://jenkins.impala.io/job/gerrit-verify-dryrun/2955/consoleFull]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IMPALA-7415) Flaky test: TestImpalaShellInteractive.test_multiline_queries_in_history
Bikramjeet Vig created IMPALA-7415:
--
Summary: Flaky test:
TestImpalaShellInteractive.test_multiline_queries_in_history
Key: IMPALA-7415
URL: https://issues.apache.org/jira/browse/IMPALA-7415
Project: IMPALA
Issue Type: Bug
Affects Versions: Impala 3.1.0
Reporter: Bikramjeet Vig
Assignee: Bikramjeet Vig
This happened on a GVO build on one of my patches:
{noformat}
22:23:51 ] FAIL
shell/test_shell_interactive.py::TestImpalaShellInteractive::()::test_multiline_queries_in_history
22:23:51 ] === FAILURES
===
22:23:51 ] _
TestImpalaShellInteractive.test_multiline_queries_in_history _
22:23:51 ] [gw5] linux2 -- Python 2.7.12
/home/ubuntu/Impala/bin/../infra/python/env/bin/python
22:23:51 ] shell/test_shell_interactive.py:303: in
test_multiline_queries_in_history
22:23:51 ] child_proc.expect(PROMPT_REGEX)
22:23:51 ]
../infra/python/env/local/lib/python2.7/site-packages/pexpect/__init__.py:1451:
in expect
22:23:51 ] timeout, searchwindowsize)
22:23:51 ]
../infra/python/env/local/lib/python2.7/site-packages/pexpect/__init__.py:1466:
in expect_list
22:23:51 ] timeout, searchwindowsize)
22:23:51 ]
../infra/python/env/local/lib/python2.7/site-packages/pexpect/__init__.py:1568:
in expect_loop
22:23:51 ] raise TIMEOUT(str(err) + '\n' + str(self))
22:23:51 ] E TIMEOUT: Timeout exceeded.
22:23:51 ] E
22:23:51 ] E version: 3.3
22:23:51 ] E command: /home/ubuntu/Impala/bin/impala-shell.sh
22:23:51 ] E args: ['/home/ubuntu/Impala/bin/impala-shell.sh']
22:23:51 ] E searcher:
22:23:51 ] E buffer (last 100 chars): 't:21000] default> '
22:23:51 ] E before (last 100 chars): 't:21000] default> '
22:23:51 ] E after:
22:23:51 ] E match: None
22:23:51 ] E match_index: None
22:23:51 ] E exitstatus: None
22:23:51 ] E flag_eof: False
22:23:51 ] E pid: 114214
22:23:51 ] E child_fd: 18
22:23:51 ] E closed: False
22:23:51 ] E timeout: 30
22:23:51 ] E delimiter:
22:23:51 ] E logfile: None
22:23:51 ] E logfile_read: None
22:23:51 ] E logfile_send: None
22:23:51 ] E maxread: 2000
22:23:51 ] E ignorecase: False
22:23:51 ] E searchwindowsize: None
22:23:51 ] E delaybeforesend: 0.05
22:23:51 ] E delayafterclose: 0.1
22:23:51 ] E delayafterterminate: 0.1
22:23:51 ] = 1 failed, 2013 passed, 71 skipped, 45 xfailed, 1 xpassed in
2219.10 seconds ==
{noformat}
Link to failed GVO:
[https://jenkins.impala.io/job/gerrit-verify-dryrun/2955/consoleFull]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7241) progress-updater.cc:43] Check failed: delta >= 0 (-3 vs. 0)
[
https://issues.apache.org/jira/browse/IMPALA-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574026#comment-16574026
]
Michael Ho commented on IMPALA-7241:
We could have updated {{dml_exec_state}} more than once due to RPC timeout and
retry in {{ReportExecStatusAux()}}
> progress-updater.cc:43] Check failed: delta >= 0 (-3 vs. 0)
> ---
>
> Key: IMPALA-7241
> URL: https://issues.apache.org/jira/browse/IMPALA-7241
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Michael Brown
>Assignee: Michael Ho
>Priority: Blocker
> Labels: crash, stress
>
> During a stress test with 8 nodes running an insecure debug build based off
> master, an impalad hit a DCHECK. The concurrency level at the time was
> between 150-180 queries.
> The DCHECK was {{progress-updater.cc:43] Check failed: delta >= 0 (-3 vs.
> 0)}}.
> The stack was:
> {noformat}
> #0 0x7f6a73e811f7 in raise () from /lib64/libc.so.6
> #1 0x7f6a73e828e8 in abort () from /lib64/libc.so.6
> #2 0x04300e34 in google::DumpStackTraceAndExit() ()
> #3 0x042f78ad in google::LogMessage::Fail() ()
> #4 0x042f9152 in google::LogMessage::SendToLog() ()
> #5 0x042f7287 in google::LogMessage::Flush() ()
> #6 0x042fa84e in google::LogMessageFatal::~LogMessageFatal() ()
> #7 0x01f7fedd in impala::ProgressUpdater::Update(long) ()
> #8 0x0313912b in
> impala::Coordinator::BackendState::InstanceStats::Update(impala::TFragmentInstanceExecStatus
> const&, impala::Coordinator::ExecSummary*, impala::ProgressUpdater*) ()
> #9 0x031369e6 in
> impala::Coordinator::BackendState::ApplyExecStatusReport(impala::TReportExecStatusParams
> const&, impala::Coordinator::ExecSummary*, impala::ProgressUpdater*) ()
> #10 0x031250b4 in
> impala::Coordinator::UpdateBackendExecStatus(impala::TReportExecStatusParams
> const&) ()
> #11 0x01e86395 in
> impala::ClientRequestState::UpdateBackendExecStatus(impala::TReportExecStatusParams
> const&) ()
> #12 0x01e27594 in
> impala::ImpalaServer::ReportExecStatus(impala::TReportExecStatusResult&,
> impala::TReportExecStatusParams const&) ()
> #13 0x01ebb8a0 in
> impala::ImpalaInternalService::ReportExecStatus(impala::TReportExecStatusResult&,
> impala::TReportExecStatusParams const&) ()
> #14 0x02fa6f62 in
> impala::ImpalaInternalServiceProcessor::process_ReportExecStatus(int,
> apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*,
> void*) ()
> #15 0x02fa6540 in
> impala::ImpalaInternalServiceProcessor::dispatchCall(apache::thrift::protocol::TProtocol*,
> apache::thrift::protocol::TProtocol*, std::string const&, int, void*) ()
> #16 0x018892b0 in
> apache::thrift::TDispatchProcessor::process(boost::shared_ptr,
> boost::shared_ptr, void*) ()
> #17 0x01c80d1b in
> apache::thrift::server::TAcceptQueueServer::Task::run() ()
> #18 0x01c78ffb in
> impala::ThriftThread::RunRunnable(boost::shared_ptr,
> impala::Promise*) ()
> #19 0x01c7a721 in boost::_mfi::mf2 boost::shared_ptr,
> impala::Promise (impala::PromiseMode)0>*>::operator()(impala::ThriftThread*,
> boost::shared_ptr,
> impala::Promise*) const ()
> #20 0x01c7a5b7 in void
> boost::_bi::list3,
> boost::_bi::value >,
> boost::_bi::value*>
> >::operator() boost::shared_ptr,
> impala::Promise*>,
> boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf2 impala::ThriftThread,
> boost::shared_ptr,
> impala::Promise*>&,
> boost::_bi::list0&, int) ()
> #21 0x01c7a303 in boost::_bi::bind_t impala::ThriftThread,
> boost::shared_ptr,
> impala::Promise*>,
> boost::_bi::list3,
> boost::_bi::value >,
> boost::_bi::value*> >
> >::operator()() ()
> #22 0x01c7a216 in
> boost::detail::function::void_function_obj_invoker0 boost::_mfi::mf2 boost::shared_ptr,
> impala::Promise*>,
> boost::_bi::list3,
> boost::_bi::value >,
> boost::_bi::value*> >
> >, void>::invoke(boost::detail::function::function_buffer&) ()
> #23 0x01bbb81c in boost::function0::operator()() const ()
> #24 0x01fb6eaf in impala::Thread::SuperviseThread(std::string const&,
> std::string const&, boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*) ()
> #25 0x01fbef87 in void
> boost::_bi::list5,
> boost::_bi::value, boost::_bi::value >,
> boost::_bi::value,
> boost::_bi::value*>
> >::operator() boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*),
> boost::_bi::list0>(boost::_bi::type, void (*&)(std::string const&,
> std::string const&, boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*), boost::_bi::list0&, int) ()
> #26
[jira] [Issue Comment Deleted] (IMPALA-6481) Impala Doc: WIDTH_BUCKET function
[ https://issues.apache.org/jira/browse/IMPALA-6481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni updated IMPALA-6481: Comment: was deleted (was: [~johnruss]) > Impala Doc: WIDTH_BUCKET function > - > > Key: IMPALA-6481 > URL: https://issues.apache.org/jira/browse/IMPALA-6481 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Alex Rodoni >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6481) Impala Doc: WIDTH_BUCKET function
[ https://issues.apache.org/jira/browse/IMPALA-6481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574003#comment-16574003 ] Alex Rodoni commented on IMPALA-6481: - https://gerrit.cloudera.org/#/c/11170/ > Impala Doc: WIDTH_BUCKET function > - > > Key: IMPALA-6481 > URL: https://issues.apache.org/jira/browse/IMPALA-6481 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Alex Rodoni >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7386) CatalogObjectVersionQueue should not be a queue
[ https://issues.apache.org/jira/browse/IMPALA-7386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573944#comment-16573944 ] ASF subversion and git services commented on IMPALA-7386: - Commit b9d44aff3ee5850dc7f08ba61091c959f08bc474 in impala's branch refs/heads/master from [~tlipcon] [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=b9d44af ] IMPALA-7386. Replace CatalogObjectVersionQueue with a multiset The implementation of CatalogObjectVersionQueue was based on a priority queue, which is not a good choice of data structure for the use case. This class exposes the following operations, with corresponding runtimes for the heap: - addVersion: O(lg n) - removeVersion: O(n) - getMinVersion: O(1) Given that every update of a catalog object requires removing the old version, the O(n) runtime could get quite expensive, particularly for operations like INVALIDATE METADATA which end up removing the old version of all of the objects in the catalog -- thus becoming accidentally quadratic. The new patch switches to a TreeMultiset coupled with a simple cache of the min element, with runtimes: - addVersion: O(lg n) - removeVersion: O(lg n) - getMinVersion: O(1) The downside of this patch is increased memory overhead: we are going from a PriorityQueue which has 8 bytes overhead per element to a TreeMultiset which has 48 bytes overhead per element. In a catalog with millions of tables this won't be insignificant; however, in such a catalog the performance savings are probably critical. According to a quick JMH benchmark, removal of an element from a PriorityQueue with 1M elements takes about 3.5ms, so an operation which invalidates all million tables would take about 3500 seconds without this patch. Meanwhile, the same benchmark using a TreeSet takes about 28ns per removal, so the whole invalidation would take about 28ms. This patch also makes the data structure synchronized, just for simplicity's sake, since uncontended locks are very cheap in Java. Change-Id: I1b6c58a1acef9b02fcc26a04d048ee9cc47dc0ef Reviewed-on: http://gerrit.cloudera.org:8080/11109 Tested-by: Todd Lipcon Reviewed-by: Todd Lipcon > CatalogObjectVersionQueue should not be a queue > --- > > Key: IMPALA-7386 > URL: https://issues.apache.org/jira/browse/IMPALA-7386 > Project: IMPALA > Issue Type: Bug > Components: Catalog >Affects Versions: Impala 3.0, Impala 2.12.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Major > Fix For: Impala 3.1.0 > > > CatalogObjectVersionQueue serves as a data structure to track all of the > currently in-use versions of the various objects in the catalog. The main > operations it needs are add(version), remove(version), and minVersion(). The > current implementation improperly uses a priority queue data structure, which > has O(n) runtime for remove(version). Instead we should use a tree-based > multiset datastructure which would have O(lgn) behavior for all operations, > potentially using a cache to retain O(1) minVersion(). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Reopened] (IMPALA-7347) Assertion Failure - test_show_create_table
[
https://issues.apache.org/jira/browse/IMPALA-7347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyi Wang reopened IMPALA-7347:
-
The fix didn't consider the case where the FS is local.
> Assertion Failure - test_show_create_table
> ---
>
> Key: IMPALA-7347
> URL: https://issues.apache.org/jira/browse/IMPALA-7347
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.1.0
>Reporter: nithya
>Assignee: Tianyi Wang
>Priority: Blocker
> Labels: build-failure
> Fix For: Impala 3.1.0
>
>
> test_show_create_table in metadata/test_show_create_table.py is failing with
> the following assertion error
> {noformat}
> metadata/test_show_create_table.py:58: in test_show_create_table
> unique_database)
> metadata/test_show_create_table.py:106: in __run_show_create_table_test_case
> self.__compare_result(expected_result, create_table_result)
> metadata/test_show_create_table.py:134: in __compare_result
> assert expected_tbl_props == actual_tbl_props
> E assert {} == {'numFilesErasureCoded': '0'}
> E Right contains more items:
> E {'numFilesErasureCoded': '0'}
> E Use -v to get the full diff
> {noformat}
>
> It appears that table property "numFilesErasureCoded" is showing up in table
> properties.
> Either the test needs updating or a bug.
>
> {code}
> h3. Error Message
> metadata/test_show_create_table.py:58: in test_show_create_table
> unique_database) metadata/test_show_create_table.py:106: in
> __run_show_create_table_test_case self.__compare_result(expected_result,
> create_table_result) metadata/test_show_create_table.py:134: in
> __compare_result assert expected_tbl_props == actual_tbl_props E assert {} ==
> \{'numFilesErasureCoded': '0'} E Right contains more items: E
> \{'numFilesErasureCoded': '0'} E Use -v to get the full diff
> {code}
>
> ---
> {code}
> h3. Standard Error
> -- connecting to: localhost:21000 SET sync_ddl=False; -- executing against
> localhost:21000 DROP DATABASE IF EXISTS `test_show_create_table_f1598d0b`
> CASCADE; SET sync_ddl=False; -- executing against localhost:21000 CREATE
> DATABASE `test_show_create_table_f1598d0b`; MainThread: Created database
> "test_show_create_table_f1598d0b" for test ID
> "metadata/test_show_create_table.py::TestShowCreateTable::()::test_show_create_table[table_format:
> text/none]" -- executing against localhost:21000 CREATE TABLE
> test_show_create_table_f1598d0b.test1 ( id INT ) STORED AS TEXTFILE; --
> executing against localhost:21000 show create table
> test_show_create_table_f1598d0b.test1; -- executing against localhost:21000
> drop table test_show_create_table_f1598d0b.test1; -- executing against
> localhost:21000 CREATE TABLE test_show_create_table_f1598d0b.test1 (id INT)
> STORED AS TEXTFILE LOCATION
> 'hdfs://localhost:20500/test-warehouse/test_show_create_table_f1598d0b.db/test1';
> -- executing against localhost:21000 show create table
> test_show_create_table_f1598d0b.test1; -- executing against localhost:21000
> drop table test_show_create_table_f1598d0b.test1; -- executing against
> localhost:21000 CREATE TABLE test_show_create_table_f1598d0b.test2 ( year
> INT, month INT, id INT COMMENT 'Add a comment', bool_col BOOLEAN, tinyint_col
> TINYINT, smallint_col SMALLINT, int_col INT, bigint_col BIGINT, float_col
> FLOAT, double_col DOUBLE, date_string_col STRING, string_col STRING,
> timestamp_col TIMESTAMP ) STORED AS TEXTFILE; -- executing against
> localhost:21000 show create table test_show_create_table_f1598d0b.test2; --
> executing against localhost:21000 drop table
> test_show_create_table_f1598d0b.test2; -- executing against localhost:21000
> CREATE TABLE test_show_create_table_f1598d0b.test2 (year INT, month INT, id
> INT COMMENT 'Add a comment', bool_col BOOLEAN, tinyint_col TINYINT,
> smallint_col SMALLINT, int_col INT, bigint_col BIGINT, float_col FLOAT,
> double_col DOUBLE, date_string_col STRING, string_col STRING, timestamp_col
> TIMESTAMP) STORED AS TEXTFILE LOCATION
> 'hdfs://localhost:20500/test-warehouse/test_show_create_table_f1598d0b.db/test2';
> -- executing against localhost:21000 show create table
> test_show_create_table_f1598d0b.test2; -- executing against localhost:21000
> drop table test_show_create_table_f1598d0b.test2; -- executing against
> localhost:21000 CREATE TABLE test_show_create_table_f1598d0b.test3 ( year
> INT, month INT, id INT COMMENT 'Add a comment', bool_col BOOLEAN, tinyint_col
> TINYINT, smallint_col SMALLINT, int_col INT, bigint_col BIGINT, float_col
> FLOAT, double_col DOUBLE, date_string_col STRING, string_col STRING,
> timestamp_col TIMESTAMP ) PARTITIONED BY ( x INT, y INT, a BOOLEAN ) COMMENT
> 'This is a test' STORED AS TEXTFILE; -- executing against localhost:21000
> show create table
[jira] [Resolved] (IMPALA-7413) test_incompatible_avro_partition_in_non_avro_table breaks in local mode
[
https://issues.apache.org/jira/browse/IMPALA-7413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon resolved IMPALA-7413.
-
Resolution: Duplicate
oops, I had filed IMPALA-7414 for the same issue.
> test_incompatible_avro_partition_in_non_avro_table breaks in local mode
> ---
>
> Key: IMPALA-7413
> URL: https://issues.apache.org/jira/browse/IMPALA-7413
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 3.1.0
>Reporter: Vuk Ercegovac
>Priority: Blocker
> Labels: broken-build
>
> New support for avro tables breaks when run in local mode:
>
> {noformat}
> metadata/test_partition_metadata.py:150: in
> test_incompatible_avro_partition_in_non_avro_table
> test_file_vars={'$MAIN_TABLE_FORMAT': main_table_format})
> common/impala_test_suite.py:424: in run_test_case
> assert False, "Expected exception: %s" % expected_str
> E AssertionError: Expected exception: Unresolvable types for column
> 'tinyint_col': declared column type: TINYINT, table's Avro schema type:
> int{noformat}
>
> Here's the change: [https://gerrit.cloudera.org/#/c/10970/]
> If the fix takes the same amount of time as a revert, lets fix it. Otherwise,
> lets revert it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7413) test_incompatible_avro_partition_in_non_avro_table breaks in local mode
[
https://issues.apache.org/jira/browse/IMPALA-7413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon resolved IMPALA-7413.
-
Resolution: Duplicate
oops, I had filed IMPALA-7414 for the same issue.
> test_incompatible_avro_partition_in_non_avro_table breaks in local mode
> ---
>
> Key: IMPALA-7413
> URL: https://issues.apache.org/jira/browse/IMPALA-7413
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 3.1.0
>Reporter: Vuk Ercegovac
>Priority: Blocker
> Labels: broken-build
>
> New support for avro tables breaks when run in local mode:
>
> {noformat}
> metadata/test_partition_metadata.py:150: in
> test_incompatible_avro_partition_in_non_avro_table
> test_file_vars={'$MAIN_TABLE_FORMAT': main_table_format})
> common/impala_test_suite.py:424: in run_test_case
> assert False, "Expected exception: %s" % expected_str
> E AssertionError: Expected exception: Unresolvable types for column
> 'tinyint_col': declared column type: TINYINT, table's Avro schema type:
> int{noformat}
>
> Here's the change: [https://gerrit.cloudera.org/#/c/10970/]
> If the fix takes the same amount of time as a revert, lets fix it. Otherwise,
> lets revert it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (IMPALA-7241) progress-updater.cc:43] Check failed: delta >= 0 (-3 vs. 0)
[
https://issues.apache.org/jira/browse/IMPALA-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573930#comment-16573930
]
Sailesh Mukil commented on IMPALA-7241:
---
[~tarmstrong] Sorry for the delayed response. Yes, it looks like there's an
existing bug there. [~kwho] found it and is fixing it as part of
IMPALA-7213
> progress-updater.cc:43] Check failed: delta >= 0 (-3 vs. 0)
> ---
>
> Key: IMPALA-7241
> URL: https://issues.apache.org/jira/browse/IMPALA-7241
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Michael Brown
>Assignee: Michael Ho
>Priority: Blocker
> Labels: crash, stress
>
> During a stress test with 8 nodes running an insecure debug build based off
> master, an impalad hit a DCHECK. The concurrency level at the time was
> between 150-180 queries.
> The DCHECK was {{progress-updater.cc:43] Check failed: delta >= 0 (-3 vs.
> 0)}}.
> The stack was:
> {noformat}
> #0 0x7f6a73e811f7 in raise () from /lib64/libc.so.6
> #1 0x7f6a73e828e8 in abort () from /lib64/libc.so.6
> #2 0x04300e34 in google::DumpStackTraceAndExit() ()
> #3 0x042f78ad in google::LogMessage::Fail() ()
> #4 0x042f9152 in google::LogMessage::SendToLog() ()
> #5 0x042f7287 in google::LogMessage::Flush() ()
> #6 0x042fa84e in google::LogMessageFatal::~LogMessageFatal() ()
> #7 0x01f7fedd in impala::ProgressUpdater::Update(long) ()
> #8 0x0313912b in
> impala::Coordinator::BackendState::InstanceStats::Update(impala::TFragmentInstanceExecStatus
> const&, impala::Coordinator::ExecSummary*, impala::ProgressUpdater*) ()
> #9 0x031369e6 in
> impala::Coordinator::BackendState::ApplyExecStatusReport(impala::TReportExecStatusParams
> const&, impala::Coordinator::ExecSummary*, impala::ProgressUpdater*) ()
> #10 0x031250b4 in
> impala::Coordinator::UpdateBackendExecStatus(impala::TReportExecStatusParams
> const&) ()
> #11 0x01e86395 in
> impala::ClientRequestState::UpdateBackendExecStatus(impala::TReportExecStatusParams
> const&) ()
> #12 0x01e27594 in
> impala::ImpalaServer::ReportExecStatus(impala::TReportExecStatusResult&,
> impala::TReportExecStatusParams const&) ()
> #13 0x01ebb8a0 in
> impala::ImpalaInternalService::ReportExecStatus(impala::TReportExecStatusResult&,
> impala::TReportExecStatusParams const&) ()
> #14 0x02fa6f62 in
> impala::ImpalaInternalServiceProcessor::process_ReportExecStatus(int,
> apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*,
> void*) ()
> #15 0x02fa6540 in
> impala::ImpalaInternalServiceProcessor::dispatchCall(apache::thrift::protocol::TProtocol*,
> apache::thrift::protocol::TProtocol*, std::string const&, int, void*) ()
> #16 0x018892b0 in
> apache::thrift::TDispatchProcessor::process(boost::shared_ptr,
> boost::shared_ptr, void*) ()
> #17 0x01c80d1b in
> apache::thrift::server::TAcceptQueueServer::Task::run() ()
> #18 0x01c78ffb in
> impala::ThriftThread::RunRunnable(boost::shared_ptr,
> impala::Promise*) ()
> #19 0x01c7a721 in boost::_mfi::mf2 boost::shared_ptr,
> impala::Promise (impala::PromiseMode)0>*>::operator()(impala::ThriftThread*,
> boost::shared_ptr,
> impala::Promise*) const ()
> #20 0x01c7a5b7 in void
> boost::_bi::list3,
> boost::_bi::value >,
> boost::_bi::value*>
> >::operator() boost::shared_ptr,
> impala::Promise*>,
> boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf2 impala::ThriftThread,
> boost::shared_ptr,
> impala::Promise*>&,
> boost::_bi::list0&, int) ()
> #21 0x01c7a303 in boost::_bi::bind_t impala::ThriftThread,
> boost::shared_ptr,
> impala::Promise*>,
> boost::_bi::list3,
> boost::_bi::value >,
> boost::_bi::value*> >
> >::operator()() ()
> #22 0x01c7a216 in
> boost::detail::function::void_function_obj_invoker0 boost::_mfi::mf2 boost::shared_ptr,
> impala::Promise*>,
> boost::_bi::list3,
> boost::_bi::value >,
> boost::_bi::value*> >
> >, void>::invoke(boost::detail::function::function_buffer&) ()
> #23 0x01bbb81c in boost::function0::operator()() const ()
> #24 0x01fb6eaf in impala::Thread::SuperviseThread(std::string const&,
> std::string const&, boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*) ()
> #25 0x01fbef87 in void
> boost::_bi::list5,
> boost::_bi::value, boost::_bi::value >,
> boost::_bi::value,
> boost::_bi::value*>
> >::operator() boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*),
> boost::_bi::list0>(boost::_bi::type, void (*&)(std::string const&,
> std::string const&, boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*),
[jira] [Comment Edited] (IMPALA-7241) progress-updater.cc:43] Check failed: delta >= 0 (-3 vs. 0)
[
https://issues.apache.org/jira/browse/IMPALA-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573930#comment-16573930
]
Sailesh Mukil edited comment on IMPALA-7241 at 8/8/18 9:42 PM:
---
[~tarmstrong] Sorry for the delayed response. Yes, it looks like there's an
existing bug there. [~kwho] found it and is fixing it as part of IMPALA-7213
was (Author: sailesh):
[~tarmstrong] Sorry for the delayed response. Yes, it looks like there's an
existing bug there. [~kwho] found it and is fixing it as part of
IMPALA-7213
> progress-updater.cc:43] Check failed: delta >= 0 (-3 vs. 0)
> ---
>
> Key: IMPALA-7241
> URL: https://issues.apache.org/jira/browse/IMPALA-7241
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Michael Brown
>Assignee: Michael Ho
>Priority: Blocker
> Labels: crash, stress
>
> During a stress test with 8 nodes running an insecure debug build based off
> master, an impalad hit a DCHECK. The concurrency level at the time was
> between 150-180 queries.
> The DCHECK was {{progress-updater.cc:43] Check failed: delta >= 0 (-3 vs.
> 0)}}.
> The stack was:
> {noformat}
> #0 0x7f6a73e811f7 in raise () from /lib64/libc.so.6
> #1 0x7f6a73e828e8 in abort () from /lib64/libc.so.6
> #2 0x04300e34 in google::DumpStackTraceAndExit() ()
> #3 0x042f78ad in google::LogMessage::Fail() ()
> #4 0x042f9152 in google::LogMessage::SendToLog() ()
> #5 0x042f7287 in google::LogMessage::Flush() ()
> #6 0x042fa84e in google::LogMessageFatal::~LogMessageFatal() ()
> #7 0x01f7fedd in impala::ProgressUpdater::Update(long) ()
> #8 0x0313912b in
> impala::Coordinator::BackendState::InstanceStats::Update(impala::TFragmentInstanceExecStatus
> const&, impala::Coordinator::ExecSummary*, impala::ProgressUpdater*) ()
> #9 0x031369e6 in
> impala::Coordinator::BackendState::ApplyExecStatusReport(impala::TReportExecStatusParams
> const&, impala::Coordinator::ExecSummary*, impala::ProgressUpdater*) ()
> #10 0x031250b4 in
> impala::Coordinator::UpdateBackendExecStatus(impala::TReportExecStatusParams
> const&) ()
> #11 0x01e86395 in
> impala::ClientRequestState::UpdateBackendExecStatus(impala::TReportExecStatusParams
> const&) ()
> #12 0x01e27594 in
> impala::ImpalaServer::ReportExecStatus(impala::TReportExecStatusResult&,
> impala::TReportExecStatusParams const&) ()
> #13 0x01ebb8a0 in
> impala::ImpalaInternalService::ReportExecStatus(impala::TReportExecStatusResult&,
> impala::TReportExecStatusParams const&) ()
> #14 0x02fa6f62 in
> impala::ImpalaInternalServiceProcessor::process_ReportExecStatus(int,
> apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*,
> void*) ()
> #15 0x02fa6540 in
> impala::ImpalaInternalServiceProcessor::dispatchCall(apache::thrift::protocol::TProtocol*,
> apache::thrift::protocol::TProtocol*, std::string const&, int, void*) ()
> #16 0x018892b0 in
> apache::thrift::TDispatchProcessor::process(boost::shared_ptr,
> boost::shared_ptr, void*) ()
> #17 0x01c80d1b in
> apache::thrift::server::TAcceptQueueServer::Task::run() ()
> #18 0x01c78ffb in
> impala::ThriftThread::RunRunnable(boost::shared_ptr,
> impala::Promise*) ()
> #19 0x01c7a721 in boost::_mfi::mf2 boost::shared_ptr,
> impala::Promise (impala::PromiseMode)0>*>::operator()(impala::ThriftThread*,
> boost::shared_ptr,
> impala::Promise*) const ()
> #20 0x01c7a5b7 in void
> boost::_bi::list3,
> boost::_bi::value >,
> boost::_bi::value*>
> >::operator() boost::shared_ptr,
> impala::Promise*>,
> boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf2 impala::ThriftThread,
> boost::shared_ptr,
> impala::Promise*>&,
> boost::_bi::list0&, int) ()
> #21 0x01c7a303 in boost::_bi::bind_t impala::ThriftThread,
> boost::shared_ptr,
> impala::Promise*>,
> boost::_bi::list3,
> boost::_bi::value >,
> boost::_bi::value*> >
> >::operator()() ()
> #22 0x01c7a216 in
> boost::detail::function::void_function_obj_invoker0 boost::_mfi::mf2 boost::shared_ptr,
> impala::Promise*>,
> boost::_bi::list3,
> boost::_bi::value >,
> boost::_bi::value*> >
> >, void>::invoke(boost::detail::function::function_buffer&) ()
> #23 0x01bbb81c in boost::function0::operator()() const ()
> #24 0x01fb6eaf in impala::Thread::SuperviseThread(std::string const&,
> std::string const&, boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*) ()
> #25 0x01fbef87 in void
> boost::_bi::list5,
> boost::_bi::value, boost::_bi::value >,
> boost::_bi::value,
> boost::_bi::value*>
> >::operator()
[jira] [Commented] (IMPALA-7386) CatalogObjectVersionQueue should not be a queue
[ https://issues.apache.org/jira/browse/IMPALA-7386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573929#comment-16573929 ] Todd Lipcon commented on IMPALA-7386: - Yea I think the extra memory cost of having the two data structures isn't worth the efficiency gain for this uncommon operation. Logarithmic updates are already pretty fast even with millions of objects. > CatalogObjectVersionQueue should not be a queue > --- > > Key: IMPALA-7386 > URL: https://issues.apache.org/jira/browse/IMPALA-7386 > Project: IMPALA > Issue Type: Bug > Components: Catalog >Affects Versions: Impala 3.0, Impala 2.12.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Major > Fix For: Impala 3.1.0 > > > CatalogObjectVersionQueue serves as a data structure to track all of the > currently in-use versions of the various objects in the catalog. The main > operations it needs are add(version), remove(version), and minVersion(). The > current implementation improperly uses a priority queue data structure, which > has O(n) runtime for remove(version). Instead we should use a tree-based > multiset datastructure which would have O(lgn) behavior for all operations, > potentially using a cache to retain O(1) minVersion(). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7406) Flatbuffer wrappers use almost as much memory as underlying data
[ https://issues.apache.org/jira/browse/IMPALA-7406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573925#comment-16573925 ] Todd Lipcon commented on IMPALA-7406: - I filed https://github.com/google/flatbuffers/issues/4865 with some measurements about different approaches to avoid the flyweight allocations at access time > Flatbuffer wrappers use almost as much memory as underlying data > > > Key: IMPALA-7406 > URL: https://issues.apache.org/jira/browse/IMPALA-7406 > Project: IMPALA > Issue Type: Improvement > Components: Catalog >Reporter: Todd Lipcon >Priority: Major > > Currently the file descriptors stored in the catalogd memory for each > partition use a FlatBuffer to reduce the number of separate objects on the > Java heap. However, the FlatBuffer objects internally each store a ByteBuffer > and int position, so each object takes 32 bytes on its own. The ByteBuffer > takes 56 bytes since it stores various references, endianness, limit, mark, > position, etc. This amounts to about 88 bytes overhead on top of the actual > underlying flatbuf byte array which is typically around 100 bytes for a > single-block file. So, we're have about a 1:1 ratio of memory overhead and a > 2:1 ratio of object count overhead for each partition. > If we simply stored the byte[] array and constructed wrappers on demand, we'd > save 88 bytes and 2 objects per partition. The downside is that we'd need to > do short-lived ByteBuffer allocations at access time, and based on some > benchmarking I did, they don't get escape-analyzed out. So, it's not a super > clear win, but still worth considering. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-6481) Impala Doc: WIDTH_BUCKET function
[ https://issues.apache.org/jira/browse/IMPALA-6481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-6481 started by Alex Rodoni. --- > Impala Doc: WIDTH_BUCKET function > - > > Key: IMPALA-6481 > URL: https://issues.apache.org/jira/browse/IMPALA-6481 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Alex Rodoni >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7386) CatalogObjectVersionQueue should not be a queue
[ https://issues.apache.org/jira/browse/IMPALA-7386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved IMPALA-7386. - Resolution: Fixed Fix Version/s: Impala 3.1.0 > CatalogObjectVersionQueue should not be a queue > --- > > Key: IMPALA-7386 > URL: https://issues.apache.org/jira/browse/IMPALA-7386 > Project: IMPALA > Issue Type: Bug > Components: Catalog >Affects Versions: Impala 3.0, Impala 2.12.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Major > Fix For: Impala 3.1.0 > > > CatalogObjectVersionQueue serves as a data structure to track all of the > currently in-use versions of the various objects in the catalog. The main > operations it needs are add(version), remove(version), and minVersion(). The > current implementation improperly uses a priority queue data structure, which > has O(n) runtime for remove(version). Instead we should use a tree-based > multiset datastructure which would have O(lgn) behavior for all operations, > potentially using a cache to retain O(1) minVersion(). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7386) CatalogObjectVersionQueue should not be a queue
[ https://issues.apache.org/jira/browse/IMPALA-7386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved IMPALA-7386. - Resolution: Fixed Fix Version/s: Impala 3.1.0 > CatalogObjectVersionQueue should not be a queue > --- > > Key: IMPALA-7386 > URL: https://issues.apache.org/jira/browse/IMPALA-7386 > Project: IMPALA > Issue Type: Bug > Components: Catalog >Affects Versions: Impala 3.0, Impala 2.12.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Major > Fix For: Impala 3.1.0 > > > CatalogObjectVersionQueue serves as a data structure to track all of the > currently in-use versions of the various objects in the catalog. The main > operations it needs are add(version), remove(version), and minVersion(). The > current implementation improperly uses a priority queue data structure, which > has O(n) runtime for remove(version). Instead we should use a tree-based > multiset datastructure which would have O(lgn) behavior for all operations, > potentially using a cache to retain O(1) minVersion(). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (IMPALA-6919) Impala 2.13 Doc: Doc the COMMENT ON statements
[ https://issues.apache.org/jira/browse/IMPALA-6919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni updated IMPALA-6919: Target Version: Impala 3.1.0 > Impala 2.13 Doc: Doc the COMMENT ON statements > -- > > Key: IMPALA-6919 > URL: https://issues.apache.org/jira/browse/IMPALA-6919 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Affects Versions: Impala 2.13.0, Impala 3.1.0 >Reporter: Alex Rodoni >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7403) AnalyticEvalNode does not manage BufferedTupleStream memory correctly
[
https://issues.apache.org/jira/browse/IMPALA-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-7403.
---
Resolution: Fixed
Fix Version/s: Impala 3.1.0
> AnalyticEvalNode does not manage BufferedTupleStream memory correctly
> -
>
> Key: IMPALA-7403
> URL: https://issues.apache.org/jira/browse/IMPALA-7403
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.5.0, Impala 2.6.0, Impala 2.7.0, Impala 2.8.0,
> Impala 2.9.0, Impala 2.10.0, Impala 2.11.0, Impala 3.0, Impala 2.12.0, Impala
> 3.1.0
>Reporter: Michael Brown
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: correctness, crash, query_generator
> Fix For: Impala 3.1.0
>
>
> The query generator has discovered a crash. I took one of the crashing
> queries and simplified it down as far as I could go. I searched for existing
> bugs with the same dcheck and didn't strictly see anything but I know there
> are other bugs right now and this may end up being a dupe.
> {noformat}
> reservation-tracker.cc:428] Check failed: used_reservation_ +
> child_reservations_ <= reservation_ (6291456 vs. 4194304)
> {noformat}
> {noformat}
> #6 0x0437961e in google::LogMessageFatal::~LogMessageFatal() ()
> #7 0x020636c6 in impala::ReservationTracker::CheckConsistency()
> const (this=0xe08da58) at
> /home/mikeb/Impala/be/src/runtime/bufferpool/reservation-tracker.cc:428
> #8 0x02062888 in
> impala::ReservationTracker::TransferReservationTo(impala::ReservationTracker*,
> long) (this=0xe08da58, other=0x11cae530, bytes=2097152) at
> /home/mikeb/Impala/be/src/runtime/bufferpool/reservation-tracker.cc:358
> #9 0x02057b10 in
> impala::BufferPool::ClientHandle::SaveReservation(impala::BufferPool::SubReservation*,
> long) (this=0xbf26190, dst=0xcfd40e8, bytes=2097152) at
> /home/mikeb/Impala/be/src/runtime/bufferpool/buffer-pool.cc:347
> #10 0x0318f209 in impala::BufferedTupleStream::NextReadPage()
> (this=0xcfd4000) at
> /home/mikeb/Impala/be/src/runtime/buffered-tuple-stream.cc:541
> #11 0x03195037 in impala::BufferedTupleStream::GetNextInternal false>(impala::RowBatch*, bool*, std::vector std::allocator >*) (this=0xcfd4000, batch=0x7efff313bc90,
> eos=0x7efff313c12f, flat_rows=0x0)
> at /home/mikeb/Impala/be/src/runtime/buffered-tuple-stream.cc:727
> #12 0x03192bf1 in
> impala::BufferedTupleStream::GetNextInternal(impala::RowBatch*, bool*,
> std::vector >*)
> (this=0xcfd4000, batch=0x7efff313bc90, eos=0x7efff313c12f, flat_rows=0x0)
> at /home/mikeb/Impala/be/src/runtime/buffered-tuple-stream.cc:698
> #13 0x0319035a in
> impala::BufferedTupleStream::GetNext(impala::RowBatch*, bool*)
> (this=0xcfd4000, batch=0x7efff313bc90, eos=0x7efff313c12f) at
> /home/mikeb/Impala/be/src/runtime/buffered-tuple-stream.cc:684
> #14 0x02e8ca0b in
> impala::AnalyticEvalNode::GetNextOutputBatch(impala::RuntimeState*,
> impala::RowBatch*, bool*) (this=0xbf26000, state=0xe575d40,
> output_batch=0xb98a5a0, eos=0x7efff313c12f)
> at /home/mikeb/Impala/be/src/exec/analytic-eval-node.cc:719
> #15 0x02e8dfe7 in
> impala::AnalyticEvalNode::GetNext(impala::RuntimeState*, impala::RowBatch*,
> bool*) (this=0xbf26000, state=0xe575d40, row_batch=0xb98a5a0, eos=0xbf27fd8)
> at /home/mikeb/Impala/be/src/exec/analytic-eval-node.cc:789
> #16 0x02e8b36a in
> impala::AnalyticEvalNode::ProcessChildBatches(impala::RuntimeState*)
> (this=0xbf27c00, state=0xe575d40) at
> /home/mikeb/Impala/be/src/exec/analytic-eval-node.cc:602
> #17 0x02e8df37 in
> impala::AnalyticEvalNode::GetNext(impala::RuntimeState*, impala::RowBatch*,
> bool*) (this=0xbf27c00, state=0xe575d40, row_batch=0xe904e40,
> eos=0x7efff313c5cf) at
> /home/mikeb/Impala/be/src/exec/analytic-eval-node.cc:786
> #18 0x01df28b8 in impala::FragmentInstanceState::ExecInternal()
> (this=0xbf39a00) at
> /home/mikeb/Impala/be/src/runtime/fragment-instance-state.cc:286
> #19 0x01defd35 in impala::FragmentInstanceState::Exec()
> (this=0xbf39a00) at
> /home/mikeb/Impala/be/src/runtime/fragment-instance-state.cc:90
> #20 0x01dff3ad in
> impala::QueryState::ExecFInstance(impala::FragmentInstanceState*)
> (this=0xd50a800, fis=0xbf39a00) at
> /home/mikeb/Impala/be/src/runtime/query-state.cc:401
> #21 0x01dfdb4c in impala::QueryStateoperator()(void)
> const (__closure=0x7efff313cca8) at
> /home/mikeb/Impala/be/src/runtime/query-state.cc:341
> #22 0x01e000cb in
> boost::detail::function::void_function_obj_invoker0,
> void>::invoke(boost::detail::function::function_buffer &)
> (function_obj_ptr=...)
> at
>
[jira] [Commented] (IMPALA-7403) AnalyticEvalNode does not manage BufferedTupleStream memory correctly
[
https://issues.apache.org/jira/browse/IMPALA-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573886#comment-16573886
]
ASF subversion and git services commented on IMPALA-7403:
-
Commit a6c356850bd4606f8506fa763d5f543229dce780 in impala's branch
refs/heads/master from [~tarmstr...@cloudera.com]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=a6c3568 ]
IMPALA-7403: fix child batch mem mgmt in analytic
The core of the fix is in ProcessChildBatches(), where we copy
'prev_input_tuple_' to 'prev_input_tuple_pool_' and reset
the child batch *before* the call to child(0)->GetNext(). This
solves a couple of problems:
* prev_input_tuple_ may be referencing memory from the child
that had the needs_deep_copy() flag set and therefore will
be freed or recycled when calling child(0)->GetNext() again.
* 'prev_child_batch_' may have been holding onto resources that
the child had flushed, which means they need to be freed before
the next GetNext() call.
Also refactors the logic around child_cmp_row_ to make the variable
lifetime and data flow clearer.
Testing:
Add regression test. The test passes both with this patch alone and with
IMPALA-7333 reapplied.
Change-Id: I09eb6213d47287f2addb72f8c1304085d2d48c55
Reviewed-on: http://gerrit.cloudera.org:8080/11155
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> AnalyticEvalNode does not manage BufferedTupleStream memory correctly
> -
>
> Key: IMPALA-7403
> URL: https://issues.apache.org/jira/browse/IMPALA-7403
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.5.0, Impala 2.6.0, Impala 2.7.0, Impala 2.8.0,
> Impala 2.9.0, Impala 2.10.0, Impala 2.11.0, Impala 3.0, Impala 2.12.0, Impala
> 3.1.0
>Reporter: Michael Brown
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: correctness, crash, query_generator
> Fix For: Impala 3.1.0
>
>
> The query generator has discovered a crash. I took one of the crashing
> queries and simplified it down as far as I could go. I searched for existing
> bugs with the same dcheck and didn't strictly see anything but I know there
> are other bugs right now and this may end up being a dupe.
> {noformat}
> reservation-tracker.cc:428] Check failed: used_reservation_ +
> child_reservations_ <= reservation_ (6291456 vs. 4194304)
> {noformat}
> {noformat}
> #6 0x0437961e in google::LogMessageFatal::~LogMessageFatal() ()
> #7 0x020636c6 in impala::ReservationTracker::CheckConsistency()
> const (this=0xe08da58) at
> /home/mikeb/Impala/be/src/runtime/bufferpool/reservation-tracker.cc:428
> #8 0x02062888 in
> impala::ReservationTracker::TransferReservationTo(impala::ReservationTracker*,
> long) (this=0xe08da58, other=0x11cae530, bytes=2097152) at
> /home/mikeb/Impala/be/src/runtime/bufferpool/reservation-tracker.cc:358
> #9 0x02057b10 in
> impala::BufferPool::ClientHandle::SaveReservation(impala::BufferPool::SubReservation*,
> long) (this=0xbf26190, dst=0xcfd40e8, bytes=2097152) at
> /home/mikeb/Impala/be/src/runtime/bufferpool/buffer-pool.cc:347
> #10 0x0318f209 in impala::BufferedTupleStream::NextReadPage()
> (this=0xcfd4000) at
> /home/mikeb/Impala/be/src/runtime/buffered-tuple-stream.cc:541
> #11 0x03195037 in impala::BufferedTupleStream::GetNextInternal false>(impala::RowBatch*, bool*, std::vector std::allocator >*) (this=0xcfd4000, batch=0x7efff313bc90,
> eos=0x7efff313c12f, flat_rows=0x0)
> at /home/mikeb/Impala/be/src/runtime/buffered-tuple-stream.cc:727
> #12 0x03192bf1 in
> impala::BufferedTupleStream::GetNextInternal(impala::RowBatch*, bool*,
> std::vector >*)
> (this=0xcfd4000, batch=0x7efff313bc90, eos=0x7efff313c12f, flat_rows=0x0)
> at /home/mikeb/Impala/be/src/runtime/buffered-tuple-stream.cc:698
> #13 0x0319035a in
> impala::BufferedTupleStream::GetNext(impala::RowBatch*, bool*)
> (this=0xcfd4000, batch=0x7efff313bc90, eos=0x7efff313c12f) at
> /home/mikeb/Impala/be/src/runtime/buffered-tuple-stream.cc:684
> #14 0x02e8ca0b in
> impala::AnalyticEvalNode::GetNextOutputBatch(impala::RuntimeState*,
> impala::RowBatch*, bool*) (this=0xbf26000, state=0xe575d40,
> output_batch=0xb98a5a0, eos=0x7efff313c12f)
> at /home/mikeb/Impala/be/src/exec/analytic-eval-node.cc:719
> #15 0x02e8dfe7 in
> impala::AnalyticEvalNode::GetNext(impala::RuntimeState*, impala::RowBatch*,
> bool*) (this=0xbf26000, state=0xe575d40, row_batch=0xb98a5a0, eos=0xbf27fd8)
> at /home/mikeb/Impala/be/src/exec/analytic-eval-node.cc:789
> #16 0x02e8b36a in
> impala::AnalyticEvalNode::ProcessChildBatches(impala::RuntimeState*)
> (this=0xbf27c00, state=0xe575d40) at
>
[jira] [Commented] (IMPALA-7400) "SQL Statements to Remove or Adapt" is out of date
[
https://issues.apache.org/jira/browse/IMPALA-7400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573885#comment-16573885
]
ASF subversion and git services commented on IMPALA-7400:
-
Commit dd8a25d4c2976bddbe40cf7b37a2b69447855f83 in impala's branch
refs/heads/master from [~arodoni_cloudera]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=dd8a25d ]
IMPALA-7400: [DOCS] Updated outdated contents from impala_porting
Change-Id: I3a1edb1ec5076354df9301ce12e7618d7cf607a6
Reviewed-on: http://gerrit.cloudera.org:8080/11154
Tested-by: Impala Public Jenkins
Reviewed-by: Tim Armstrong
> "SQL Statements to Remove or Adapt" is out of date
> --
>
> Key: IMPALA-7400
> URL: https://issues.apache.org/jira/browse/IMPALA-7400
> Project: IMPALA
> Issue Type: Bug
> Components: Docs
>Affects Versions: Impala 3.0
>Reporter: Tim Armstrong
>Assignee: Alex Rodoni
>Priority: Major
> Labels: docs
> Fix For: Impala 3.1.0
>
>
> "Impala has no DELETE statement." and "Impala has no UPDATE statement. " are
> not totally true - Impala has those statements but only for Kudu tables.
> "For example, Impala does not support natural joins or anti-joins," - Impala
> does support Anti-joins via NOT IN/NOT EXISTS or even explicitly like:
> {code}
> select * from functional.alltypes a1 left anti join functional.alltypestiny
> a2 on a1.id = a2.id;
> {code}
> "Within queries, Impala requires query aliases for any subqueries:" - this is
> only true for subqueries used as inline views in the FROM clause. E.g. the
> following works:
> {code}
> select * from functional.alltypes where id = (select min(id) from
> functional.alltypes);
> {code}
> " Impala .. requires the CROSS JOIN operator for Cartesian products." -
> untrue, this works:
> {code}
> select * from functional.alltypes t1, functional.alltypes t2;
> {code}
> "Have you run the COMPUTE STATS statement on each table involved in join
> queries". This isn't specific to queries with joins, although may have more
> impact. We recommend that users run COMPUTE STATS on all tables.
> "A CREATE TABLE statement with no PARTITIONED BY clause stores all the data
> files in the same physical location," - unpartitioned tables with multiple
> files can have files residing in different locations (and there are already 3
> replicas per file by default, so the statement is a little misleading even if
> there's a single file). I think the latest statement about "Have you
> partitioned at the right granularity so that there is enough data in each
> partition to parallelize the work for each query?" is also misleading for the
> same reason.
> "The INSERT ... VALUES syntax is suitable for setting up toy tables with a
> few rows for functional testing, but because each such statement creates a
> separate tiny file in HDFS". This advice only applies to HDFS, this should
> work fine for Kudu tables although the INSERT statements are not particularly
> fast.
> "The number of expressions allowed in an Impala query might be smaller than
> for some other database systems, causing failures for very complicated
> queries" - this doesn't seem right - I don't know why the queries would fail.
> Also the codegen time isn't really specific to expressions or where clauses.
> There seems to be a point buried in there, but maybe it's just essentially
> that "Complex queries may have high codegen time"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7333) Remove MarkNeedsDeepCopy from Aggregation and Hash Join Nodes
[ https://issues.apache.org/jira/browse/IMPALA-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573887#comment-16573887 ] ASF subversion and git services commented on IMPALA-7333: - Commit a6c356850bd4606f8506fa763d5f543229dce780 in impala's branch refs/heads/master from [~tarmstr...@cloudera.com] [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=a6c3568 ] IMPALA-7403: fix child batch mem mgmt in analytic The core of the fix is in ProcessChildBatches(), where we copy 'prev_input_tuple_' to 'prev_input_tuple_pool_' and reset the child batch *before* the call to child(0)->GetNext(). This solves a couple of problems: * prev_input_tuple_ may be referencing memory from the child that had the needs_deep_copy() flag set and therefore will be freed or recycled when calling child(0)->GetNext() again. * 'prev_child_batch_' may have been holding onto resources that the child had flushed, which means they need to be freed before the next GetNext() call. Also refactors the logic around child_cmp_row_ to make the variable lifetime and data flow clearer. Testing: Add regression test. The test passes both with this patch alone and with IMPALA-7333 reapplied. Change-Id: I09eb6213d47287f2addb72f8c1304085d2d48c55 Reviewed-on: http://gerrit.cloudera.org:8080/11155 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Remove MarkNeedsDeepCopy from Aggregation and Hash Join Nodes > - > > Key: IMPALA-7333 > URL: https://issues.apache.org/jira/browse/IMPALA-7333 > Project: IMPALA > Issue Type: Sub-task > Components: Backend >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > Labels: resource-management > Fix For: Impala 3.1.0 > > > The main part of this is fixing BufferedTupleStream. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7403) AnalyticEvalNode does not manage BufferedTupleStream memory correctly
[
https://issues.apache.org/jira/browse/IMPALA-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-7403.
---
Resolution: Fixed
Fix Version/s: Impala 3.1.0
> AnalyticEvalNode does not manage BufferedTupleStream memory correctly
> -
>
> Key: IMPALA-7403
> URL: https://issues.apache.org/jira/browse/IMPALA-7403
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.5.0, Impala 2.6.0, Impala 2.7.0, Impala 2.8.0,
> Impala 2.9.0, Impala 2.10.0, Impala 2.11.0, Impala 3.0, Impala 2.12.0, Impala
> 3.1.0
>Reporter: Michael Brown
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: correctness, crash, query_generator
> Fix For: Impala 3.1.0
>
>
> The query generator has discovered a crash. I took one of the crashing
> queries and simplified it down as far as I could go. I searched for existing
> bugs with the same dcheck and didn't strictly see anything but I know there
> are other bugs right now and this may end up being a dupe.
> {noformat}
> reservation-tracker.cc:428] Check failed: used_reservation_ +
> child_reservations_ <= reservation_ (6291456 vs. 4194304)
> {noformat}
> {noformat}
> #6 0x0437961e in google::LogMessageFatal::~LogMessageFatal() ()
> #7 0x020636c6 in impala::ReservationTracker::CheckConsistency()
> const (this=0xe08da58) at
> /home/mikeb/Impala/be/src/runtime/bufferpool/reservation-tracker.cc:428
> #8 0x02062888 in
> impala::ReservationTracker::TransferReservationTo(impala::ReservationTracker*,
> long) (this=0xe08da58, other=0x11cae530, bytes=2097152) at
> /home/mikeb/Impala/be/src/runtime/bufferpool/reservation-tracker.cc:358
> #9 0x02057b10 in
> impala::BufferPool::ClientHandle::SaveReservation(impala::BufferPool::SubReservation*,
> long) (this=0xbf26190, dst=0xcfd40e8, bytes=2097152) at
> /home/mikeb/Impala/be/src/runtime/bufferpool/buffer-pool.cc:347
> #10 0x0318f209 in impala::BufferedTupleStream::NextReadPage()
> (this=0xcfd4000) at
> /home/mikeb/Impala/be/src/runtime/buffered-tuple-stream.cc:541
> #11 0x03195037 in impala::BufferedTupleStream::GetNextInternal false>(impala::RowBatch*, bool*, std::vector std::allocator >*) (this=0xcfd4000, batch=0x7efff313bc90,
> eos=0x7efff313c12f, flat_rows=0x0)
> at /home/mikeb/Impala/be/src/runtime/buffered-tuple-stream.cc:727
> #12 0x03192bf1 in
> impala::BufferedTupleStream::GetNextInternal(impala::RowBatch*, bool*,
> std::vector >*)
> (this=0xcfd4000, batch=0x7efff313bc90, eos=0x7efff313c12f, flat_rows=0x0)
> at /home/mikeb/Impala/be/src/runtime/buffered-tuple-stream.cc:698
> #13 0x0319035a in
> impala::BufferedTupleStream::GetNext(impala::RowBatch*, bool*)
> (this=0xcfd4000, batch=0x7efff313bc90, eos=0x7efff313c12f) at
> /home/mikeb/Impala/be/src/runtime/buffered-tuple-stream.cc:684
> #14 0x02e8ca0b in
> impala::AnalyticEvalNode::GetNextOutputBatch(impala::RuntimeState*,
> impala::RowBatch*, bool*) (this=0xbf26000, state=0xe575d40,
> output_batch=0xb98a5a0, eos=0x7efff313c12f)
> at /home/mikeb/Impala/be/src/exec/analytic-eval-node.cc:719
> #15 0x02e8dfe7 in
> impala::AnalyticEvalNode::GetNext(impala::RuntimeState*, impala::RowBatch*,
> bool*) (this=0xbf26000, state=0xe575d40, row_batch=0xb98a5a0, eos=0xbf27fd8)
> at /home/mikeb/Impala/be/src/exec/analytic-eval-node.cc:789
> #16 0x02e8b36a in
> impala::AnalyticEvalNode::ProcessChildBatches(impala::RuntimeState*)
> (this=0xbf27c00, state=0xe575d40) at
> /home/mikeb/Impala/be/src/exec/analytic-eval-node.cc:602
> #17 0x02e8df37 in
> impala::AnalyticEvalNode::GetNext(impala::RuntimeState*, impala::RowBatch*,
> bool*) (this=0xbf27c00, state=0xe575d40, row_batch=0xe904e40,
> eos=0x7efff313c5cf) at
> /home/mikeb/Impala/be/src/exec/analytic-eval-node.cc:786
> #18 0x01df28b8 in impala::FragmentInstanceState::ExecInternal()
> (this=0xbf39a00) at
> /home/mikeb/Impala/be/src/runtime/fragment-instance-state.cc:286
> #19 0x01defd35 in impala::FragmentInstanceState::Exec()
> (this=0xbf39a00) at
> /home/mikeb/Impala/be/src/runtime/fragment-instance-state.cc:90
> #20 0x01dff3ad in
> impala::QueryState::ExecFInstance(impala::FragmentInstanceState*)
> (this=0xd50a800, fis=0xbf39a00) at
> /home/mikeb/Impala/be/src/runtime/query-state.cc:401
> #21 0x01dfdb4c in impala::QueryStateoperator()(void)
> const (__closure=0x7efff313cca8) at
> /home/mikeb/Impala/be/src/runtime/query-state.cc:341
> #22 0x01e000cb in
> boost::detail::function::void_function_obj_invoker0,
> void>::invoke(boost::detail::function::function_buffer &)
> (function_obj_ptr=...)
> at
>
[jira] [Closed] (IMPALA-7195) Impala 3.1 Doc: Proxy user list should support groups
[ https://issues.apache.org/jira/browse/IMPALA-7195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni closed IMPALA-7195. --- Resolution: Fixed Fix Version/s: Impala 3.1.0 > Impala 3.1 Doc: Proxy user list should support groups > - > > Key: IMPALA-7195 > URL: https://issues.apache.org/jira/browse/IMPALA-7195 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Alex Rodoni >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc > Fix For: Impala 3.1.0 > > > https://gerrit.cloudera.org/#/c/11068/ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Closed] (IMPALA-7195) Impala 3.1 Doc: Proxy user list should support groups
[ https://issues.apache.org/jira/browse/IMPALA-7195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni closed IMPALA-7195. --- Resolution: Fixed Fix Version/s: Impala 3.1.0 > Impala 3.1 Doc: Proxy user list should support groups > - > > Key: IMPALA-7195 > URL: https://issues.apache.org/jira/browse/IMPALA-7195 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Alex Rodoni >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc > Fix For: Impala 3.1.0 > > > https://gerrit.cloudera.org/#/c/11068/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (IMPALA-7400) "SQL Statements to Remove or Adapt" is out of date
[
https://issues.apache.org/jira/browse/IMPALA-7400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Rodoni closed IMPALA-7400.
---
Resolution: Fixed
Fix Version/s: Impala 3.1.0
> "SQL Statements to Remove or Adapt" is out of date
> --
>
> Key: IMPALA-7400
> URL: https://issues.apache.org/jira/browse/IMPALA-7400
> Project: IMPALA
> Issue Type: Bug
> Components: Docs
>Affects Versions: Impala 3.0
>Reporter: Tim Armstrong
>Assignee: Alex Rodoni
>Priority: Major
> Labels: docs
> Fix For: Impala 3.1.0
>
>
> "Impala has no DELETE statement." and "Impala has no UPDATE statement. " are
> not totally true - Impala has those statements but only for Kudu tables.
> "For example, Impala does not support natural joins or anti-joins," - Impala
> does support Anti-joins via NOT IN/NOT EXISTS or even explicitly like:
> {code}
> select * from functional.alltypes a1 left anti join functional.alltypestiny
> a2 on a1.id = a2.id;
> {code}
> "Within queries, Impala requires query aliases for any subqueries:" - this is
> only true for subqueries used as inline views in the FROM clause. E.g. the
> following works:
> {code}
> select * from functional.alltypes where id = (select min(id) from
> functional.alltypes);
> {code}
> " Impala .. requires the CROSS JOIN operator for Cartesian products." -
> untrue, this works:
> {code}
> select * from functional.alltypes t1, functional.alltypes t2;
> {code}
> "Have you run the COMPUTE STATS statement on each table involved in join
> queries". This isn't specific to queries with joins, although may have more
> impact. We recommend that users run COMPUTE STATS on all tables.
> "A CREATE TABLE statement with no PARTITIONED BY clause stores all the data
> files in the same physical location," - unpartitioned tables with multiple
> files can have files residing in different locations (and there are already 3
> replicas per file by default, so the statement is a little misleading even if
> there's a single file). I think the latest statement about "Have you
> partitioned at the right granularity so that there is enough data in each
> partition to parallelize the work for each query?" is also misleading for the
> same reason.
> "The INSERT ... VALUES syntax is suitable for setting up toy tables with a
> few rows for functional testing, but because each such statement creates a
> separate tiny file in HDFS". This advice only applies to HDFS, this should
> work fine for Kudu tables although the INSERT statements are not particularly
> fast.
> "The number of expressions allowed in an Impala query might be smaller than
> for some other database systems, causing failures for very complicated
> queries" - this doesn't seem right - I don't know why the queries would fail.
> Also the codegen time isn't really specific to expressions or where clauses.
> There seems to be a point buried in there, but maybe it's just essentially
> that "Complex queries may have high codegen time"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (IMPALA-7400) "SQL Statements to Remove or Adapt" is out of date
[
https://issues.apache.org/jira/browse/IMPALA-7400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Rodoni closed IMPALA-7400.
---
Resolution: Fixed
Fix Version/s: Impala 3.1.0
> "SQL Statements to Remove or Adapt" is out of date
> --
>
> Key: IMPALA-7400
> URL: https://issues.apache.org/jira/browse/IMPALA-7400
> Project: IMPALA
> Issue Type: Bug
> Components: Docs
>Affects Versions: Impala 3.0
>Reporter: Tim Armstrong
>Assignee: Alex Rodoni
>Priority: Major
> Labels: docs
> Fix For: Impala 3.1.0
>
>
> "Impala has no DELETE statement." and "Impala has no UPDATE statement. " are
> not totally true - Impala has those statements but only for Kudu tables.
> "For example, Impala does not support natural joins or anti-joins," - Impala
> does support Anti-joins via NOT IN/NOT EXISTS or even explicitly like:
> {code}
> select * from functional.alltypes a1 left anti join functional.alltypestiny
> a2 on a1.id = a2.id;
> {code}
> "Within queries, Impala requires query aliases for any subqueries:" - this is
> only true for subqueries used as inline views in the FROM clause. E.g. the
> following works:
> {code}
> select * from functional.alltypes where id = (select min(id) from
> functional.alltypes);
> {code}
> " Impala .. requires the CROSS JOIN operator for Cartesian products." -
> untrue, this works:
> {code}
> select * from functional.alltypes t1, functional.alltypes t2;
> {code}
> "Have you run the COMPUTE STATS statement on each table involved in join
> queries". This isn't specific to queries with joins, although may have more
> impact. We recommend that users run COMPUTE STATS on all tables.
> "A CREATE TABLE statement with no PARTITIONED BY clause stores all the data
> files in the same physical location," - unpartitioned tables with multiple
> files can have files residing in different locations (and there are already 3
> replicas per file by default, so the statement is a little misleading even if
> there's a single file). I think the latest statement about "Have you
> partitioned at the right granularity so that there is enough data in each
> partition to parallelize the work for each query?" is also misleading for the
> same reason.
> "The INSERT ... VALUES syntax is suitable for setting up toy tables with a
> few rows for functional testing, but because each such statement creates a
> separate tiny file in HDFS". This advice only applies to HDFS, this should
> work fine for Kudu tables although the INSERT statements are not particularly
> fast.
> "The number of expressions allowed in an Impala query might be smaller than
> for some other database systems, causing failures for very complicated
> queries" - this doesn't seem right - I don't know why the queries would fail.
> Also the codegen time isn't really specific to expressions or where clauses.
> There seems to be a point buried in there, but maybe it's just essentially
> that "Complex queries may have high codegen time"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7400) "SQL Statements to Remove or Adapt" is out of date
[
https://issues.apache.org/jira/browse/IMPALA-7400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573828#comment-16573828
]
Alex Rodoni commented on IMPALA-7400:
-
https://gerrit.cloudera.org/#/c/11154/
> "SQL Statements to Remove or Adapt" is out of date
> --
>
> Key: IMPALA-7400
> URL: https://issues.apache.org/jira/browse/IMPALA-7400
> Project: IMPALA
> Issue Type: Bug
> Components: Docs
>Affects Versions: Impala 3.0
>Reporter: Tim Armstrong
>Assignee: Alex Rodoni
>Priority: Major
> Labels: docs
>
> "Impala has no DELETE statement." and "Impala has no UPDATE statement. " are
> not totally true - Impala has those statements but only for Kudu tables.
> "For example, Impala does not support natural joins or anti-joins," - Impala
> does support Anti-joins via NOT IN/NOT EXISTS or even explicitly like:
> {code}
> select * from functional.alltypes a1 left anti join functional.alltypestiny
> a2 on a1.id = a2.id;
> {code}
> "Within queries, Impala requires query aliases for any subqueries:" - this is
> only true for subqueries used as inline views in the FROM clause. E.g. the
> following works:
> {code}
> select * from functional.alltypes where id = (select min(id) from
> functional.alltypes);
> {code}
> " Impala .. requires the CROSS JOIN operator for Cartesian products." -
> untrue, this works:
> {code}
> select * from functional.alltypes t1, functional.alltypes t2;
> {code}
> "Have you run the COMPUTE STATS statement on each table involved in join
> queries". This isn't specific to queries with joins, although may have more
> impact. We recommend that users run COMPUTE STATS on all tables.
> "A CREATE TABLE statement with no PARTITIONED BY clause stores all the data
> files in the same physical location," - unpartitioned tables with multiple
> files can have files residing in different locations (and there are already 3
> replicas per file by default, so the statement is a little misleading even if
> there's a single file). I think the latest statement about "Have you
> partitioned at the right granularity so that there is enough data in each
> partition to parallelize the work for each query?" is also misleading for the
> same reason.
> "The INSERT ... VALUES syntax is suitable for setting up toy tables with a
> few rows for functional testing, but because each such statement creates a
> separate tiny file in HDFS". This advice only applies to HDFS, this should
> work fine for Kudu tables although the INSERT statements are not particularly
> fast.
> "The number of expressions allowed in an Impala query might be smaller than
> for some other database systems, causing failures for very complicated
> queries" - this doesn't seem right - I don't know why the queries would fail.
> Also the codegen time isn't really specific to expressions or where clauses.
> There seems to be a point buried in there, but maybe it's just essentially
> that "Complex queries may have high codegen time"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7394) Don't print stack trace in ImpalaServer::ExpireSessions()
[
https://issues.apache.org/jira/browse/IMPALA-7394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Ho resolved IMPALA-7394.
Resolution: Fixed
Fix Version/s: Impala 3.1.0
> Don't print stack trace in ImpalaServer::ExpireSessions()
> --
>
> Key: IMPALA-7394
> URL: https://issues.apache.org/jira/browse/IMPALA-7394
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Michael Ho
>Assignee: Michael Ho
>Priority: Minor
> Fix For: Impala 3.1.0
>
>
> It appears that session expiration can result in huge amount of log spew like
> the following. While IMPALA-7014 reduces the overhead by avoiding the
> symbolization, the log spew still seems unnecessary.
> {noformat}
> I0730 13:00:10.241578 400820 status.cc:125] Session expired due to inactivity
> @ 0x96678a impala::Status::Status()
> @ 0xc00d3c impala::ImpalaServer::ExpireSessions()
> @ 0xd6101f impala::Thread::SuperviseThread()
> @ 0xd6181a boost::detail::thread_data<>::run()
> @ 0x12d85ca (unknown)
> @ 0x7f16285fbdc5 start_thread
> @ 0x7f162832c17d __clone
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7394) Don't print stack trace in ImpalaServer::ExpireSessions()
[
https://issues.apache.org/jira/browse/IMPALA-7394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Ho resolved IMPALA-7394.
Resolution: Fixed
Fix Version/s: Impala 3.1.0
> Don't print stack trace in ImpalaServer::ExpireSessions()
> --
>
> Key: IMPALA-7394
> URL: https://issues.apache.org/jira/browse/IMPALA-7394
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Michael Ho
>Assignee: Michael Ho
>Priority: Minor
> Fix For: Impala 3.1.0
>
>
> It appears that session expiration can result in huge amount of log spew like
> the following. While IMPALA-7014 reduces the overhead by avoiding the
> symbolization, the log spew still seems unnecessary.
> {noformat}
> I0730 13:00:10.241578 400820 status.cc:125] Session expired due to inactivity
> @ 0x96678a impala::Status::Status()
> @ 0xc00d3c impala::ImpalaServer::ExpireSessions()
> @ 0xd6101f impala::Thread::SuperviseThread()
> @ 0xd6181a boost::detail::thread_data<>::run()
> @ 0x12d85ca (unknown)
> @ 0x7f16285fbdc5 start_thread
> @ 0x7f162832c17d __clone
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (IMPALA-7163) Implement a state machine for the QueryState class
[ https://issues.apache.org/jira/browse/IMPALA-7163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sailesh Mukil resolved IMPALA-7163. --- Resolution: Fixed Fix Version/s: Impala 3.1.0 > Implement a state machine for the QueryState class > -- > > Key: IMPALA-7163 > URL: https://issues.apache.org/jira/browse/IMPALA-7163 > Project: IMPALA > Issue Type: Sub-task > Components: Distributed Exec >Reporter: Sailesh Mukil >Assignee: Sailesh Mukil >Priority: Major > Fix For: Impala 3.1.0 > > > We've recently been improving our query lifecycle by adding explicit state > transitions so it's easier to reason about what should happen at a given > stage in the lifetime of a query or a fragment instance. > On the coordinator side. The coordinator's view of the query: > https://github.com/apache/impala/commit/6ca87e46736a1e591ed7d7d5fee05b4b4d2fbb50 > On the fragment instance state side. A FIS's view of its own execution: > https://github.com/apache/impala/blob/e12ee485cf4c77203b144c053ee167509cc39374/be/src/runtime/fragment-instance-state.h#L182-L203 > We don't have something like this for the QueryState class which maintains > query wide state per executor. Adding it should make the lifecycle of a query > from an executors point of view much easier to reason about. > Additional info: This was identified as part of work for > IMPALA-2990/IMPALA-4063, and is a precursor to it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7163) Implement a state machine for the QueryState class
[ https://issues.apache.org/jira/browse/IMPALA-7163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sailesh Mukil resolved IMPALA-7163. --- Resolution: Fixed Fix Version/s: Impala 3.1.0 > Implement a state machine for the QueryState class > -- > > Key: IMPALA-7163 > URL: https://issues.apache.org/jira/browse/IMPALA-7163 > Project: IMPALA > Issue Type: Sub-task > Components: Distributed Exec >Reporter: Sailesh Mukil >Assignee: Sailesh Mukil >Priority: Major > Fix For: Impala 3.1.0 > > > We've recently been improving our query lifecycle by adding explicit state > transitions so it's easier to reason about what should happen at a given > stage in the lifetime of a query or a fragment instance. > On the coordinator side. The coordinator's view of the query: > https://github.com/apache/impala/commit/6ca87e46736a1e591ed7d7d5fee05b4b4d2fbb50 > On the fragment instance state side. A FIS's view of its own execution: > https://github.com/apache/impala/blob/e12ee485cf4c77203b144c053ee167509cc39374/be/src/runtime/fragment-instance-state.h#L182-L203 > We don't have something like this for the QueryState class which maintains > query wide state per executor. Adding it should make the lifecycle of a query > from an executors point of view much easier to reason about. > Additional info: This was identified as part of work for > IMPALA-2990/IMPALA-4063, and is a precursor to it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (IMPALA-7376) Impala hits a DCHECK if a fragment instance fails to initialize the filter bank
[ https://issues.apache.org/jira/browse/IMPALA-7376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sailesh Mukil resolved IMPALA-7376. --- Resolution: Fixed Fix Version/s: Impala 3.1.0 > Impala hits a DCHECK if a fragment instance fails to initialize the filter > bank > --- > > Key: IMPALA-7376 > URL: https://issues.apache.org/jira/browse/IMPALA-7376 > Project: IMPALA > Issue Type: Bug > Components: Backend >Reporter: Sailesh Mukil >Assignee: Sailesh Mukil >Priority: Critical > Labels: crash > Fix For: Impala 3.1.0 > > > While Prepare()-ing a fragment instance, if we fail to initialize the runtime > filter bank, we will exit FIS::Prepare() without acquiring a thread token > (AcquireThreadToken()): > https://github.com/apache/impala/blob/316b17ac55adb3d1deeb1289b4045688269b201d/be/src/runtime/fragment-instance-state.cc#L135-L139 > FIS::Finalize() is called *always* regardless of whether the fragment > instance succeeded or failed. And FIS::Finalize() tries to > ReleaseThreadToken() even though it might not have gotten acquired: > https://github.com/apache/impala/blob/316b17ac55adb3d1deeb1289b4045688269b201d/be/src/runtime/fragment-instance-state.cc#L464 > , causing a DCHECK to be hit. > This was found while I was adding global debug actions (IMPALA-7046) to the > FIS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7376) Impala hits a DCHECK if a fragment instance fails to initialize the filter bank
[ https://issues.apache.org/jira/browse/IMPALA-7376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sailesh Mukil resolved IMPALA-7376. --- Resolution: Fixed Fix Version/s: Impala 3.1.0 > Impala hits a DCHECK if a fragment instance fails to initialize the filter > bank > --- > > Key: IMPALA-7376 > URL: https://issues.apache.org/jira/browse/IMPALA-7376 > Project: IMPALA > Issue Type: Bug > Components: Backend >Reporter: Sailesh Mukil >Assignee: Sailesh Mukil >Priority: Critical > Labels: crash > Fix For: Impala 3.1.0 > > > While Prepare()-ing a fragment instance, if we fail to initialize the runtime > filter bank, we will exit FIS::Prepare() without acquiring a thread token > (AcquireThreadToken()): > https://github.com/apache/impala/blob/316b17ac55adb3d1deeb1289b4045688269b201d/be/src/runtime/fragment-instance-state.cc#L135-L139 > FIS::Finalize() is called *always* regardless of whether the fragment > instance succeeded or failed. And FIS::Finalize() tries to > ReleaseThreadToken() even though it might not have gotten acquired: > https://github.com/apache/impala/blob/316b17ac55adb3d1deeb1289b4045688269b201d/be/src/runtime/fragment-instance-state.cc#L464 > , causing a DCHECK to be hit. > This was found while I was adding global debug actions (IMPALA-7046) to the > FIS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IMPALA-7414) Recently added tests fail on localfs build
[ https://issues.apache.org/jira/browse/IMPALA-7414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573602#comment-16573602 ] Todd Lipcon commented on IMPALA-7414: - Looks like the decimal-related test failures were already tracked by IMPALA-7411. Will fix both in one patch. > Recently added tests fail on localfs build > -- > > Key: IMPALA-7414 > URL: https://issues.apache.org/jira/browse/IMPALA-7414 > Project: IMPALA > Issue Type: Bug >Affects Versions: Impala 3.1.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > > A couple recently-added tests started failing the localfs build. Seems like > just missing cases of using the filesystem prefix. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7414) Recently added tests fail on localfs build
[
https://issues.apache.org/jira/browse/IMPALA-7414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573571#comment-16573571
]
Todd Lipcon commented on IMPALA-7414:
-
Following tests failed:
metadata.test_partition_metadata.TestMixedPartitions.test_incompatible_avro_partition_in_non_avro_table[exec_option:
{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000,
'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
'exec_single_node_rows_threshold': 0} | table_format: text/none-textfile]
query_test.test_scanners.TestParquet.test_decimal_encodings[exec_option:
{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
'disable_codegen': True, 'abort_on_error': 1, 'debug_action': None,
'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
query_test.test_scanners.TestParquet.test_decimal_encodings[exec_option:
{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
query_test.test_scanners.TestParquet.test_decimal_encodings[exec_option:
{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
'disable_codegen': True, 'abort_on_error': 1, 'debug_action':
'-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.5',
'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
query_test.test_scanners.TestParquet.test_decimal_encodings[exec_option:
{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
'disable_codegen': False, 'abort_on_error': 1, 'debug_action':
'-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.5',
'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
query_test.test_scanners.TestParquet.test_decimal_encodings[exec_option:
{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
'disable_codegen': True, 'abort_on_error': 1, 'debug_action':
'-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0',
'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
query_test.test_scanners.TestParquet.test_decimal_encodings[exec_option:
{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
'disable_codegen': False, 'abort_on_error': 1, 'debug_action':
'-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0',
'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
> Recently added tests fail on localfs build
> --
>
> Key: IMPALA-7414
> URL: https://issues.apache.org/jira/browse/IMPALA-7414
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.1.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Minor
>
> A couple recently-added tests started failing the localfs build. Seems like
> just missing cases of using the filesystem prefix.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7414) Recently added tests fail on localfs build
Todd Lipcon created IMPALA-7414: --- Summary: Recently added tests fail on localfs build Key: IMPALA-7414 URL: https://issues.apache.org/jira/browse/IMPALA-7414 Project: IMPALA Issue Type: Bug Affects Versions: Impala 3.1.0 Reporter: Todd Lipcon Assignee: Todd Lipcon A couple recently-added tests started failing the localfs build. Seems like just missing cases of using the filesystem prefix. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7414) Recently added tests fail on localfs build
Todd Lipcon created IMPALA-7414: --- Summary: Recently added tests fail on localfs build Key: IMPALA-7414 URL: https://issues.apache.org/jira/browse/IMPALA-7414 Project: IMPALA Issue Type: Bug Affects Versions: Impala 3.1.0 Reporter: Todd Lipcon Assignee: Todd Lipcon A couple recently-added tests started failing the localfs build. Seems like just missing cases of using the filesystem prefix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IMPALA-7361) test_heterogeneous_proc_mem_limit: Query aborted: Not enough memory available on host (s3)
[
https://issues.apache.org/jira/browse/IMPALA-7361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573557#comment-16573557
]
Bikramjeet Vig commented on IMPALA-7361:
I looked at the logs and it seems like the instance of the first query that ran
in this test (which reserves 2G of memory on each host) never completed on the
host on the port 22001. Since admission controller looks at the max of
(mem_admitted[host], mem_reserved[host]) == max (0, 2GB) it queued the query
based on that and it eventually timed out with the wrong error message.
Note: The mem reserved for a query that is currently executing is its limit_,
if set (which should be the common case with admission control). Otherwise, if
the query has no limit or the query is finished executing, the current
consumption is used.
> test_heterogeneous_proc_mem_limit: Query aborted: Not enough memory
> available on host (s3)
> ---
>
> Key: IMPALA-7361
> URL: https://issues.apache.org/jira/browse/IMPALA-7361
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: nithya
>Assignee: Bikramjeet Vig
>Priority: Major
> Labels: broken-build
>
> texttest_heterogeneous_proc_mem_limit custom cluster test fails with the
> following assertion error
> Stacktrace
> {noformat}
> custom_cluster/test_admission_controller.py:514: in
> test_heterogeneous_proc_mem_limit
> assert re.search("Queued reason: Not enough memory available on host
> \S+.Needed "
> E AssertionError: ImpalaBeeswaxException:
> E Query aborted:Admission for query exceeded timeout 200ms in pool
> default-pool. Queued reason: Not enough memory available on host
> impala-ec2-centos74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:22001.Needed
> 2.00 GB but only 1.00 GB out of 3.00 GB was available.
> E
> E
> E assert None
> E+ where None = ('Queued reason: Not
> enough memory available on host \\S+.Needed 2.00 GB but only 1.00 GB out of
> 2.00 GB was available.', 'ImpalaBeeswaxException:\n Query aborted:Admission
> for query exceeded timeout 200ms in pool default-pool. Queued
> reaso...os74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:22001.Needed 2.00 GB
> but only 1.00 GB out of 3.00 GB was available.\n\n')
> E+where = re.search
> E+and 'ImpalaBeeswaxException:\n Query aborted:Admission for query
> exceeded timeout 200ms in pool default-pool. Queued
> reaso...os74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:22001.Needed 2.00 GB
> but only 1.00 GB out of 3.00 GB was available.\n\n' =
> str(ImpalaBeeswaxException())
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7402) DCHECK failed min_bytes_to_write <= dirty_unpinned_pages_ in buffer-pool
[
https://issues.apache.org/jira/browse/IMPALA-7402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573480#comment-16573480
]
Tim Armstrong commented on IMPALA-7402:
---
Tests running at the time:
{noformat}
03:52:28 [gw5] PASSED
query_test/test_scratch_limit.py::TestScratchLimit::test_with_zero_scratch_limit_no_memory_limit[exec_option:
{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000,
'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
'exec_single_node_rows_threshold': 0} | table_format: text/none]
03:53:21
query_test/test_sort.py::TestQueryFullSort::test_multiple_buffer_pool_limits[exec_option:
{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
03:53:21 [gw3] FAILED
query_test/test_spilling.py::TestSpillingDebugActionDimensions::test_spilling[exec_option:
{'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0',
'default_spillable_buffer_size': '256k'} | table_format: parquet/none]
03:53:22
query_test/test_spilling.py::TestSpillingDebugActionDimensions::test_spilling_aggs[exec_option:
{'debug_action': None, 'default_spillable_buffer_size': '256k'} |
table_format: parquet/none]
03:53:22 [gw3] FAILED
query_test/test_spilling.py::TestSpillingDebugActionDimensions::test_spilling_aggs[exec_option:
{'debug_action': None, 'default_spillable_buffer_size': '256k'} |
table_format: parquet/none]
03:53:23
query_test/test_spilling.py::TestSpillingDebugActionDimensions::test_spilling_aggs[exec_option:
{'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0',
'default_spillable_buffer_size': '256k'} | table_format: parquet/none]
03:53:23 [gw3] FAILED
query_test/test_spilling.py::TestSpillingDebugActionDimensions::test_spilling_aggs[exec_option:
{'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0',
'default_spillable_buffer_size': '256k'} | table_format: parquet/none]
03:53:24
query_test/test_spilling.py::TestSpillingDebugActionDimensions::test_spilling_large_rows[exec_option:
{'debug_action': None, 'default_spillable_buffer_size': '256k'} |
table_format: parquet/none]
03:53:24 [gw3] FAILED
query_test/test_spilling.py::TestSpillingDebugActionDimensions::test_spilling_large_rows[exec_option:
{'debug_action': None, 'default_spillable_buffer_size': '256k'} |
table_format: parquet/none]
03:53:28
query_test/test_spilling.py::TestSpillingDebugActionDimensions::test_spilling_large_rows[exec_option:
{'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0',
'default_spillable_buffer_size': '256k'} | table_format: parquet/none]
03:53:28 [gw2] PASSED
query_test/test_scanners_fuzz.py::TestScannersFuzzing::test_fuzz_decimal_tbl[exec_option:
{'debug_action': None, 'abort_on_error': False, 'mem_limit': '512m',
'num_nodes': 0} | table_format: parquet/none]
03:53:29
query_test/test_scanners_fuzz.py::TestScannersFuzzing::test_fuzz_decimal_tbl[exec_option:
{'debug_action': None, 'abort_on_error': False, 'mem_limit': '512m',
'num_nodes': 0} | table_format: avro/snap/block]
03:53:29 [gw1] FAILED
query_test/test_tpcds_queries.py::TestTpcdsQuery::test_tpcds_q53[exec_option:
{'decimal_v2': 0, 'batch_size': 0, 'num_nodes': 0,
'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold':
0} | table_format: parquet/none]
03:53:33
query_test/test_tpcds_queries.py::TestTpcdsQuery::test_tpcds_q54[exec_option:
{'decimal_v2': 0, 'batch_size': 0, 'num_nodes': 0,
'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold':
0} | table_format: parquet/none]
03:53:33 [gw7] FAILED
query_test/test_tpcds_queries.py::TestTpcdsQuery::test_tpcds_q4[exec_option:
{'decimal_v2': 0, 'batch_size': 0, 'num_nodes': 0,
'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold':
0} | table_format: parquet/none]
03:53:35
query_test/test_tpcds_queries.py::TestTpcdsQuery::test_tpcds_q6[exec_option:
{'decimal_v2': 0, 'batch_size': 0, 'num_nodes': 0,
'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold':
0} | table_format: parquet/none]
03:53:35 [gw0] FAILED
query_test/test_tpcds_queries.py::TestTpcdsQuery::test_tpcds_q75[exec_option:
{'decimal_v2': 0, 'batch_size': 0, 'num_nodes': 0,
'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold':
0} | table_format: parquet/none]
03:54:08
query_test/test_tpcds_queries.py::TestTpcdsQuery::test_tpcds_q76[exec_option:
{'decimal_v2': 0, 'batch_size': 0,
[jira] [Commented] (IMPALA-7402) DCHECK failed min_bytes_to_write <= dirty_unpinned_pages_ in buffer-pool
[
https://issues.apache.org/jira/browse/IMPALA-7402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573456#comment-16573456
]
Tim Armstrong commented on IMPALA-7402:
---
Saw this again with some more debug info:
{noformat}
F0808 03:53:14.250478 28112 buffer-pool.cc:645] Check failed:
min_bytes_to_write <= dirty_unpinned_pages_.bytes() (8192 vs. 0)
0x615003d63580 name: HDFS_SCAN_NODE id=0
ptr=0x61d0009c7c80 write_status: buffers allocated 49152 num_pages: 0
pinned_bytes: 0 dirty_unpinned_bytes: 0 in_flight_write_bytes: 0 reservation:
{: reservation_limit 9223372036854775807 reservation 81920
used_reservation 49152 child_reservations 0 parent:
: reservation_limit 9223372036854775807 reservation
35733504 used_reservation 0 child_reservations 35733504 parent:
: reservation_limit 429496729 reservation 71385088
used_reservation 0 child_reservations 71385088 parent:
: reservation_limit 10952163328 reservation 106577920
used_reservation 0 child_reservations 106577920 parent:
NULL}
0 pinned pages:
0 dirty unpinned pages:
0 in flight write pages:
{noformat}
> DCHECK failed min_bytes_to_write <= dirty_unpinned_pages_ in buffer-pool
>
>
> Key: IMPALA-7402
> URL: https://issues.apache.org/jira/browse/IMPALA-7402
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Vuk Ercegovac
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: broken-build
>
> One of the impalad's crashed with the following DCHECK failure:
> {noformat}
> F0806 01:26:21.905500 5101 buffer-pool.cc:645] Check failed:
> min_bytes_to_write <= dirty_unpinned_pages_.bytes() (8192 vs. 0)
> Here is the backtrace:{noformat}
> {noformat}
> #0 0x003af1e328e5 in raise () from /lib64/libc.so.6
> #1 0x003af1e340c5 in abort () from /lib64/libc.so.6
> #2 0x0437f454 in google::DumpStackTraceAndExit() ()
> #3 0x04375ead in google::LogMessage::Fail() ()
> #4 0x04377752 in google::LogMessage::SendToLog() ()
> #5 0x04375887 in google::LogMessage::Flush() ()
> #6 0x04378e4e in google::LogMessageFatal::~LogMessageFatal() ()
> #7 0x0205ad16 in impala::BufferPool::Client::WriteDirtyPagesAsync
> (this=0x17d03f0e0, min_bytes_to_write=8192) at
> /data/jenkins/workspace/impala-cdh6.x-exhaustive-centos6/repos/Impala/be/src/runtime/bufferpool/buffer-pool.cc:645
>
> #8 0x0205a835 in impala::BufferPool::Client::CleanPages
> (this=0x17d03f0e0, client_lock=0x7f324cb12220, len=8192) at
> /data/jenkins/workspace/impala-cdh6.x-exhaustive-centos6/repos/Impala/be/src/runtime/bufferpool/buffer-pool.cc:625
>
> #9 0x0205a646 in impala::BufferPool::Client::DecreaseReservationTo
> (this=0x17d03f0e0, max_decrease=8192, target_bytes=8192) at
> /data/jenkins/workspace/impala-cdh6.x-exhaustive-centos6/repos/Impala/be/src/runtime/bufferpool/buffer-pool.cc:609
>
> #10 0x02057583 in
> impala::BufferPool::ClientHandle::DecreaseReservationTo (this=0x181c0a990,
> max_decrease=8192, target_bytes=8192) at
> /data/jenkins/workspace/impala-cdh6.x-exhaustive-centos6/repos/Impala/be/src/runtime/bufferpool/buffer-pool.cc:319
>
> #11 0x020d9419 in
> impala::HdfsScanNode::ReturnReservationFromScannerThread (this=0x181c0a800,
> lock=..., bytes=8192) at
> /data/jenkins/workspace/impala-cdh6.x-exhaustive-centos6/repos/Impala/be/src/exec/hdfs-scan-node.cc:194
>
> #12 0x020da485 in impala::HdfsScanNode::ScannerThread
> (this=0x181c0a800, first_thread=false, scanner_thread_reservation=8192) at
> /data/jenkins/workspace/impala-cdh6.x-exhaustive-centos6/repos/Impala/be/src/exec/hdfs-scan-node.cc:367
>
> #13 0x020d96b0 in impala::HdfsScanNodeoperator()(void)
> const (__closure=0x7f324cb12b88) at
> /data/jenkins/workspace/impala-cdh6.x-exhaustive-centos6/repos/Impala/be/src/exec/hdfs-scan-node.cc:261
>
> #14 0x020db6d6 in
> boost::detail::function::void_function_obj_invoker0,
> void>::invoke(boost::detail::function::function_buffer &)
> (function_obj_ptr=...) at
> /data/jenkins/workspace/impala-cdh6.x-exhaustive-centos6/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:153
> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-7202) Add WIDTH_BUCKET() function to the decimal fuzz test
[ https://issues.apache.org/jira/browse/IMPALA-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong reassigned IMPALA-7202: - Assignee: Zoltán Borók-Nagy (was: Anuj Phadke) > Add WIDTH_BUCKET() function to the decimal fuzz test > > > Key: IMPALA-7202 > URL: https://issues.apache.org/jira/browse/IMPALA-7202 > Project: IMPALA > Issue Type: Task > Components: Infrastructure >Reporter: Taras Bobrovytsky >Assignee: Zoltán Borók-Nagy >Priority: Critical > > We need to add the new WIDTH_BUCKET() function to the decimal fuzz test for > better coverage. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7413) test_incompatible_avro_partition_in_non_avro_table breaks in local mode
Vuk Ercegovac created IMPALA-7413:
-
Summary: test_incompatible_avro_partition_in_non_avro_table breaks
in local mode
Key: IMPALA-7413
URL: https://issues.apache.org/jira/browse/IMPALA-7413
Project: IMPALA
Issue Type: Bug
Components: Frontend
Affects Versions: Impala 3.1.0
Reporter: Vuk Ercegovac
New support for avro tables breaks when run in local mode:
{noformat}
metadata/test_partition_metadata.py:150: in
test_incompatible_avro_partition_in_non_avro_table
test_file_vars={'$MAIN_TABLE_FORMAT': main_table_format})
common/impala_test_suite.py:424: in run_test_case
assert False, "Expected exception: %s" % expected_str
E AssertionError: Expected exception: Unresolvable types for column
'tinyint_col': declared column type: TINYINT, table's Avro schema type:
int{noformat}
Here's the change: [https://gerrit.cloudera.org/#/c/10970/]
If the fix takes the same amount of time as a revert, lets fix it. Otherwise,
lets revert it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7413) test_incompatible_avro_partition_in_non_avro_table breaks in local mode
Vuk Ercegovac created IMPALA-7413:
-
Summary: test_incompatible_avro_partition_in_non_avro_table breaks
in local mode
Key: IMPALA-7413
URL: https://issues.apache.org/jira/browse/IMPALA-7413
Project: IMPALA
Issue Type: Bug
Components: Frontend
Affects Versions: Impala 3.1.0
Reporter: Vuk Ercegovac
New support for avro tables breaks when run in local mode:
{noformat}
metadata/test_partition_metadata.py:150: in
test_incompatible_avro_partition_in_non_avro_table
test_file_vars={'$MAIN_TABLE_FORMAT': main_table_format})
common/impala_test_suite.py:424: in run_test_case
assert False, "Expected exception: %s" % expected_str
E AssertionError: Expected exception: Unresolvable types for column
'tinyint_col': declared column type: TINYINT, table's Avro schema type:
int{noformat}
Here's the change: [https://gerrit.cloudera.org/#/c/10970/]
If the fix takes the same amount of time as a revert, lets fix it. Otherwise,
lets revert it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (IMPALA-7410) HDFS Datanodes unable to start
[
https://issues.apache.org/jira/browse/IMPALA-7410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573399#comment-16573399
]
Philip Zeyliger commented on IMPALA-7410:
-
The build of Hadoop we used here was built on centos7 (I checked this) and the
native libraries are refusing to work on a centos6 machine (I didn't check
this, but that's what this error usually means). We need to update the build
we're using.
> HDFS Datanodes unable to start
> --
>
> Key: IMPALA-7410
> URL: https://issues.apache.org/jira/browse/IMPALA-7410
> Project: IMPALA
> Issue Type: Improvement
> Components: Infrastructure
>Affects Versions: Impala 3.1.0
>Reporter: Vuk Ercegovac
>Priority: Blocker
> Labels: broken-build
>
> A recent test job err'd out when HDFS could not be setup. From the console:
> {noformat}
> ...
> 14:59:31 Stopping hdfs
> 14:59:33 Starting hdfs (Web UI - http://localhost:5070)
> 14:59:38 Failed to start hdfs-datanode. The end of the log
> (/data/jenkins/workspace/impala-asf-master-exhaustive-centos6/repos/Impala/testdata/cluster/cdh6/node-3/var/log/hdfs-datanode.out)
> is:
> 14:59:39 WARNING:
> /data/jenkins/workspace/impala-asf-master-exhaustive-centos6/repos/Impala/testdata/cluster/cdh6/node-3/var/log/hadoop-hdfs
> does not exist. Creating.
> 14:59:39 Failed to start hdfs-datanode. The end of the log
> (/data/jenkins/workspace/impala-asf-master-exhaustive-centos6/repos/Impala/testdata/cluster/cdh6/node-2/var/log/hdfs-datanode.out)
> is:
> 14:59:39 WARNING:
> /data/jenkins/workspace/impala-asf-master-exhaustive-centos6/repos/Impala/testdata/cluster/cdh6/node-2/var/log/hadoop-hdfs
> does not exist. Creating.
> 14:59:39 Failed to start hdfs-datanode. The end of the log
> (/data/jenkins/workspace/impala-asf-master-exhaustive-centos6/repos/Impala/testdata/cluster/cdh6/node-1/var/log/hdfs-datanode.out)
> is:
> 14:59:39 WARNING:
> /data/jenkins/workspace/impala-asf-master-exhaustive-centos6/repos/Impala/testdata/cluster/cdh6/node-1/var/log/hadoop-hdfs
> does not exist. Creating.
> 14:59:47 Namenode started
> 14:59:47 Error in
> /data/jenkins/workspace/impala-asf-master-exhaustive-centos6/repos/Impala/testdata/bin/run-mini-dfs.sh
> at line 41: $IMPALA_HOME/testdata/cluster/admin start_cluster
> 14:59:48 Error in
> /data/jenkins/workspace/impala-asf-master-exhaustive-centos6/repos/Impala/testdata/bin/run-all.sh
> at line 44: tee ${IMPALA_CLUSTER_LOGS_DIR}/run-mini-dfs.log
> ...{noformat}
> From one of the datanodes that could not start:
> {noformat}
> ...
> 2018-08-07 14:59:38,561 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
> java.lang.RuntimeException: Cannot start datanode because the configured max
> locked memory size (dfs.datanode.max.locked.memory) is greater than zero and
> native code is not available.
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1365)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:497)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2778)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2681)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2728)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2872)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2896)
> 2018-08-07 14:59:38,568 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1: java.lang.RuntimeException: Cannot start datanode because the
> configured max locked memory size (dfs.datanode.max.locked.memory) is greater
> than zero and native code is not available.
> 2018-08-07 14:59:38,575 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> SHUTDOWN_MSG:{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7412) width_bucket() function overflows too easily
[
https://issues.apache.org/jira/browse/IMPALA-7412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltán Borók-Nagy updated IMPALA-7412:
--
Description:
I looked at the failing query:
{noformat}
select width_bucket(cast(-0.5065802795359821573 as decimal(28,28)),
cast(-7.247919472444366553466758690723 as decimal(31,30)), cast(2.6 as
decimal(22,17)), 189748626);{noformat}
and the problem is that in WidthBucketImpl() we want to cast num_buckets to a
decimal with ARG_TYPE_PRECISION and ARG_TYPE_SCALE.
ARG_TYPE_PRECISION and ARG_TYPE_SCALE are determined by the Frontend in
{noformat}
FunctionCallExpr.analyzeImpl()->castForFunctionCall()->getResolvedWildCardType(){noformat}
getResolvedWildCardType() only looks at the arguments that correspond to
wildcard decimal parameters:
{noformat}
if (!fnArgs[ix].isWildcardDecimal()) continue;{noformat}
Therefore it doesn't take the INT argument (num_buckets) into account and
determines a decimal with precision 35 and scale 30.
We could modify getResolvedWildCardType() to consider other arguments as well.
This query would fail again because INT needs 10 digits precision with 0 digits
scale => the determined decimal would need precision 40 instead of 35. It is an
error because Impala decimals can only have precision 38 at most.
A better approach for this case would be to figure out the exact number of the
digits from the literal expression 189748626 => 9. However, that would also
fail because it would need precision 39.
If we want to cast num_buckets to a decimal type we cannot make this query
successful without information loss.
The other approach is to modify WidthBucketImpl() to interpret its parameters
as integers, because all of them have the same byte size, precision, and scale.
was:
I looked at the failing query:
select width_bucket(cast(-0.5065802795359821573 as decimal(28,28)),
cast(-7.247919472444366553466758690723 as decimal(31,30)), cast(2.6 as
decimal(22,17)), 189748626);
and the problem is that in WidthBucketImpl() we want to cast num_buckets to a
decimal with ARG_TYPE_PRECISION and ARG_TYPE_SCALE.
ARG_TYPE_PRECISION and ARG_TYPE_SCALE are determined by the Frontend in
FunctionCallExpr.analyzeImpl()->castForFunctionCall()->getResolvedWildCardType().
getResolvedWildCardType() only looks at the arguments that correspond to
wildcard decimal parameters:
if (!fnArgs[ix].isWildcardDecimal()) continue;
Therefore it doesn't take the INT argument (num_buckets) into account and
determines a decimal with precision 35 and scale 30.
We could modify getResolvedWildCardType() to consider other arguments as well.
This query would fail again because INT needs 10 digits precision with 0 digits
scale => the determined decimal would need precision 40 instead of 35. It is an
error because Impala decimals can only have precision 38 at most.
A better approach for this case would be to figure out the exact number of the
digits from the literal expression 189748626 => 9. However, that would also
fail because it would need precision 39.
If we want to cast num_buckets to a decimal type we cannot make this query
successful without information loss.
The other approach is to modify WidthBucketImpl() to interpret its parameters
as integers, because all of them have the same byte size, precision, and scale.
> width_bucket() function overflows too easily
>
>
> Key: IMPALA-7412
> URL: https://issues.apache.org/jira/browse/IMPALA-7412
> Project: IMPALA
> Issue Type: Bug
>Reporter: Zoltán Borók-Nagy
>Priority: Major
>
> I looked at the failing query:
> {noformat}
> select width_bucket(cast(-0.5065802795359821573 as decimal(28,28)),
> cast(-7.247919472444366553466758690723 as decimal(31,30)), cast(2.6 as
> decimal(22,17)), 189748626);{noformat}
> and the problem is that in WidthBucketImpl() we want to cast num_buckets to a
> decimal with ARG_TYPE_PRECISION and ARG_TYPE_SCALE.
> ARG_TYPE_PRECISION and ARG_TYPE_SCALE are determined by the Frontend in
> {noformat}
> FunctionCallExpr.analyzeImpl()->castForFunctionCall()->getResolvedWildCardType(){noformat}
> getResolvedWildCardType() only looks at the arguments that correspond to
> wildcard decimal parameters:
> {noformat}
> if (!fnArgs[ix].isWildcardDecimal()) continue;{noformat}
> Therefore it doesn't take the INT argument (num_buckets) into account and
> determines a decimal with precision 35 and scale 30.
> We could modify getResolvedWildCardType() to consider other arguments as
> well. This query would fail again because INT needs 10 digits precision with
> 0 digits scale => the determined decimal would need precision 40 instead of
> 35. It is an error because Impala decimals can only have precision 38 at most.
> A better approach for this case would be to figure out the exact number of
[jira] [Created] (IMPALA-7412) width_bucket() function overflows too easily
Zoltán Borók-Nagy created IMPALA-7412: - Summary: width_bucket() function overflows too easily Key: IMPALA-7412 URL: https://issues.apache.org/jira/browse/IMPALA-7412 Project: IMPALA Issue Type: Bug Reporter: Zoltán Borók-Nagy I looked at the failing query: select width_bucket(cast(-0.5065802795359821573 as decimal(28,28)), cast(-7.247919472444366553466758690723 as decimal(31,30)), cast(2.6 as decimal(22,17)), 189748626); and the problem is that in WidthBucketImpl() we want to cast num_buckets to a decimal with ARG_TYPE_PRECISION and ARG_TYPE_SCALE. ARG_TYPE_PRECISION and ARG_TYPE_SCALE are determined by the Frontend in FunctionCallExpr.analyzeImpl()->castForFunctionCall()->getResolvedWildCardType(). getResolvedWildCardType() only looks at the arguments that correspond to wildcard decimal parameters: if (!fnArgs[ix].isWildcardDecimal()) continue; Therefore it doesn't take the INT argument (num_buckets) into account and determines a decimal with precision 35 and scale 30. We could modify getResolvedWildCardType() to consider other arguments as well. This query would fail again because INT needs 10 digits precision with 0 digits scale => the determined decimal would need precision 40 instead of 35. It is an error because Impala decimals can only have precision 38 at most. A better approach for this case would be to figure out the exact number of the digits from the literal expression 189748626 => 9. However, that would also fail because it would need precision 39. If we want to cast num_buckets to a decimal type we cannot make this query successful without information loss. The other approach is to modify WidthBucketImpl() to interpret its parameters as integers, because all of them have the same byte size, precision, and scale. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-7412) width_bucket() function overflows too easily
Zoltán Borók-Nagy created IMPALA-7412: - Summary: width_bucket() function overflows too easily Key: IMPALA-7412 URL: https://issues.apache.org/jira/browse/IMPALA-7412 Project: IMPALA Issue Type: Bug Reporter: Zoltán Borók-Nagy I looked at the failing query: select width_bucket(cast(-0.5065802795359821573 as decimal(28,28)), cast(-7.247919472444366553466758690723 as decimal(31,30)), cast(2.6 as decimal(22,17)), 189748626); and the problem is that in WidthBucketImpl() we want to cast num_buckets to a decimal with ARG_TYPE_PRECISION and ARG_TYPE_SCALE. ARG_TYPE_PRECISION and ARG_TYPE_SCALE are determined by the Frontend in FunctionCallExpr.analyzeImpl()->castForFunctionCall()->getResolvedWildCardType(). getResolvedWildCardType() only looks at the arguments that correspond to wildcard decimal parameters: if (!fnArgs[ix].isWildcardDecimal()) continue; Therefore it doesn't take the INT argument (num_buckets) into account and determines a decimal with precision 35 and scale 30. We could modify getResolvedWildCardType() to consider other arguments as well. This query would fail again because INT needs 10 digits precision with 0 digits scale => the determined decimal would need precision 40 instead of 35. It is an error because Impala decimals can only have precision 38 at most. A better approach for this case would be to figure out the exact number of the digits from the literal expression 189748626 => 9. However, that would also fail because it would need precision 39. If we want to cast num_buckets to a decimal type we cannot make this query successful without information loss. The other approach is to modify WidthBucketImpl() to interpret its parameters as integers, because all of them have the same byte size, precision, and scale. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7376) Impala hits a DCHECK if a fragment instance fails to initialize the filter bank
[ https://issues.apache.org/jira/browse/IMPALA-7376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572801#comment-16572801 ] ASF subversion and git services commented on IMPALA-7376: - Commit cbc8c63ef0446550bd080e226b38307c4de967eb in impala's branch refs/heads/master from [~sailesh] [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=cbc8c63 ] IMPALA-7163: Implement a state machine for the QueryState class This patch adds a state machine for the QueryState class. The motivation behind this patch is to make the query lifecycle from the point of view of an executor much easier to reason about and this patch is key for a follow on patch for IMPALA-2990 where the status reporting will be per-query rather than per-fragment-instance. Currently, the state machine provides no other purpose, and it will mostly be used for IMPALA-2990. We introduce 5 possible states for the QueryState which include 3 terminal states (FINISHED, CANCELLED and ERROR) and 2 non-terminal states (PREPARING, EXECUTING). The transition from one state to the next is always handled by a single thread which is also the QueryState thread. This thread will additionally bear the purpose of sending periodic updates after IMPALA-4063, which is the primary reason behind having only this thread modify the state of the query. Counting barriers are introduced to keep a count of how many fragment instances have finished Preparing and Executing. These barriers also block until all the fragment instances have finished a respective state. The fragment instances update the query wide query status if an error is hit and unblocks the barrier if it is in the EXECUTING state. The PREPARING state blocks regardless of whether a fragment instance hit an error or not, until all the fragment instances have completed successfully or unsuccessfully, to maintain the invariant that fragment instances cannot be cancelled until the entire QueryState has finished PREPARING. The status reporting protocol has not been changed and remains exactly as it was. Testing: - Added 3 failure points in the query lifecycle using debug actions and added tests to validate the same (extension of IMPALA-7376). - Ran 'core' and 'exhaustive' tests. Future related work: 1) IMPALA-2990: Make status reporting per-query. 2) Try to logically align the FIS states with the QueryState states. 3) Consider mirroring the QueryState state machine to CoordinatorBackendState Change-Id: Iec5670a7db83ecae4656d7bb2ea372d3767ba7fe Reviewed-on: http://gerrit.cloudera.org:8080/10813 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Impala hits a DCHECK if a fragment instance fails to initialize the filter > bank > --- > > Key: IMPALA-7376 > URL: https://issues.apache.org/jira/browse/IMPALA-7376 > Project: IMPALA > Issue Type: Bug > Components: Backend >Reporter: Sailesh Mukil >Assignee: Sailesh Mukil >Priority: Critical > Labels: crash > > While Prepare()-ing a fragment instance, if we fail to initialize the runtime > filter bank, we will exit FIS::Prepare() without acquiring a thread token > (AcquireThreadToken()): > https://github.com/apache/impala/blob/316b17ac55adb3d1deeb1289b4045688269b201d/be/src/runtime/fragment-instance-state.cc#L135-L139 > FIS::Finalize() is called *always* regardless of whether the fragment > instance succeeded or failed. And FIS::Finalize() tries to > ReleaseThreadToken() even though it might not have gotten acquired: > https://github.com/apache/impala/blob/316b17ac55adb3d1deeb1289b4045688269b201d/be/src/runtime/fragment-instance-state.cc#L464 > , causing a DCHECK to be hit. > This was found while I was adding global debug actions (IMPALA-7046) to the > FIS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-2990) Coordinator should timeout a connection for an unresponsive backend
[ https://issues.apache.org/jira/browse/IMPALA-2990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572799#comment-16572799 ] ASF subversion and git services commented on IMPALA-2990: - Commit cbc8c63ef0446550bd080e226b38307c4de967eb in impala's branch refs/heads/master from [~sailesh] [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=cbc8c63 ] IMPALA-7163: Implement a state machine for the QueryState class This patch adds a state machine for the QueryState class. The motivation behind this patch is to make the query lifecycle from the point of view of an executor much easier to reason about and this patch is key for a follow on patch for IMPALA-2990 where the status reporting will be per-query rather than per-fragment-instance. Currently, the state machine provides no other purpose, and it will mostly be used for IMPALA-2990. We introduce 5 possible states for the QueryState which include 3 terminal states (FINISHED, CANCELLED and ERROR) and 2 non-terminal states (PREPARING, EXECUTING). The transition from one state to the next is always handled by a single thread which is also the QueryState thread. This thread will additionally bear the purpose of sending periodic updates after IMPALA-4063, which is the primary reason behind having only this thread modify the state of the query. Counting barriers are introduced to keep a count of how many fragment instances have finished Preparing and Executing. These barriers also block until all the fragment instances have finished a respective state. The fragment instances update the query wide query status if an error is hit and unblocks the barrier if it is in the EXECUTING state. The PREPARING state blocks regardless of whether a fragment instance hit an error or not, until all the fragment instances have completed successfully or unsuccessfully, to maintain the invariant that fragment instances cannot be cancelled until the entire QueryState has finished PREPARING. The status reporting protocol has not been changed and remains exactly as it was. Testing: - Added 3 failure points in the query lifecycle using debug actions and added tests to validate the same (extension of IMPALA-7376). - Ran 'core' and 'exhaustive' tests. Future related work: 1) IMPALA-2990: Make status reporting per-query. 2) Try to logically align the FIS states with the QueryState states. 3) Consider mirroring the QueryState state machine to CoordinatorBackendState Change-Id: Iec5670a7db83ecae4656d7bb2ea372d3767ba7fe Reviewed-on: http://gerrit.cloudera.org:8080/10813 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Coordinator should timeout a connection for an unresponsive backend > --- > > Key: IMPALA-2990 > URL: https://issues.apache.org/jira/browse/IMPALA-2990 > Project: IMPALA > Issue Type: Bug > Components: Distributed Exec >Affects Versions: Impala 2.3.0 >Reporter: Sailesh Mukil >Assignee: Sailesh Mukil >Priority: Critical > Labels: hang, observability, supportability > > The coordinator currently waits indefinitely if it does not hear back from a > backend. This could cause a query to hang indefinitely in case of a network > error, etc. > We should add logic for determining when a backend is unresponsive and kill > the query. The logic should mostly revolve around Coordinator::Wait() and > Coordinator::UpdateFragmentExecStatus() based on whether it receives periodic > updates from a backed (via FragmentExecState::ReportStatusCb()). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7390) rpc-mgr-kerberized-test fails inside of test-with-docker for hostname resolution reasons
[
https://issues.apache.org/jira/browse/IMPALA-7390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572803#comment-16572803
]
ASF subversion and git services commented on IMPALA-7390:
-
Commit 2d6a459c76edbe619543fd50d68be72b79a63bc5 in impala's branch
refs/heads/master from [~philip]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=2d6a459 ]
IMPALA-7390: Configure /etc/hosts to avoid rpc-mgr-kerberized-test issues.
In the test-with-docker context, rpc-mgr-kerberized-test was failing,
ultimately due to the fact that the hostname was resolving to 127.0.0.1
and then back to 'localhost'.
This commit applies a workaround of adding "127.0.0.1 $(hostnahostname)"
to /etc/hosts, which allows the test to pass, and is what's done in
bootstrap_system.sh. In the Docker context, /etc/hosts needs to be
updated on every container start, because Docker tries to provide an
/etc/hosts for you. Previously, we were doing a different customization
(adding "hostname" to the existing "127.0.0.1" line), which wasn't good
enough for rpc-mgr-kerberized-test.
The original workaround, which is in "boostrap_system.sh" is still
in place, but I've added more documentation about reproduction there.
I've also filed HDFS-13797 to track the HDFS issue.
Change-Id: I91003cbc86177feb7f99563f61297f7da7fabab4
Reviewed-on: http://gerrit.cloudera.org:8080/3
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> rpc-mgr-kerberized-test fails inside of test-with-docker for hostname
> resolution reasons
>
>
> Key: IMPALA-7390
> URL: https://issues.apache.org/jira/browse/IMPALA-7390
> Project: IMPALA
> Issue Type: Task
>Reporter: Philip Zeyliger
>Priority: Major
>
> When running inside of test-with-docker, the test fails like so:
> {code}
> 2018-08-02 15:11:38.846042 [==] Running 1 test from 1 test case.
> 2018-08-02 15:11:38.846073 [--] Global test environment set-up.
> 2018-08-02 15:11:38.846114 [--] 1 test from
> KerberosOnAndOff/RpcMgrKerberizedTest
> 2018-08-02 15:11:38.846159 [ RUN ]
> KerberosOnAndOff/RpcMgrKerberizedTest.MultipleServicesTls/0
> 2018-08-02 15:11:38.846284 Aug 02 22:11:38 i-20180802-132959
> krb5kdc[13121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime
> 1533247898, etypes {rep=17 tkt=17 ses=17},
> impala-test/i-20180802-132...@krbtest.com for krbtgt/krbtest@krbtest.com
> 2018-08-02 15:11:38.846417 Aug 02 22:11:38 i-20180802-132959
> krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1:
> LOOKING_UP_SERVER: authtime 0, impala-test/i-20180802-132...@krbtest.com for
> impala-test/localh...@krbtest.com, Server not found in Kerberos database
> 2018-08-02 15:11:38.846555 Aug 02 22:11:38 i-20180802-132959
> krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1:
> LOOKING_UP_SERVER: authtime 0, impala-test/i-20180802-132...@krbtest.com for
> impala-test/localh...@krbtest.com, Server not found in Kerberos database
> 2018-08-02 15:11:38.846599
> /home/impdev/Impala/be/src/rpc/rpc-mgr-kerberized-test.cc:72: Failure
> 2018-08-02 15:11:38.846619 Value of: status_.ok()
> 2018-08-02 15:11:38.846635 Actual: false
> 2018-08-02 15:11:38.846651 Expected: true
> 2018-08-02 15:11:38.846763 Error: unable to execute ScanMem() RPC.: Not
> authorized: Client connection negotiation failed: client connection to
> 127.0.0.1:53900: Server impala-test/localh...@krbtest.com not found in
> Kerberos database
> {code}
> The issue is that in this context the "hostname" is roughly mapping to
> 127.0.0.1 but then maps back to localhost.
> The workaround is pretty straight-forward; patch forthcoming.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6335) Remove the unnecessary decorator "pytest.mark.execute_serially" from tests which can be run in parallel
[ https://issues.apache.org/jira/browse/IMPALA-6335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572804#comment-16572804 ] ASF subversion and git services commented on IMPALA-6335: - Commit bf24a814ccf568fd0f165f05c90488ebc4c2db47 in impala's branch refs/heads/master from [~tlipcon] [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=bf24a81 ] IMPALA-6335. Allow most shell tests to run in parallel This adds an IMPALA_HISTFILE environment variable (and --history_file argument) to the shell which overrides the default location of ~/.impalahistory for the shell history. The shell tests now override this variable to /dev/null so they don't store history. The tests that need history use a pytest fixture to use a temporary file for their history. This allows so that they can run in parallel without stomping on each other's history. This also fixes a couple flaky test which were previously missing the "execute_serially" annotation -- that annotation is no longer needed after this fix. A couple of the tests still need to be executed serially because they look at metrics such as the number of executed or running queries, and those metrics are unstable if other tests run in parallel. I tested this by running: ./bin/impala-py.test tests/shell/test_shell_interactive.py \ -m 'not execute_serially' \ -n 80 \ --random ... several times in a row on an 88-core box. Prior to the change, several would fail each time. Now they pass. Change-Id: I1da5739276e63a50590dfcb2b050703f8e35fec7 Reviewed-on: http://gerrit.cloudera.org:8080/11045 Tested-by: Impala Public Jenkins Reviewed-by: Todd Lipcon > Remove the unnecessary decorator "pytest.mark.execute_serially" from tests > which can be run in parallel > --- > > Key: IMPALA-6335 > URL: https://issues.apache.org/jira/browse/IMPALA-6335 > Project: IMPALA > Issue Type: Improvement > Components: Infrastructure >Affects Versions: Impala 2.10.0 >Reporter: Jinchul Kim >Assignee: Todd Lipcon >Priority: Minor > Labels: newbie > Fix For: Impala 3.1.0 > > > Regarding the decorator "pytest.mark.execute_serially", I saw all the test > cases are marked as execute_serially in > tests/shell/test_shell_interactive.py. I guess a few tests should be run > exclusively, but the other tests do not require the serial option. What do > you think about this? I think we should consider to use the decorator when > adding test cases. Minimized serial run can help to reduce overall test > running time by parallelism. > I think the following cases do not require the decorator. > a. Check consistency from test result > b. Test query cancellation > c. Test shell options which are effective on local shell > > From Tim's comment: > I think you're right that many of the shell tests don't inherently require to > be executed serially. Some of them would require work to execute in parallel, > particularly the ones that inspect files like .impalahistory and tests that > check the values of global impala daemon metrics. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7163) Implement a state machine for the QueryState class
[ https://issues.apache.org/jira/browse/IMPALA-7163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572797#comment-16572797 ] ASF subversion and git services commented on IMPALA-7163: - Commit cbc8c63ef0446550bd080e226b38307c4de967eb in impala's branch refs/heads/master from [~sailesh] [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=cbc8c63 ] IMPALA-7163: Implement a state machine for the QueryState class This patch adds a state machine for the QueryState class. The motivation behind this patch is to make the query lifecycle from the point of view of an executor much easier to reason about and this patch is key for a follow on patch for IMPALA-2990 where the status reporting will be per-query rather than per-fragment-instance. Currently, the state machine provides no other purpose, and it will mostly be used for IMPALA-2990. We introduce 5 possible states for the QueryState which include 3 terminal states (FINISHED, CANCELLED and ERROR) and 2 non-terminal states (PREPARING, EXECUTING). The transition from one state to the next is always handled by a single thread which is also the QueryState thread. This thread will additionally bear the purpose of sending periodic updates after IMPALA-4063, which is the primary reason behind having only this thread modify the state of the query. Counting barriers are introduced to keep a count of how many fragment instances have finished Preparing and Executing. These barriers also block until all the fragment instances have finished a respective state. The fragment instances update the query wide query status if an error is hit and unblocks the barrier if it is in the EXECUTING state. The PREPARING state blocks regardless of whether a fragment instance hit an error or not, until all the fragment instances have completed successfully or unsuccessfully, to maintain the invariant that fragment instances cannot be cancelled until the entire QueryState has finished PREPARING. The status reporting protocol has not been changed and remains exactly as it was. Testing: - Added 3 failure points in the query lifecycle using debug actions and added tests to validate the same (extension of IMPALA-7376). - Ran 'core' and 'exhaustive' tests. Future related work: 1) IMPALA-2990: Make status reporting per-query. 2) Try to logically align the FIS states with the QueryState states. 3) Consider mirroring the QueryState state machine to CoordinatorBackendState Change-Id: Iec5670a7db83ecae4656d7bb2ea372d3767ba7fe Reviewed-on: http://gerrit.cloudera.org:8080/10813 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Implement a state machine for the QueryState class > -- > > Key: IMPALA-7163 > URL: https://issues.apache.org/jira/browse/IMPALA-7163 > Project: IMPALA > Issue Type: Sub-task > Components: Distributed Exec >Reporter: Sailesh Mukil >Assignee: Sailesh Mukil >Priority: Major > > We've recently been improving our query lifecycle by adding explicit state > transitions so it's easier to reason about what should happen at a given > stage in the lifetime of a query or a fragment instance. > On the coordinator side. The coordinator's view of the query: > https://github.com/apache/impala/commit/6ca87e46736a1e591ed7d7d5fee05b4b4d2fbb50 > On the fragment instance state side. A FIS's view of its own execution: > https://github.com/apache/impala/blob/e12ee485cf4c77203b144c053ee167509cc39374/be/src/runtime/fragment-instance-state.h#L182-L203 > We don't have something like this for the QueryState class which maintains > query wide state per executor. Adding it should make the lifecycle of a query > from an executors point of view much easier to reason about. > Additional info: This was identified as part of work for > IMPALA-2990/IMPALA-4063, and is a precursor to it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-4063) Make fragment instance reports per-query (or per-host) instead of per-fragment instance.
[ https://issues.apache.org/jira/browse/IMPALA-4063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572800#comment-16572800 ] ASF subversion and git services commented on IMPALA-4063: - Commit cbc8c63ef0446550bd080e226b38307c4de967eb in impala's branch refs/heads/master from [~sailesh] [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=cbc8c63 ] IMPALA-7163: Implement a state machine for the QueryState class This patch adds a state machine for the QueryState class. The motivation behind this patch is to make the query lifecycle from the point of view of an executor much easier to reason about and this patch is key for a follow on patch for IMPALA-2990 where the status reporting will be per-query rather than per-fragment-instance. Currently, the state machine provides no other purpose, and it will mostly be used for IMPALA-2990. We introduce 5 possible states for the QueryState which include 3 terminal states (FINISHED, CANCELLED and ERROR) and 2 non-terminal states (PREPARING, EXECUTING). The transition from one state to the next is always handled by a single thread which is also the QueryState thread. This thread will additionally bear the purpose of sending periodic updates after IMPALA-4063, which is the primary reason behind having only this thread modify the state of the query. Counting barriers are introduced to keep a count of how many fragment instances have finished Preparing and Executing. These barriers also block until all the fragment instances have finished a respective state. The fragment instances update the query wide query status if an error is hit and unblocks the barrier if it is in the EXECUTING state. The PREPARING state blocks regardless of whether a fragment instance hit an error or not, until all the fragment instances have completed successfully or unsuccessfully, to maintain the invariant that fragment instances cannot be cancelled until the entire QueryState has finished PREPARING. The status reporting protocol has not been changed and remains exactly as it was. Testing: - Added 3 failure points in the query lifecycle using debug actions and added tests to validate the same (extension of IMPALA-7376). - Ran 'core' and 'exhaustive' tests. Future related work: 1) IMPALA-2990: Make status reporting per-query. 2) Try to logically align the FIS states with the QueryState states. 3) Consider mirroring the QueryState state machine to CoordinatorBackendState Change-Id: Iec5670a7db83ecae4656d7bb2ea372d3767ba7fe Reviewed-on: http://gerrit.cloudera.org:8080/10813 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Make fragment instance reports per-query (or per-host) instead of > per-fragment instance. > > > Key: IMPALA-4063 > URL: https://issues.apache.org/jira/browse/IMPALA-4063 > Project: IMPALA > Issue Type: Sub-task > Components: Distributed Exec >Affects Versions: Impala 2.7.0 >Reporter: Sailesh Mukil >Assignee: Sailesh Mukil >Priority: Major > Labels: performance > > Currently we send a report per-fragment instance to the coordinator every 5 > seconds (by default; modifiable via query option 'status_report_interval'). > For queries with a large number of fragment instances, this generates > tremendous amounts of network traffic to the coordinator, which will only be > aggravated with higher a DOP. > We should instead queue per-fragment instance reports and send out a > per-query report to the coordinator instead. > For code references, see: > PlanFragmentExecutor:: ReportProfile() > PlanFragmentExecutor:: SendReport() > FragmentExecState:: ReportStatusCb() -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-2990) Coordinator should timeout a connection for an unresponsive backend
[ https://issues.apache.org/jira/browse/IMPALA-2990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572802#comment-16572802 ] ASF subversion and git services commented on IMPALA-2990: - Commit cbc8c63ef0446550bd080e226b38307c4de967eb in impala's branch refs/heads/master from [~sailesh] [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=cbc8c63 ] IMPALA-7163: Implement a state machine for the QueryState class This patch adds a state machine for the QueryState class. The motivation behind this patch is to make the query lifecycle from the point of view of an executor much easier to reason about and this patch is key for a follow on patch for IMPALA-2990 where the status reporting will be per-query rather than per-fragment-instance. Currently, the state machine provides no other purpose, and it will mostly be used for IMPALA-2990. We introduce 5 possible states for the QueryState which include 3 terminal states (FINISHED, CANCELLED and ERROR) and 2 non-terminal states (PREPARING, EXECUTING). The transition from one state to the next is always handled by a single thread which is also the QueryState thread. This thread will additionally bear the purpose of sending periodic updates after IMPALA-4063, which is the primary reason behind having only this thread modify the state of the query. Counting barriers are introduced to keep a count of how many fragment instances have finished Preparing and Executing. These barriers also block until all the fragment instances have finished a respective state. The fragment instances update the query wide query status if an error is hit and unblocks the barrier if it is in the EXECUTING state. The PREPARING state blocks regardless of whether a fragment instance hit an error or not, until all the fragment instances have completed successfully or unsuccessfully, to maintain the invariant that fragment instances cannot be cancelled until the entire QueryState has finished PREPARING. The status reporting protocol has not been changed and remains exactly as it was. Testing: - Added 3 failure points in the query lifecycle using debug actions and added tests to validate the same (extension of IMPALA-7376). - Ran 'core' and 'exhaustive' tests. Future related work: 1) IMPALA-2990: Make status reporting per-query. 2) Try to logically align the FIS states with the QueryState states. 3) Consider mirroring the QueryState state machine to CoordinatorBackendState Change-Id: Iec5670a7db83ecae4656d7bb2ea372d3767ba7fe Reviewed-on: http://gerrit.cloudera.org:8080/10813 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Coordinator should timeout a connection for an unresponsive backend > --- > > Key: IMPALA-2990 > URL: https://issues.apache.org/jira/browse/IMPALA-2990 > Project: IMPALA > Issue Type: Bug > Components: Distributed Exec >Affects Versions: Impala 2.3.0 >Reporter: Sailesh Mukil >Assignee: Sailesh Mukil >Priority: Critical > Labels: hang, observability, supportability > > The coordinator currently waits indefinitely if it does not hear back from a > backend. This could cause a query to hang indefinitely in case of a network > error, etc. > We should add logic for determining when a backend is unresponsive and kill > the query. The logic should mostly revolve around Coordinator::Wait() and > Coordinator::UpdateFragmentExecStatus() based on whether it receives periodic > updates from a backed (via FragmentExecState::ReportStatusCb()). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7411) copyFromLocal fails in local fs for query_test.test_scanners.TestParquet.test_decimal_encodings
Vuk Ercegovac created IMPALA-7411:
-
Summary: copyFromLocal fails in local fs for
query_test.test_scanners.TestParquet.test_decimal_encodings
Key: IMPALA-7411
URL: https://issues.apache.org/jira/browse/IMPALA-7411
Project: IMPALA
Issue Type: Improvement
Components: Infrastructure
Affects Versions: Impala 3.1.0
Reporter: Vuk Ercegovac
A recent core local test failed in
query_test.test_scanners.TestParquet.test_decimal_encodings when copy a file:
{noformat}
copyFromLocal:
`/test-warehouse/test_decimal_encodings_7677d707.db/decimal_stored_as_int32.parquet':
No such file or directory:
`file:///test-warehouse/test_decimal_encodings_7677d707.db/decimal_stored_as_int32.parquet'
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IMPALA-7411) copyFromLocal fails in local fs for query_test.test_scanners.TestParquet.test_decimal_encodings
Vuk Ercegovac created IMPALA-7411:
-
Summary: copyFromLocal fails in local fs for
query_test.test_scanners.TestParquet.test_decimal_encodings
Key: IMPALA-7411
URL: https://issues.apache.org/jira/browse/IMPALA-7411
Project: IMPALA
Issue Type: Improvement
Components: Infrastructure
Affects Versions: Impala 3.1.0
Reporter: Vuk Ercegovac
A recent core local test failed in
query_test.test_scanners.TestParquet.test_decimal_encodings when copy a file:
{noformat}
copyFromLocal:
`/test-warehouse/test_decimal_encodings_7677d707.db/decimal_stored_as_int32.parquet':
No such file or directory:
`file:///test-warehouse/test_decimal_encodings_7677d707.db/decimal_stored_as_int32.parquet'
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7410) HDFS Datanodes unable to start
Vuk Ercegovac created IMPALA-7410:
-
Summary: HDFS Datanodes unable to start
Key: IMPALA-7410
URL: https://issues.apache.org/jira/browse/IMPALA-7410
Project: IMPALA
Issue Type: Improvement
Components: Infrastructure
Affects Versions: Impala 3.1.0
Reporter: Vuk Ercegovac
A recent test job err'd out when HDFS could not be setup. From the console:
{noformat}
...
14:59:31 Stopping hdfs
14:59:33 Starting hdfs (Web UI - http://localhost:5070)
14:59:38 Failed to start hdfs-datanode. The end of the log
(/data/jenkins/workspace/impala-asf-master-exhaustive-centos6/repos/Impala/testdata/cluster/cdh6/node-3/var/log/hdfs-datanode.out)
is:
14:59:39 WARNING:
/data/jenkins/workspace/impala-asf-master-exhaustive-centos6/repos/Impala/testdata/cluster/cdh6/node-3/var/log/hadoop-hdfs
does not exist. Creating.
14:59:39 Failed to start hdfs-datanode. The end of the log
(/data/jenkins/workspace/impala-asf-master-exhaustive-centos6/repos/Impala/testdata/cluster/cdh6/node-2/var/log/hdfs-datanode.out)
is:
14:59:39 WARNING:
/data/jenkins/workspace/impala-asf-master-exhaustive-centos6/repos/Impala/testdata/cluster/cdh6/node-2/var/log/hadoop-hdfs
does not exist. Creating.
14:59:39 Failed to start hdfs-datanode. The end of the log
(/data/jenkins/workspace/impala-asf-master-exhaustive-centos6/repos/Impala/testdata/cluster/cdh6/node-1/var/log/hdfs-datanode.out)
is:
14:59:39 WARNING:
/data/jenkins/workspace/impala-asf-master-exhaustive-centos6/repos/Impala/testdata/cluster/cdh6/node-1/var/log/hadoop-hdfs
does not exist. Creating.
14:59:47 Namenode started
14:59:47 Error in
/data/jenkins/workspace/impala-asf-master-exhaustive-centos6/repos/Impala/testdata/bin/run-mini-dfs.sh
at line 41: $IMPALA_HOME/testdata/cluster/admin start_cluster
14:59:48 Error in
/data/jenkins/workspace/impala-asf-master-exhaustive-centos6/repos/Impala/testdata/bin/run-all.sh
at line 44: tee ${IMPALA_CLUSTER_LOGS_DIR}/run-mini-dfs.log
...{noformat}
>From one of the datanodes that could not start:
{noformat}
...
2018-08-07 14:59:38,561 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
Exception in secureMain
java.lang.RuntimeException: Cannot start datanode because the configured max
locked memory size (dfs.datanode.max.locked.memory) is greater than zero and
native code is not available.
at
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1365)
at org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:497)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2778)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2681)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2728)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2872)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2896)
2018-08-07 14:59:38,568 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 1: java.lang.RuntimeException: Cannot start datanode because the
configured max locked memory size (dfs.datanode.max.locked.memory) is greater
than zero and native code is not available.
2018-08-07 14:59:38,575 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
SHUTDOWN_MSG:{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
