[jira] [Resolved] (IMPALA-9736) "MT_DOP not supported for plans with base table joins or table sinks" error is out of date
[ https://issues.apache.org/jira/browse/IMPALA-9736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong resolved IMPALA-9736. --- Fix Version/s: Impala 4.0 Resolution: Fixed > "MT_DOP not supported for plans with base table joins or table sinks" error > is out of date > -- > > Key: IMPALA-9736 > URL: https://issues.apache.org/jira/browse/IMPALA-9736 > Project: IMPALA > Issue Type: Bug > Components: Frontend >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > Fix For: Impala 4.0 > > > We do support base table joins now. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8755) Implement Z-ordering for Impala
[ https://issues.apache.org/jira/browse/IMPALA-8755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17104050#comment-17104050 ] Jim Apple commented on IMPALA-8755: --- Hi all! What's work is left to complete this feature? > Implement Z-ordering for Impala > --- > > Key: IMPALA-8755 > URL: https://issues.apache.org/jira/browse/IMPALA-8755 > Project: IMPALA > Issue Type: New Feature >Reporter: Zoltán Borók-Nagy >Assignee: Norbert Luksa >Priority: Major > > Implement Z-ordering for Impala: [https://en.wikipedia.org/wiki/Z-order_curve] > A Z-order curve defines an ordering on multi-dimensional data. Data sorted > that way can be efficiently filtered by min/max statistics regarding to the > columns participating in the ordering. > Impala currently only supports lexicographic ordering via the SORT BY clause. > This strongly prefers the first column, i.e. given the "SORT BY A, B, C" > clause => A will be totally ordered (hence filtering on A will be very > efficient), but values belonging to B and C will be scattered throughout the > data set (hence filtering on B or C will barely do any good). > We could add a new clause, e.g. a "ZSORT BY" clause to Impala that writes the > data in Z-order. > "ZSORT BY A, B C" would cluster the rows in a way that filtering on A, B, or > C would be equally efficient. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-9741) Support query iceberg table by impala
[ https://issues.apache.org/jira/browse/IMPALA-9741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-9741 started by WangSheng. - > Support query iceberg table by impala > - > > Key: IMPALA-9741 > URL: https://issues.apache.org/jira/browse/IMPALA-9741 > Project: IMPALA > Issue Type: Sub-task >Reporter: WangSheng >Assignee: WangSheng >Priority: Major > > Since we have submit an patch of supporting create iceberg table by impala in > IMPALA-9688, we are preparing to implement iceberg table query by impala. But > we need to read the impala and iceberg code deeply to determine how to do > this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-9741) Support query iceberg table by impala
[ https://issues.apache.org/jira/browse/IMPALA-9741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangSheng updated IMPALA-9741: -- Description: Since we have submit an patch of supporting create iceberg table by impala in IMPALA-9688, we are preparing to implement iceberg table query by impala. But we need to read the impala and iceberg code deeply to determine how to do this. > Support query iceberg table by impala > - > > Key: IMPALA-9741 > URL: https://issues.apache.org/jira/browse/IMPALA-9741 > Project: IMPALA > Issue Type: Sub-task >Reporter: WangSheng >Assignee: WangSheng >Priority: Major > > Since we have submit an patch of supporting create iceberg table by impala in > IMPALA-9688, we are preparing to implement iceberg table query by impala. But > we need to read the impala and iceberg code deeply to determine how to do > this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-9741) Support query iceberg table by impala
WangSheng created IMPALA-9741: - Summary: Support query iceberg table by impala Key: IMPALA-9741 URL: https://issues.apache.org/jira/browse/IMPALA-9741 Project: IMPALA Issue Type: Sub-task Reporter: WangSheng Assignee: WangSheng -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-9621) Support iceberg on hdfs
[ https://issues.apache.org/jira/browse/IMPALA-9621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-9621 started by WangSheng. - > Support iceberg on hdfs > --- > > Key: IMPALA-9621 > URL: https://issues.apache.org/jira/browse/IMPALA-9621 > Project: IMPALA > Issue Type: Improvement >Reporter: WangSheng >Assignee: WangSheng >Priority: Major > > We are investigating iceberg recently, and preparing to implement select > iceberg data by impala. Our production use hdfs, so we will try to support > iceberg on hdfs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9382) Revisit representation of runtime profile objects to be denser
[ https://issues.apache.org/jira/browse/IMPALA-9382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103981#comment-17103981 ] Tim Armstrong commented on IMPALA-9382: --- Here is the latest iteration, showing profiles with the new format enabled and disabled. [^tpcds_q10_profile_v1.txt] [^tpcds_q10_profile_v2.txt] > Revisit representation of runtime profile objects to be denser > -- > > Key: IMPALA-9382 > URL: https://issues.apache.org/jira/browse/IMPALA-9382 > Project: IMPALA > Issue Type: Sub-task > Components: Backend >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > Attachments: profile_504b379400cba9f2_2d2cf007, > tpcds_q10_profile_v1.txt, tpcds_q10_profile_v2.txt > > > RuntimeProfile trees can potentially stress the memory allocator and use up a > lot more memory and cache than is really necessary: > * std::map is used throughout, and allocates a node per map entry. We do > depend on the counters being displayed in-order, but we would probably be > better of storing the counters in a vector and lazily sorting when needed > (since the set of counters is generally static after Prepare()). > * We store the same counter names redundantly all over the place. We'd > probably be best off using a pool of constant counter names (we could just > require registering them upfront). > There may be a small gain from switching thrift to using unordered_map, e.g. > for the info strings that appear with some frequency in profiles. > However, I think we need to restructure the thrift representation and > in-memory representation to get significant gains. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-9382) Revisit representation of runtime profile objects to be denser
[ https://issues.apache.org/jira/browse/IMPALA-9382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong updated IMPALA-9382: -- Attachment: tpcds_q10_profile_v2.txt tpcds_q10_profile_v1.txt > Revisit representation of runtime profile objects to be denser > -- > > Key: IMPALA-9382 > URL: https://issues.apache.org/jira/browse/IMPALA-9382 > Project: IMPALA > Issue Type: Sub-task > Components: Backend >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > Attachments: profile_504b379400cba9f2_2d2cf007, > tpcds_q10_profile_v1.txt, tpcds_q10_profile_v2.txt > > > RuntimeProfile trees can potentially stress the memory allocator and use up a > lot more memory and cache than is really necessary: > * std::map is used throughout, and allocates a node per map entry. We do > depend on the counters being displayed in-order, but we would probably be > better of storing the counters in a vector and lazily sorting when needed > (since the set of counters is generally static after Prepare()). > * We store the same counter names redundantly all over the place. We'd > probably be best off using a pool of constant counter names (we could just > require registering them upfront). > There may be a small gain from switching thrift to using unordered_map, e.g. > for the info strings that appear with some frequency in profiles. > However, I think we need to restructure the thrift representation and > in-memory representation to get significant gains. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-9740) TSAN data race in hdfs-bulk-ops
Sahil Takiar created IMPALA-9740:
Summary: TSAN data race in hdfs-bulk-ops
Key: IMPALA-9740
URL: https://issues.apache.org/jira/browse/IMPALA-9740
Project: IMPALA
Issue Type: Sub-task
Components: Backend
Reporter: Sahil Takiar
hdfs-bulk-ops usage of a local connection cache (HdfsFsCache::HdfsFsMap) has a
data race:
{code:java}
WARNING: ThreadSanitizer: data race (pid=23205)
Write of size 8 at 0x7b24005642d8 by thread T47:
#0
boost::unordered::detail::table_impl >, std::string, hdfs_internal*,
boost::hash, std::equal_to >
>::add_node(boost::unordered::detail::node_constructor > > >&, unsigned long)
/data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/unordered/detail/unique.hpp:329:26
(impalad+0x1f93832)
#1
std::pair > >, bool>
boost::unordered::detail::table_impl >, std::string, hdfs_internal*,
boost::hash, std::equal_to >
>::emplace_impl >(std::string
const&, std::pair&&)
/data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/unordered/detail/unique.hpp:420:41
(impalad+0x1f933ed)
#2
std::pair > >, bool>
boost::unordered::detail::table_impl >, std::string, hdfs_internal*,
boost::hash, std::equal_to >
>::emplace >(std::pair&&)
/data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/unordered/detail/unique.hpp:384:20
(impalad+0x1f932d1)
#3
std::pair > >, bool> boost::unordered::unordered_map, std::equal_to,
std::allocator >
>::emplace >(std::pair&&)
/data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/unordered/unordered_map.hpp:241:27
(impalad+0x1f93238)
#4 boost::unordered::unordered_map, std::equal_to,
std::allocator >
>::insert(std::pair&&)
/data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/unordered/unordered_map.hpp:390:26
(impalad+0x1f92038)
#5 impala::HdfsFsCache::GetConnection(std::string const&, hdfs_internal**,
boost::unordered::unordered_map, std::equal_to,
std::allocator > >*)
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/hdfs-fs-cache.cc:115:18
(impalad+0x1f916b3)
#6 impala::HdfsOp::Execute() const
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/hdfs-bulk-ops.cc:84:55
(impalad+0x23444d5)
#7 HdfsThreadPoolHelper(int, impala::HdfsOp const&)
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/hdfs-bulk-ops.cc:137:6
(impalad+0x2344ea9)
#8 boost::detail::function::void_function_invoker2::invoke(boost::detail::function::function_buffer&, int, impala::HdfsOp
const&)
/data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:118:11
(impalad+0x2345e80)
#9 boost::function2::operator()(int,
impala::HdfsOp const&) const
/data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:770:14
(impalad+0x1f883be)
#10 impala::ThreadPool::WorkerThread(int)
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/thread-pool.h:166:9
(impalad+0x1f874e5)
#11 boost::_mfi::mf1,
int>::operator()(impala::ThreadPool*, int) const
/data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/mem_fn_template.hpp:165:29
(impalad+0x1f87b7d)
#12 void
boost::_bi::list2*>,
boost::_bi::value >::operator(), int>,
boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf1, int>&, boost::_bi::list0&, int)
/data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:319:9
(impalad+0x1f87abc)
#13 boost::_bi::bind_t, int>,
boost::_bi::list2*>,
boost::_bi::value > >::operator()()
/data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16
(impalad+0x1f87a23)
#14
boost::detail::function::void_function_obj_invoker0, int>,
boost::_bi::list2*>,
boost::_bi::value > >,
void>::invoke(boost::detail::function::function_buffer&)
/data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:159:11
(impalad+0x1f877c1)
#15 boost::function0::operator()() const
/data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:770:14
(impalad+0x1e192b1)
#16 impala::Thread::SuperviseThread(std::string const&, std::string const&,
boost::function, impala::ThreadDebugInfo const*, impala::Promise*)
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/thread.cc:360:3
(impalad+0x23df196)
#17 void
[jira] [Assigned] (IMPALA-9739) TSAN data races during impalad shutdown
[
https://issues.apache.org/jira/browse/IMPALA-9739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar reassigned IMPALA-9739:
Assignee: Bikramjeet Vig
Not really sure what is going on here. Bikram, assigning to you since you
touched this code at some point. At the very least, it would be good to
understanding why the data race is occurring.
> TSAN data races during impalad shutdown
> ---
>
> Key: IMPALA-9739
> URL: https://issues.apache.org/jira/browse/IMPALA-9739
> Project: IMPALA
> Issue Type: Sub-task
> Components: Backend
>Reporter: Sahil Takiar
>Assignee: Bikramjeet Vig
>Priority: Major
>
> A TSAN run of the custom cluster tests shows several instances of the
> following data race during impalad shutdown:
> {code:java}
> WARNING: ThreadSanitizer: data race (pid=12660)
> Read of size 8 at 0x07786f60 by thread T338:
> #0 std::unique_ptr
> >::~unique_ptr()
> /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/bits/unique_ptr.h:235:6
> (impalad+0x19bd895)
> #1 at_exit_wrapper(void*)
> /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:361
> (impalad+0x191cf13)
> #2 impala::ImpalaServer::StartShutdown(long,
> impala::ShutdownStatusPB*)::$_2::operator()() const
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/service/impala-server.cc:2622:57
> (impalad+0x21bd871)
> #3
> boost::detail::function::void_function_obj_invoker0 impala::ShutdownStatusPB*)::$_2,
> void>::invoke(boost::detail::function::function_buffer&)
> /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:159:11
> (impalad+0x21bd6d9)
> #4 boost::function0::operator()() const
> /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:770:14
> (impalad+0x1e192b1)
> #5 impala::Thread::SuperviseThread(std::string const&, std::string
> const&, boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*)
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/thread.cc:360:3
> (impalad+0x23df196)
> #6 void boost::_bi::list5,
> boost::_bi::value, boost::_bi::value >,
> boost::_bi::value,
> boost::_bi::value*>
> >::operator() boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*),
> boost::_bi::list0>(boost::_bi::type, void (*&)(std::string const&,
> std::string const&, boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*), boost::_bi::list0&, int)
> /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:531:9
> (impalad+0x23e735c)
> #7 boost::_bi::bind_t const&, boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*),
> boost::_bi::list5,
> boost::_bi::value, boost::_bi::value >,
> boost::_bi::value,
> boost::_bi::value*> >
> >::operator()()
> /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16
> (impalad+0x23e7273)
> #8 boost::detail::thread_data (*)(std::string const&, std::string const&, boost::function,
> impala::ThreadDebugInfo const*, impala::Promise (impala::PromiseMode)0>*), boost::_bi::list5,
> boost::_bi::value, boost::_bi::value >,
> boost::_bi::value,
> boost::_bi::value*> > >
> >::run()
> /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/thread/detail/thread.hpp:116:17
> (impalad+0x23e6f60)
> #9 thread_proxy (impalad+0x30e44f9)
> Previous write of size 8 at 0x07786f60 by main thread:
> #0 void std::swap(impala::Thread*&, impala::Thread*&)
> /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/bits/move.h:176:11
> (impalad+0x221f370)
> #1 std::unique_ptr
> >::reset(impala::Thread*)
> /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/bits/unique_ptr.h:342:2
> (impalad+0x221a5ab)
> #2 std::unique_ptr
> >::operator=(std::unique_ptr std::default_delete >&&)
> /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/bits/unique_ptr.h:251:2
> (impalad+0x23e1ad8)
> #3 impala::Thread::StartThread(std::string const&, std::string const&,
> boost::function const&, std::unique_ptr std::default_delete >*, bool)
>
[jira] [Created] (IMPALA-9739) TSAN data races during impalad shutdown
Sahil Takiar created IMPALA-9739:
Summary: TSAN data races during impalad shutdown
Key: IMPALA-9739
URL: https://issues.apache.org/jira/browse/IMPALA-9739
Project: IMPALA
Issue Type: Sub-task
Components: Backend
Reporter: Sahil Takiar
A TSAN run of the custom cluster tests shows several instances of the following
data race during impalad shutdown:
{code:java}
WARNING: ThreadSanitizer: data race (pid=12660)
Read of size 8 at 0x07786f60 by thread T338:
#0 std::unique_ptr
>::~unique_ptr()
/data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/bits/unique_ptr.h:235:6
(impalad+0x19bd895)
#1 at_exit_wrapper(void*)
/mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:361
(impalad+0x191cf13)
#2 impala::ImpalaServer::StartShutdown(long,
impala::ShutdownStatusPB*)::$_2::operator()() const
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/service/impala-server.cc:2622:57
(impalad+0x21bd871)
#3
boost::detail::function::void_function_obj_invoker0::invoke(boost::detail::function::function_buffer&)
/data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:159:11
(impalad+0x21bd6d9)
#4 boost::function0::operator()() const
/data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:770:14
(impalad+0x1e192b1)
#5 impala::Thread::SuperviseThread(std::string const&, std::string const&,
boost::function, impala::ThreadDebugInfo const*, impala::Promise*)
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/thread.cc:360:3
(impalad+0x23df196)
#6 void boost::_bi::list5,
boost::_bi::value, boost::_bi::value >,
boost::_bi::value,
boost::_bi::value*>
>::operator(), impala::ThreadDebugInfo const*, impala::Promise*), boost::_bi::list0>(boost::_bi::type, void
(*&)(std::string const&, std::string const&, boost::function,
impala::ThreadDebugInfo const*, impala::Promise*), boost::_bi::list0&, int)
/data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:531:9
(impalad+0x23e735c)
#7 boost::_bi::bind_t, impala::ThreadDebugInfo const*,
impala::Promise*),
boost::_bi::list5,
boost::_bi::value, boost::_bi::value >,
boost::_bi::value,
boost::_bi::value*> >
>::operator()()
/data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16
(impalad+0x23e7273)
#8 boost::detail::thread_data, impala::ThreadDebugInfo
const*, impala::Promise*),
boost::_bi::list5,
boost::_bi::value, boost::_bi::value >,
boost::_bi::value,
boost::_bi::value*> > > >::run()
/data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/thread/detail/thread.hpp:116:17
(impalad+0x23e6f60)
#9 thread_proxy (impalad+0x30e44f9)
Previous write of size 8 at 0x07786f60 by main thread:
#0 void std::swap(impala::Thread*&, impala::Thread*&)
/data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/bits/move.h:176:11
(impalad+0x221f370)
#1 std::unique_ptr
>::reset(impala::Thread*)
/data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/bits/unique_ptr.h:342:2
(impalad+0x221a5ab)
#2 std::unique_ptr
>::operator=(std::unique_ptr >&&)
/data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/bits/unique_ptr.h:251:2
(impalad+0x23e1ad8)
#3 impala::Thread::StartThread(std::string const&, std::string const&,
boost::function const&, std::unique_ptr >*, bool)
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/thread.cc:329:11
(impalad+0x23decac)
#4 impala::Status impala::Thread::Create(std::string const&,
std::string const&, void (* const&)(), std::unique_ptr >*, bool)
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/thread.h:74:12
(impalad+0x1a1cd8c)
#5 impala::StartImpalaShutdownSignalHandlerThread()
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/common/init.cc:407:10
(impalad+0x1a1c1e8)
#6 ImpaladMain(int, char**)
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/service/impalad-main.cc:96:3
(impalad+0x21a525a)
#7 main
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/service/daemon-main.cc:37:12
(impalad+0x19b860a)
As if synchronized via sleep:
#0
[jira] [Commented] (IMPALA-9716) Add jitter to the exponential backoff in status reporting
[ https://issues.apache.org/jira/browse/IMPALA-9716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103906#comment-17103906 ] ASF subversion and git services commented on IMPALA-9716: - Commit e8d17948fb94e9a5cf8b6cfce9e05d51ebb65492 in impala's branch refs/heads/master from Thomas Tauber-Marshall [ https://gitbox.apache.org/repos/asf?p=impala.git;h=e8d1794 ] IMPALA-9716: Add jitter to the exponential backoff in status reporting When status reports fail, we use exponential backoff when retrying sending them. However, currently the backoff is deterministic, leading to a thundering herd problem where all of the backends for a particular query may try to report at the same time, the coordinator is overwhelmed and rejects some of the rpcs, then the backends all backoff by the same amount and retry sending at the same time, leading the coordinator to be overwhelmed again. This patch alleviates this problem by adding some random jitter to the exponential backoff used when a status report fails. Testing: - Passed a full run of existing tests. - Code path is covered by test_reportexecstatus_retries Change-Id: Id05c224517aa606057117328f480dfa98676b923 Reviewed-on: http://gerrit.cloudera.org:8080/15860 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Add jitter to the exponential backoff in status reporting > - > > Key: IMPALA-9716 > URL: https://issues.apache.org/jira/browse/IMPALA-9716 > Project: IMPALA > Issue Type: Improvement > Components: Distributed Exec >Affects Versions: Impala 4.0 >Reporter: Thomas Tauber-Marshall >Assignee: Thomas Tauber-Marshall >Priority: Major > > When status reports fail, we use exponential backoff when retrying sending > them. However, currently the backoff is deterministic, leading to a > thundering herd problem where all of the backends for a particular query may > try to report at the same time, the coordinator is overwhelmed and rejects > some of the rpcs, then the backends all backoff by the same amount and retry > sending at the same time, leading the coordinator to be overwhelmed again. > We can help solve this by adding some random jitter to the exponential > backoff time. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9570) Document about memory management for writing UDAFs
[ https://issues.apache.org/jira/browse/IMPALA-9570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103905#comment-17103905 ] ASF subversion and git services commented on IMPALA-9570: - Commit 4846751c84b70673ea25cd88e8c9d2085f7ae55e in impala's branch refs/heads/master from Shajini Thayasingh [ https://gitbox.apache.org/repos/asf?p=impala.git;h=4846751 ] IMPALA-9570: [DOCS] add memory management add memory management and fix broken links. Incorporated review changes. Change-Id: I6e8b6d0c3fe2e1746831665b3d3ae98a0beaa1e7 Reviewed-on: http://gerrit.cloudera.org:8080/15836 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Document about memory management for writing UDAFs > -- > > Key: IMPALA-9570 > URL: https://issues.apache.org/jira/browse/IMPALA-9570 > Project: IMPALA > Issue Type: Documentation >Reporter: Quanlong Huang >Assignee: shajini thayasingh >Priority: Major > Attachments: apache doc.png, cloudera doc.png, > image-2020-04-24-14-11-18-449.png > > > In the current document, we have few words about memory management for > writing UDAFs: > [https://impala.apache.org/docs/build/html/topics/impala_udf.html] > The only related sentense is "A serialize function that ... frees any memory > allocated during the init, update, and merge phases." > We can add more for clarifying it or add this link for further reading: > [https://github.com/apache/impala/blob/2f53783b14ddf215a4c96bc52a511d3a83896128/be/src/udf/udf.h#L358-L365] > BTW, the links of uda-sample.h and uda-sample.cc are not shown: > [http://impala.apache.org/docs/build/html/topics/impala_udf.html#uda_functions] > !apache doc.png|width=326,height=109! > They are normal in Cloudera's doc: > [https://docs.cloudera.com/documentation/enterprise/latest/topics/impala_udf.html#uda_functions] > !cloudera doc.png|width=725,height=226! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9736) "MT_DOP not supported for plans with base table joins or table sinks" error is out of date
[ https://issues.apache.org/jira/browse/IMPALA-9736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103907#comment-17103907 ] ASF subversion and git services commented on IMPALA-9736: - Commit dcf497910a8db4796985fa4b01a37e747b12029a in impala's branch refs/heads/master from Tim Armstrong [ https://gitbox.apache.org/repos/asf?p=impala.git;h=dcf4979 ] IMPALA-9736: fix mt_dop not supported error The error was not accurate, because joins are now supported. Also updated it to refer to DML statements instead of table sinks to be more user-appropriate. Change-Id: I8eb8106f86c47a14cc951c4a77966fe51b5c30e3 Reviewed-on: http://gerrit.cloudera.org:8080/15884 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > "MT_DOP not supported for plans with base table joins or table sinks" error > is out of date > -- > > Key: IMPALA-9736 > URL: https://issues.apache.org/jira/browse/IMPALA-9736 > Project: IMPALA > Issue Type: Bug > Components: Frontend >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > > We do support base table joins now. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
