[
https://issues.apache.org/jira/browse/IMPALA-8118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16752498#comment-16752498
]
Tim Armstrong commented on IMPALA-8118:
---------------------------------------
>From ee_tests/impalad.ERROR
{noformat}
E0125 05:25:21.153029 65917 krpc-data-stream-sender.cc:344]
974ec7e179ce3698:6f26a17d00000005] channel send to 127.0.0.1:27002 failed:
(fragment_instance_id=974ec7e179ce3698:6f26a17d0000000b): Mem
ory limit exceeded: Failed to allocate row batch
EXCHANGE_NODE (id=11) could not allocate 32.00 KB without exceeding limit.
Error occurred on backend
impala-ec2-centos74-r4-4xlarge-ondemand-1162.vpc.cloudera.com:22002
Memory left in process limit: 11.08 GB
Memory left in query limit: 1.83 KB
Query(974ec7e179ce3698:6f26a17d00000000): Limit=130.00 MB Reservation=80.88 MB
ReservationLimit=98.00 MB OtherMemory=49.12 MB Total=130.00 MB Peak=130.01 MB
Unclaimed reservations: Reservation=26.88 MB OtherMemory=0 Total=26.88 MB
Peak=70.88 MB
Fragment 974ec7e179ce3698:6f26a17d0000000e: Reservation=0 OtherMemory=1.67 MB
Total=1.67 MB Peak=1.67 MB
SORT_NODE (id=9): Total=0 Peak=0
AGGREGATION_NODE (id=16): Reservation=0 OtherMemory=61.12 KB Total=61.12 KB
Peak=61.12 KB
GroupingAggregator 0: Total=61.12 KB Peak=61.12 KB
Exprs: Total=61.12 KB Peak=61.12 KB
EXCHANGE_NODE (id=15): Reservation=0 OtherMemory=0 Total=0 Peak=0
KrpcDeferredRpcs: Total=0 Peak=0
KrpcDataStreamSender (dst_id=17): Total=1.97 KB Peak=1.97 KB
CodeGen: Total=1.60 MB Peak=1.60 MB
Fragment 974ec7e179ce3698:6f26a17d0000000b: Reservation=3.00 MB
OtherMemory=40.26 MB Total=43.26 MB Peak=43.26 MB
Runtime Filter Bank: Reservation=3.00 MB ReservationLimit=3.00 MB
OtherMemory=0 Total=3.00 MB Peak=3.00 MB
AGGREGATION_NODE (id=8): Reservation=0 OtherMemory=61.12 KB Total=61.12 KB
Peak=61.12 KB
GroupingAggregator 0: Total=61.12 KB Peak=61.12 KB
Exprs: Total=61.12 KB Peak=61.12 KB
HASH_JOIN_NODE (id=7): Total=58.25 KB Peak=58.25 KB
Exprs: Total=13.12 KB Peak=13.12 KB
Hash Join Builder (join_node_id=7): Total=13.12 KB Peak=13.12 KB
Hash Join Builder (join_node_id=7) Exprs: Total=13.12 KB Peak=13.12 KB
HASH_JOIN_NODE (id=6): Total=50.25 KB Peak=50.25 KB
Exprs: Total=13.12 KB Peak=13.12 KB
Hash Join Builder (join_node_id=6): Total=13.12 KB Peak=13.12 KB
Hash Join Builder (join_node_id=6) Exprs: Total=13.12 KB Peak=13.12 KB
HASH_JOIN_NODE (id=5): Total=42.25 KB Peak=42.25 KB
Exprs: Total=13.12 KB Peak=13.12 KB
Hash Join Builder (join_node_id=5): Total=13.12 KB Peak=13.12 KB
Hash Join Builder (join_node_id=5) Exprs: Total=13.12 KB Peak=13.12 KB
EXCHANGE_NODE (id=10): Reservation=10.20 MB OtherMemory=19.29 KB
Total=10.21 MB Peak=10.21 MB
KrpcDeferredRpcs: Total=19.29 KB Peak=19.29 KB
EXCHANGE_NODE (id=11): Reservation=13.14 MB OtherMemory=0 Total=13.14 MB
Peak=13.14 MB
KrpcDeferredRpcs: Total=0 Peak=0
EXCHANGE_NODE (id=12): Reservation=10.30 MB OtherMemory=0 Total=10.30 MB
Peak=10.30 MB
KrpcDeferredRpcs: Total=0 Peak=0
AGGREGATION_NODE (id=14): Reservation=0 OtherMemory=17.12 KB Total=17.12 KB
Peak=17.12 KB
GroupingAggregator 0: Total=17.12 KB Peak=17.12 KB
Exprs: Total=17.12 KB Peak=17.12 KB
:
EXCHANGE_NODE (id=13): Reservation=360.00 KB OtherMemory=0 Total=360.00 KB
Peak=360.00 KB
KrpcDeferredRpcs: Total=0 Peak=0
KrpcDataStreamSender (dst_id=15): Total=5.91 KB Peak=5.91 KB
CodeGen: Total=6.01 MB Peak=6.01 MB
Fragment 974ec7e179ce3698:6f26a17d00000002: Reservation=10.00 MB
OtherMemory=1.80 MB Total=11.80 MB Peak=11.82 MB
Runtime Filter Bank: Reservation=2.00 MB ReservationLimit=2.00 MB
OtherMemory=0 Total=2.00 MB Peak=2.00 MB
HDFS_SCAN_NODE (id=2): Reservation=8.00 MB OtherMemory=1.68 MB Total=9.68
MB Peak=9.70 MB
Queued Batches: Total=1.25 MB Peak=1.27 MB
KrpcDataStreamSender (dst_id=10): Total=90.57 KB Peak=106.57 KB
CodeGen: Total=3.28 KB Peak=403.50 KB
Fragment 974ec7e179ce3698:6f26a17d00000004: Reservation=28.00 MB
OtherMemory=3.58 MB Total=31.58 MB Peak=31.62 MB
Runtime Filter Bank: Reservation=2.00 MB ReservationLimit=2.00 MB
OtherMemory=0 Total=2.00 MB Peak=2.00 MB
HDFS_SCAN_NODE (id=1): Reservation=26.00 MB OtherMemory=3.48 MB Total=29.48
MB Peak=29.52 MB
Queued Batches: Total=2.24 MB Peak=2.29 MB
KrpcDataStreamSender (dst_id=11): Total=54.36 KB Peak=94.36 KB
CodeGen: Total=3.28 KB Peak=412.50 KB
Fragment 974ec7e179ce3698:6f26a17d00000008: Reservation=13.00 MB
OtherMemory=1.81 MB Total=14.81 MB Peak=14.86 MB
AGGREGATION_NODE (id=4): Reservation=5.00 MB OtherMemory=42.12 KB
Total=5.04 MB Peak=5.04 MB
GroupingAggregator 0: Reservation=5.00 MB OtherMemory=17.12 KB Total=5.02
MB Peak=5.02 MB
Exprs: Total=17.12 KB Peak=17.12 KB
HDFS_SCAN_NODE (id=3): Reservation=8.00 MB OtherMemory=1.66 MB Total=9.66
MB Peak=9.70 MB
Queued Batches: Total=1.23 MB Peak=1.27 MB
KrpcDataStreamSender (dst_id=13): Total=83.35 KB Peak=99.35 KB
CodeGen: Total=6.09 KB Peak=650.00 KB
=================================================================
==5504==ERROR: AddressSanitizer: use-after-poison on address 0x631001cd518c at
pc 0x000002012a0c bp 0x7fe2fda4c640 sp 0x7fe2fda4c638
READ of size 1 at 0x631001cd518c thread T113733
#0 0x2012a0b in impala::Tuple::IsNull(impala::NullIndicatorOffset const&)
const
/data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/runtime/tuple.h:241:13
#1 0x28088c1 in impala::AggFnEvaluator::SerializeOrFinalize(impala::Tuple*,
impala::SlotDescriptor const&, impala::Tuple*, void*)
/data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/exprs/agg-fn-evaluator.cc:393:29
#2 0x27740b8 in
impala::AggFnEvaluator::Finalize(std::vector<impala::AggFnEvaluator*,
std::allocator<impala::AggFnEvaluator*> > const&, impala::Tuple*,
impala::Tuple*)
/data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/exprs/agg-fn-evaluator.h:307:15
#3 0x27aa286 in
impala::GroupingAggregator::CleanupHashTbl(std::vector<impala::AggFnEvaluator*,
std::allocator<impala::AggFnEvaluator*> > const&, impala::HashTable::Iterator)
/data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/exec/grouping-aggregator.cc:351:7
#4 0x27aa7a2 in impala::GroupingAggregator::ClosePartitions()
/data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/exec/grouping-aggregator.cc:930:5
#5 0x27aaae4 in impala::GroupingAggregator::Close(impala::RuntimeState*)
/data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/exec/grouping-aggregator.cc:383:3
#6 0x275fce7 in impala::AggregationNode::Close(impala::RuntimeState*)
/data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/exec/aggregation-node.cc:139:32
#7 0x2067e49 in impala::FragmentInstanceState::Close()
/data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/runtime/fragment-instance-state.cc:368:42
#8 0x206317a in impala::FragmentInstanceState::Exec()
/data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/runtime/fragment-instance-state.cc:99:3
#9 0x207d432 in
impala::QueryState::ExecFInstance(impala::FragmentInstanceState*)
/data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/runtime/query-state.cc:584:24
#10 0x1d77076 in boost::function0<void>::operator()() const
/data/jenkins/workspace/impala-cdh6.x-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:766:14
#11 0x24b8326 in impala::Thread::SuperviseThread(std::string const&,
std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
impala::Promise<long, (impala::PromiseMode)0>*)
/data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/util/thread.cc:359:3
#12 0x24c3818 in void boost::_bi::list5<boost::_bi::value<std::string>,
boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >,
boost::_bi::value<impala::ThreadDebugInfo*>,
boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*>
>::operator()<void (*)(std::string const&, std::string const&,
boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long,
(impala::PromiseMode)0>*), boost::_bi::list0>(boost::_bi::type<void>, void
(*&)(std::string const&, std::string const&, boost::function<void ()>,
impala::ThreadDebugInfo const*, impala::Promise<long,
(impala::PromiseMode)0>*), boost::_bi::list0&, int)
/data/jenkins/workspace/impala-cdh6.x-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:525:9
#13 0x24c366b in boost::_bi::bind_t<void, void (*)(std::string const&,
std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
impala::Promise<long, (impala::PromiseMode)0>*),
boost::_bi::list5<boost::_bi::value<std::string>,
boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >,
boost::_bi::value<impala::ThreadDebugInfo*>,
boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >
>::operator()()
/data/jenkins/workspace/impala-cdh6.x-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20:27
#14 0x3c7fbd9 in thread_proxy
(/data0/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/build/debug/service/impalad+0x3c7fbd9)
#15 0x7fecdb4a4e24 in start_thread (/lib64/libpthread.so.0+0x7e24)
#16 0x7fecdafbb34c in __clone (/lib64/libc.so.6+0xf834c)
0x631001cd518c is located 396 bytes inside of 65536-byte region
[0x631001cd5000,0x631001ce5000)
allocated by thread T113733 here:
#0 0x177f2e5 in posix_memalign
/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent70-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:148
#1 0x250ccb1 in impala::SystemAllocator::AllocateViaMalloc(long, unsigned
char**)
/data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/runtime/bufferpool/system-allocator.cc:123:12
#2 0x250c339 in impala::SystemAllocator::Allocate(long,
impala::BufferPool::BufferHandle*)
/data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/runtime/bufferpool/system-allocator.cc:67:5
#3 0x250f759 in
impala::BufferPool::BufferAllocator::AllocateInternal(impala::BufferPool::Client*,
long, impala::BufferPool::BufferHandle*)
/data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/runtime/bufferpool/buffer-allocator.cc:362:38
#4 0x250eaeb in
impala::BufferPool::BufferAllocator::Allocate(impala::BufferPool::ClientHandle*,
long, impala::BufferPool::BufferHandle*)
/data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/runtime/bufferpool/buffer-allocator.cc:269:3
#5 0x24f2399 in
impala::BufferPool::AllocateBuffer(impala::BufferPool::ClientHandle*, long,
impala::BufferPool::BufferHandle*)
/data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/runtime/bufferpool/buffer-pool.cc:232:31
#6 0x24f1f2e in
impala::BufferPool::CreatePage(impala::BufferPool::ClientHandle*, long,
impala::BufferPool::PageHandle*, impala::BufferPool::BufferHandle const**)
/data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/runtime/bufferpool/buffer-pool.cc:145:3
#7 0x2ce8255 in impala::BufferedTupleStream::NewWritePage(long)
/data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/runtime/buffered-tuple-stream.cc:371:3
#8 0x2ce9016 in impala::BufferedTupleStream::AdvanceWritePage(long, bool*)
/data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/runtime/buffered-tuple-stream.cc:443:3
#9 0x2ced09b in impala::BufferedTupleStream::AddRowCustomBeginSlow(long,
impala::Status*)
/data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/runtime/buffered-tuple-stream.cc:905:13
#10 0x27adc5e in impala::BufferedTupleStream::AddRowCustomBegin(long,
impala::Status*)
/data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/runtime/buffered-tuple-stream.inline.h:38:12
#11 0x27adc5e in
impala::GroupingAggregator::ConstructIntermediateTuple(std::vector<impala::AggFnEvaluator*,
std::allocator<impala::AggFnEvaluator*> > const&,
impala::BufferedTupleStream*, impala::Status*)
/data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/exec/grouping-aggregator.cc:497
#12 0x7fe416b19f11 (<unknown module>)
{noformat}
> Build failure: Connection failure in multiple tests
> ---------------------------------------------------
>
> Key: IMPALA-8118
> URL: https://issues.apache.org/jira/browse/IMPALA-8118
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 3.1.0
> Reporter: Paul Rogers
> Assignee: Lenisha Gandhi
> Priority: Blocker
> Fix For: Impala 3.1.0
>
>
> Build of latest master, with ASAN, failed with the following error, which to
> my newbie eyes looks like a connection failure:
> {noformat}
> 05:42:04 =================================== FAILURES
> ===================================
> 05:42:04 TestQueriesTextTables.test_data_source_tables[protocol: beeswax |
> exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none]
> 05:42:04 [gw5] linux2 -- Python 2.7.5
> /data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/bin/../infra/python/env/bin/python
> 05:42:04 query_test/test_queries.py:174: in test_data_source_tables
> 05:42:04 self.run_test_case('QueryTest/data-source-tables', vector)
> 05:42:04 common/impala_test_suite.py:472: in run_test_case
> 05:42:04 result = self.__execute_query(target_impalad_client, query,
> user=user)
> ...
> 05:42:04 handle = self.execute_query_async(query_string, user=user)
> 05:42:04 beeswax/impala_beeswax.py:351: in execute_query_async
> 05:42:04 handle = self.__do_rpc(lambda: self.imp_service.query(query,))
> 05:42:04 beeswax/impala_beeswax.py:512: in __do_rpc
> 05:42:04 raise ImpalaBeeswaxException(self.__build_error_message(e), e)
> 05:42:04 E ImpalaBeeswaxException: ImpalaBeeswaxException:
> 05:42:04 E INNER EXCEPTION: <class
> 'thrift.transport.TTransport.TTransportException'>
> 05:42:04 E MESSAGE: TSocket read 0 bytes
> 05:42:04 ----------------------------- Captured stderr call
> -----------------------------
> ...
> 05:42:04 -- executing against localhost:21000
> 05:42:04 select *
> 05:42:04 from alltypes_datasource
> 05:42:04 where float_col != 0 and
> 05:42:04 int_col >= 1990 limit 5;
> {noformat}
> A similar error appears for multiple other tests. Then:
> {noformat}
> 05:42:04 TTransportException: Could not connect to localhost:21050
> 05:42:04 !!!!!!!!!!!!!!!!!!! Interrupted: stopping after 10 failures
> !!!!!!!!!!!!!!!!!!!!
> {noformat}
> I wonder if these are just symptoms of a failure in the BE code due to ASAN
> being enabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
