[jira] [Commented] (IMPALA-7030) crash in impala::PartitionedAggregationNode::ProcessBatchNoGrouping

2018-05-21 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/IMPALA-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16483012#comment-16483012
 ] 

ASF subversion and git services commented on IMPALA-7030:
-

Commit 75d19c874f2daf7e42231a257a97c07367660226 in impala's branch 
refs/heads/2.x from [~dhecht]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=75d19c8 ]

IMPALA-5384, part 2: Simplify Coordinator locking and clarify state

The is the final change to clarify and break up the Coordinator's lock.
The state machine for the coordinator is made explicit, distinguishing
between executing state and multiple terminal states. Logic to
transition into a terminal state is centralized in one location and
executes exactly once for each coordinator object.

Derived from a patch for IMPALA_5384 by Marcel Kornacker.

Testing:
- exhaustive functional tests
- stress test on minicluster with memory overcommitment. Verified from
  the logs that this exercises all these paths:
  - successful queries
  - client requested cancellation
  - error from exec FInstances RPC
  - error reported asynchronously via report status RPC
  - eos before backend execution completed
- loop query_test & failure for 12 hours with no dchecks or crashes
  (This had previously reproduced IMPALA-7030 and IMPALA-7033 with
  the previous version of this change).

Change-Id: I6dc08da1295f1df3c9dce6d35d65d887b2c00a1c
Reviewed-on: http://gerrit.cloudera.org:8080/10440
Reviewed-by: Dan Hecht 
Tested-by: Impala Public Jenkins 
Reviewed-on: http://gerrit.cloudera.org:8080/10465


>  crash in impala::PartitionedAggregationNode::ProcessBatchNoGrouping
> 
>
> Key: IMPALA-7030
> URL: https://issues.apache.org/jira/browse/IMPALA-7030
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Michael Brown
>Assignee: Dan Hecht
>Priority: Blocker
> Attachments: crash.dump.gz, gdb.out.gz, hs_err_pid1621.log.gz
>
>
> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/2176/
> {noformat}
> #0  0x7fc896430428 in __GI_raise (sig=sig@entry=6) at 
> ../sysdeps/unix/sysv/linux/raise.c:54
> #1  0x7fc89643202a in __GI_abort () at abort.c:89
> #2  0x7fc899379c59 in os::abort(bool) (dump_core=) at 
> /build/openjdk-8-wnL82d/openjdk-8-8u171-b11/src/hotspot/src/os/linux/vm/os_linux.cpp:1509
> #3  0x7fc89952f047 in VMError::report_and_die() 
> (this=this@entry=0x7fc7e90287d0) at 
> /build/openjdk-8-wnL82d/openjdk-8-8u171-b11/src/hotspot/src/share/vm/utilities/vmError.cpp:1060
> #4  0x7fc8993836ef in JVM_handle_linux_signal(int, siginfo_t*, void*, 
> int) (sig=sig@entry=11, info=info@entry=0x7fc7e9028a70, 
> ucVoid=ucVoid@entry=0x7fc7e9028940, 
> abort_if_unrecognized=abort_if_unrecognized@entry=1)
> at 
> /build/openjdk-8-wnL82d/openjdk-8-8u171-b11/src/hotspot/src/os_cpu/linux_x86/vm/os_linux_x86.cpp:541
> #5  0x7fc899376d88 in signalHandler(int, siginfo_t*, void*) (sig=11, 
> info=0x7fc7e9028a70, uc=0x7fc7e9028940) at 
> /build/openjdk-8-wnL82d/openjdk-8-8u171-b11/src/hotspot/src/os/linux/vm/os_linux.cpp:4432
> #6  0x7fc8967d6390 in  () at 
> /lib/x86_64-linux-gnu/libpthread.so.0
> #7  0x7fc8584ca000 in 
> impala::PartitionedAggregationNode::ProcessBatchNoGrouping(impala::RowBatch*) 
> [clone .1] ()
> #8  0x02cd5bcf in 
> impala::PartitionedAggregationNode::Open(impala::RuntimeState*) 
> (this=0x15795200, state=0x14e95d40) at 
> /home/ubuntu/Impala/be/src/exec/partitioned-aggregation-node.cc:314
> #9  0x01c94775 in impala::FragmentInstanceState::Open() 
> (this=0x1cc19e00) at 
> /home/ubuntu/Impala/be/src/runtime/fragment-instance-state.cc:268
> #10 0x01c91faf in impala::FragmentInstanceState::Exec() 
> (this=0x1cc19e00) at 
> /home/ubuntu/Impala/be/src/runtime/fragment-instance-state.cc:81
> #11 0x01ca175b in 
> impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) 
> (this=0x3a1f6000, fis=0x1cc19e00) at 
> /home/ubuntu/Impala/be/src/runtime/query-state.cc:401
> #12 0x01c9ffce in impala::QueryState::::operator()(void) 
> const (__closure=0x7fc7e9029ce8) at 
> /home/ubuntu/Impala/be/src/runtime/query-state.cc:341
> #13 0x01ca2479 in 
> boost::detail::function::void_function_obj_invoker0,
>  void>::invoke(boost::detail::function::function_buffer &) 
> (function_obj_ptr=...)
> at 
> /home/ubuntu/Impala/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:153
> #14 0x01bd9e58 in boost::function0::operator()() const 
> (this=0x7fc7e9029ce0) at 
> /home/ubuntu/Impala/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
>

[jira] [Commented] (IMPALA-7030) crash in impala::PartitionedAggregationNode::ProcessBatchNoGrouping

2018-05-17 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/IMPALA-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480166#comment-16480166
 ] 

ASF subversion and git services commented on IMPALA-7030:
-

Commit c1c122a10177920903009420d2faac673d867c4b in impala's branch 
refs/heads/master from [~dhecht]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=c1c122a ]

IMPALA-5384, part 2: Simplify Coordinator locking and clarify state

The is the final change to clarify and break up the Coordinator's lock.
The state machine for the coordinator is made explicit, distinguishing
between executing state and multiple terminal states. Logic to
transition into a terminal state is centralized in one location and
executes exactly once for each coordinator object.

Derived from a patch for IMPALA_5384 by Marcel Kornacker.

Testing:
- exhaustive functional tests
- stress test on minicluster with memory overcommitment. Verified from
  the logs that this exercises all these paths:
  - successful queries
  - client requested cancellation
  - error from exec FInstances RPC
  - error reported asynchronously via report status RPC
  - eos before backend execution completed
- loop query_test & failure for 12 hours with no dchecks or crashes
  (This had previously reproduced IMPALA-7030 and IMPALA-7033 with
  the previous version of this change).

Change-Id: I6dc08da1295f1df3c9dce6d35d65d887b2c00a1c
Reviewed-on: http://gerrit.cloudera.org:8080/10440
Reviewed-by: Dan Hecht 
Tested-by: Impala Public Jenkins 


>  crash in impala::PartitionedAggregationNode::ProcessBatchNoGrouping
> 
>
> Key: IMPALA-7030
> URL: https://issues.apache.org/jira/browse/IMPALA-7030
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Michael Brown
>Assignee: Dan Hecht
>Priority: Blocker
> Attachments: crash.dump.gz, gdb.out.gz, hs_err_pid1621.log.gz
>
>
> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/2176/
> {noformat}
> #0  0x7fc896430428 in __GI_raise (sig=sig@entry=6) at 
> ../sysdeps/unix/sysv/linux/raise.c:54
> #1  0x7fc89643202a in __GI_abort () at abort.c:89
> #2  0x7fc899379c59 in os::abort(bool) (dump_core=) at 
> /build/openjdk-8-wnL82d/openjdk-8-8u171-b11/src/hotspot/src/os/linux/vm/os_linux.cpp:1509
> #3  0x7fc89952f047 in VMError::report_and_die() 
> (this=this@entry=0x7fc7e90287d0) at 
> /build/openjdk-8-wnL82d/openjdk-8-8u171-b11/src/hotspot/src/share/vm/utilities/vmError.cpp:1060
> #4  0x7fc8993836ef in JVM_handle_linux_signal(int, siginfo_t*, void*, 
> int) (sig=sig@entry=11, info=info@entry=0x7fc7e9028a70, 
> ucVoid=ucVoid@entry=0x7fc7e9028940, 
> abort_if_unrecognized=abort_if_unrecognized@entry=1)
> at 
> /build/openjdk-8-wnL82d/openjdk-8-8u171-b11/src/hotspot/src/os_cpu/linux_x86/vm/os_linux_x86.cpp:541
> #5  0x7fc899376d88 in signalHandler(int, siginfo_t*, void*) (sig=11, 
> info=0x7fc7e9028a70, uc=0x7fc7e9028940) at 
> /build/openjdk-8-wnL82d/openjdk-8-8u171-b11/src/hotspot/src/os/linux/vm/os_linux.cpp:4432
> #6  0x7fc8967d6390 in  () at 
> /lib/x86_64-linux-gnu/libpthread.so.0
> #7  0x7fc8584ca000 in 
> impala::PartitionedAggregationNode::ProcessBatchNoGrouping(impala::RowBatch*) 
> [clone .1] ()
> #8  0x02cd5bcf in 
> impala::PartitionedAggregationNode::Open(impala::RuntimeState*) 
> (this=0x15795200, state=0x14e95d40) at 
> /home/ubuntu/Impala/be/src/exec/partitioned-aggregation-node.cc:314
> #9  0x01c94775 in impala::FragmentInstanceState::Open() 
> (this=0x1cc19e00) at 
> /home/ubuntu/Impala/be/src/runtime/fragment-instance-state.cc:268
> #10 0x01c91faf in impala::FragmentInstanceState::Exec() 
> (this=0x1cc19e00) at 
> /home/ubuntu/Impala/be/src/runtime/fragment-instance-state.cc:81
> #11 0x01ca175b in 
> impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) 
> (this=0x3a1f6000, fis=0x1cc19e00) at 
> /home/ubuntu/Impala/be/src/runtime/query-state.cc:401
> #12 0x01c9ffce in impala::QueryState::::operator()(void) 
> const (__closure=0x7fc7e9029ce8) at 
> /home/ubuntu/Impala/be/src/runtime/query-state.cc:341
> #13 0x01ca2479 in 
> boost::detail::function::void_function_obj_invoker0,
>  void>::invoke(boost::detail::function::function_buffer &) 
> (function_obj_ptr=...)
> at 
> /home/ubuntu/Impala/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:153
> #14 0x01bd9e58 in boost::function0::operator()() const 
> (this=0x7fc7e9029ce0) at 
> /home/ubuntu/Impala/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
> #15 0x01ec50a9 in

[jira] [Commented] (IMPALA-7030) crash in impala::PartitionedAggregationNode::ProcessBatchNoGrouping

2018-05-16 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/IMPALA-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16477563#comment-16477563
 ] 

ASF subversion and git services commented on IMPALA-7030:
-

Commit 0e6117be3be8475ea1aa727dec55979cbffd05b7 in impala's branch 
refs/heads/2.x from [~dhecht]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=0e6117b ]

IMPALA-7033/IMPALA-7030: Backout suspected change leading to crash

Revert "IMPALA-5384, part 2: Simplify Coordinator locking and clarify state"

This reverts commit 6ca87e46736a1e591ed7d7d5fee05b4b4d2fbb50.

Change-Id: Idc63006e6e04130b2873a6a9730e434c563327c5
Reviewed-on: http://gerrit.cloudera.org:8080/10412
Reviewed-by: Tim Armstrong 
Tested-by: Impala Public Jenkins 
Reviewed-on: http://gerrit.cloudera.org:8080/10423
Reviewed-by: Dan Hecht 


>  crash in impala::PartitionedAggregationNode::ProcessBatchNoGrouping
> 
>
> Key: IMPALA-7030
> URL: https://issues.apache.org/jira/browse/IMPALA-7030
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Michael Brown
>Priority: Blocker
> Attachments: crash.dump.gz, gdb.out.gz, hs_err_pid1621.log.gz
>
>
> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/2176/
> {noformat}
> #0  0x7fc896430428 in __GI_raise (sig=sig@entry=6) at 
> ../sysdeps/unix/sysv/linux/raise.c:54
> #1  0x7fc89643202a in __GI_abort () at abort.c:89
> #2  0x7fc899379c59 in os::abort(bool) (dump_core=) at 
> /build/openjdk-8-wnL82d/openjdk-8-8u171-b11/src/hotspot/src/os/linux/vm/os_linux.cpp:1509
> #3  0x7fc89952f047 in VMError::report_and_die() 
> (this=this@entry=0x7fc7e90287d0) at 
> /build/openjdk-8-wnL82d/openjdk-8-8u171-b11/src/hotspot/src/share/vm/utilities/vmError.cpp:1060
> #4  0x7fc8993836ef in JVM_handle_linux_signal(int, siginfo_t*, void*, 
> int) (sig=sig@entry=11, info=info@entry=0x7fc7e9028a70, 
> ucVoid=ucVoid@entry=0x7fc7e9028940, 
> abort_if_unrecognized=abort_if_unrecognized@entry=1)
> at 
> /build/openjdk-8-wnL82d/openjdk-8-8u171-b11/src/hotspot/src/os_cpu/linux_x86/vm/os_linux_x86.cpp:541
> #5  0x7fc899376d88 in signalHandler(int, siginfo_t*, void*) (sig=11, 
> info=0x7fc7e9028a70, uc=0x7fc7e9028940) at 
> /build/openjdk-8-wnL82d/openjdk-8-8u171-b11/src/hotspot/src/os/linux/vm/os_linux.cpp:4432
> #6  0x7fc8967d6390 in  () at 
> /lib/x86_64-linux-gnu/libpthread.so.0
> #7  0x7fc8584ca000 in 
> impala::PartitionedAggregationNode::ProcessBatchNoGrouping(impala::RowBatch*) 
> [clone .1] ()
> #8  0x02cd5bcf in 
> impala::PartitionedAggregationNode::Open(impala::RuntimeState*) 
> (this=0x15795200, state=0x14e95d40) at 
> /home/ubuntu/Impala/be/src/exec/partitioned-aggregation-node.cc:314
> #9  0x01c94775 in impala::FragmentInstanceState::Open() 
> (this=0x1cc19e00) at 
> /home/ubuntu/Impala/be/src/runtime/fragment-instance-state.cc:268
> #10 0x01c91faf in impala::FragmentInstanceState::Exec() 
> (this=0x1cc19e00) at 
> /home/ubuntu/Impala/be/src/runtime/fragment-instance-state.cc:81
> #11 0x01ca175b in 
> impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) 
> (this=0x3a1f6000, fis=0x1cc19e00) at 
> /home/ubuntu/Impala/be/src/runtime/query-state.cc:401
> #12 0x01c9ffce in impala::QueryState::::operator()(void) 
> const (__closure=0x7fc7e9029ce8) at 
> /home/ubuntu/Impala/be/src/runtime/query-state.cc:341
> #13 0x01ca2479 in 
> boost::detail::function::void_function_obj_invoker0,
>  void>::invoke(boost::detail::function::function_buffer &) 
> (function_obj_ptr=...)
> at 
> /home/ubuntu/Impala/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:153
> #14 0x01bd9e58 in boost::function0::operator()() const 
> (this=0x7fc7e9029ce0) at 
> /home/ubuntu/Impala/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
> #15 0x01ec50a9 in impala::Thread::SuperviseThread(std::string const&, 
> std::string const&, boost::function, impala::ThreadDebugInfo const*, 
> impala::Promise*) (name="exec-finstance 
> (finst:844ac2a49a9a6486:396846db0005)", category="fragment-execution", 
> functor=..., parent_thread_info=0x7fc7ea82c990, 
> thread_started=0x7fc7ea82b940) at 
> /home/ubuntu/Impala/be/src/util/thread.cc:356
> #16 0x01ecd245 in boost::_bi::list5 boost::_bi::value, boost::_bi::value, 
> boost::_bi::value, 
> boost::_bi::value >::operator() const&, std::string const&, boost::function, impala::ThreadDebugInfo 
> const*, impala::Promise*),

[jira] [Commented] (IMPALA-7030) crash in impala::PartitionedAggregationNode::ProcessBatchNoGrouping

2018-05-15 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/IMPALA-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16476766#comment-16476766
 ] 

ASF subversion and git services commented on IMPALA-7030:
-

Commit 4fab42883184c247b951dba4237f4303c502d410 in impala's branch 
refs/heads/master from [~dhecht]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=4fab428 ]

IMPALA-7033/IMPALA-7030: Backout suspected change leading to crash

Revert "IMPALA-5384, part 2: Simplify Coordinator locking and clarify state"

This reverts commit 6ca87e46736a1e591ed7d7d5fee05b4b4d2fbb50.

Change-Id: Idc63006e6e04130b2873a6a9730e434c563327c5
Reviewed-on: http://gerrit.cloudera.org:8080/10412
Reviewed-by: Tim Armstrong 
Tested-by: Impala Public Jenkins 


>  crash in impala::PartitionedAggregationNode::ProcessBatchNoGrouping
> 
>
> Key: IMPALA-7030
> URL: https://issues.apache.org/jira/browse/IMPALA-7030
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Michael Brown
>Priority: Blocker
> Attachments: crash.dump.gz, gdb.out.gz, hs_err_pid1621.log.gz
>
>
> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/2176/
> {noformat}
> #0  0x7fc896430428 in __GI_raise (sig=sig@entry=6) at 
> ../sysdeps/unix/sysv/linux/raise.c:54
> #1  0x7fc89643202a in __GI_abort () at abort.c:89
> #2  0x7fc899379c59 in os::abort(bool) (dump_core=) at 
> /build/openjdk-8-wnL82d/openjdk-8-8u171-b11/src/hotspot/src/os/linux/vm/os_linux.cpp:1509
> #3  0x7fc89952f047 in VMError::report_and_die() 
> (this=this@entry=0x7fc7e90287d0) at 
> /build/openjdk-8-wnL82d/openjdk-8-8u171-b11/src/hotspot/src/share/vm/utilities/vmError.cpp:1060
> #4  0x7fc8993836ef in JVM_handle_linux_signal(int, siginfo_t*, void*, 
> int) (sig=sig@entry=11, info=info@entry=0x7fc7e9028a70, 
> ucVoid=ucVoid@entry=0x7fc7e9028940, 
> abort_if_unrecognized=abort_if_unrecognized@entry=1)
> at 
> /build/openjdk-8-wnL82d/openjdk-8-8u171-b11/src/hotspot/src/os_cpu/linux_x86/vm/os_linux_x86.cpp:541
> #5  0x7fc899376d88 in signalHandler(int, siginfo_t*, void*) (sig=11, 
> info=0x7fc7e9028a70, uc=0x7fc7e9028940) at 
> /build/openjdk-8-wnL82d/openjdk-8-8u171-b11/src/hotspot/src/os/linux/vm/os_linux.cpp:4432
> #6  0x7fc8967d6390 in  () at 
> /lib/x86_64-linux-gnu/libpthread.so.0
> #7  0x7fc8584ca000 in 
> impala::PartitionedAggregationNode::ProcessBatchNoGrouping(impala::RowBatch*) 
> [clone .1] ()
> #8  0x02cd5bcf in 
> impala::PartitionedAggregationNode::Open(impala::RuntimeState*) 
> (this=0x15795200, state=0x14e95d40) at 
> /home/ubuntu/Impala/be/src/exec/partitioned-aggregation-node.cc:314
> #9  0x01c94775 in impala::FragmentInstanceState::Open() 
> (this=0x1cc19e00) at 
> /home/ubuntu/Impala/be/src/runtime/fragment-instance-state.cc:268
> #10 0x01c91faf in impala::FragmentInstanceState::Exec() 
> (this=0x1cc19e00) at 
> /home/ubuntu/Impala/be/src/runtime/fragment-instance-state.cc:81
> #11 0x01ca175b in 
> impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) 
> (this=0x3a1f6000, fis=0x1cc19e00) at 
> /home/ubuntu/Impala/be/src/runtime/query-state.cc:401
> #12 0x01c9ffce in impala::QueryState::::operator()(void) 
> const (__closure=0x7fc7e9029ce8) at 
> /home/ubuntu/Impala/be/src/runtime/query-state.cc:341
> #13 0x01ca2479 in 
> boost::detail::function::void_function_obj_invoker0,
>  void>::invoke(boost::detail::function::function_buffer &) 
> (function_obj_ptr=...)
> at 
> /home/ubuntu/Impala/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:153
> #14 0x01bd9e58 in boost::function0::operator()() const 
> (this=0x7fc7e9029ce0) at 
> /home/ubuntu/Impala/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
> #15 0x01ec50a9 in impala::Thread::SuperviseThread(std::string const&, 
> std::string const&, boost::function, impala::ThreadDebugInfo const*, 
> impala::Promise*) (name="exec-finstance 
> (finst:844ac2a49a9a6486:396846db0005)", category="fragment-execution", 
> functor=..., parent_thread_info=0x7fc7ea82c990, 
> thread_started=0x7fc7ea82b940) at 
> /home/ubuntu/Impala/be/src/util/thread.cc:356
> #16 0x01ecd245 in boost::_bi::list5 boost::_bi::value, boost::_bi::value, 
> boost::_bi::value, 
> boost::_bi::value >::operator() const&, std::string const&, boost::function, impala::ThreadDebugInfo 
> const*, impala::Promise*), boost::_bi::list0>(boost::_bi::type, 
> void (*&)(std::string const&, std::string const&, boost::function, 
>

[jira] [Commented] (IMPALA-7030) crash in impala::PartitionedAggregationNode::ProcessBatchNoGrouping

2018-05-15 Thread Tim Armstrong (JIRA) 

[ 
https://issues.apache.org/jira/browse/IMPALA-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16476076#comment-16476076
 ] 

Tim Armstrong commented on IMPALA-7030:
---

I also saw a crash in codegen'd code on the same job. I wasn't able to get a 
backtrace but I reconfigured the job to collect the hs_err_pid* file in future. 
https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/2183/

{noformat}
Crash reason:  SIGSEGV
Crash address: 0x7f87bd39d000
Process uptime: not available

Thread 223 (crashed)
 0  0x7f87bd39d000
rax = 0x7f87bd39d000   rdx = 0x0ec12000
rcx = 0x0f547540   rbx = 0x
rsi = 0x0f547540   rdi = 0x0ec12000
rbp = 0x7f8767575850   rsp = 0x7f8767575658
 r8 = 0x0f7255a0r9 = 0x7f87675751ef
r10 = 0x17966000   r11 = 0x0020
r12 = 0x0400   r13 = 0x0011
r14 = 0x2d2a9000   r15 = 0x0f549210
rip = 0x7f87bd39d000
Found by: given as instruction pointer in context
 1  impalad + 0x1961f63
rbp = 0x7f8767575b20   rsp = 0x7f8767575860
rip = 0x01d61f63
Found by: previous frame's frame pointer
 2  impalad + 0x195e6cf
rbp = 0x7f87675760e0   rsp = 0x7f8767575b30
rip = 0x01d5e6cf
Found by: previous frame's frame pointer
 3  impalad + 0x195c89e
rbp = 0x7f87675761a0   rsp = 0x7f87675760f0
rip = 0x01d5c89e
Found by: previous frame's frame pointer
 4  impalad + 0x18e42c8
rbp = 0x7f8767576510   rsp = 0x7f87675761b0
rip = 0x01ce42c8
Found by: previous frame's frame pointer
 5  impalad + 0x18e3669
rbp = 0x7f87675768c0   rsp = 0x7f8767576520
rip = 0x01ce3669
Found by: previous frame's frame pointer
 6  impalad + 0x18e2adc
rbp = 0x7f87675768e0   rsp = 0x7f87675768d0
rip = 0x01ce2adc
Found by: previous frame's frame pointer
Crash reason:  SIGSEGV
Crash address: 0x7f87bd39d000
Process uptime: not available

Thread 223 (crashed)
 0  0x7f87bd39d000
rax = 0x7f87bd39d000   rdx = 0x0ec12000
rcx = 0x0f547540   rbx = 0x
rsi = 0x0f547540   rdi = 0x0ec12000
rbp = 0x7f8767575850   rsp = 0x7f8767575658
 r8 = 0x0f7255a0r9 = 0x7f87675751ef
r10 = 0x17966000   r11 = 0x0020
r12 = 0x0400   r13 = 0x0011
r14 = 0x2d2a9000   r15 = 0x0f549210
rip = 0x7f87bd39d000
Found by: given as instruction pointer in context
 1  impalad + 0x1961f63
rbp = 0x7f8767575b20   rsp = 0x7f8767575860
rip = 0x01d61f63
Found by: previous frame's frame pointer
 2  impalad + 0x195e6cf
rbp = 0x7f87675760e0   rsp = 0x7f8767575b30
rip = 0x01d5e6cf
Found by: previous frame's frame pointer
 3  impalad + 0x195c89e
rbp = 0x7f87675761a0   rsp = 0x7f87675760f0
rip = 0x01d5c89e
Found by: previous frame's frame pointer
 4  impalad + 0x18e42c8
rbp = 0x7f8767576510   rsp = 0x7f87675761b0
rip = 0x01ce42c8
Found by: previous frame's frame pointer
 5  impalad + 0x18e3669
rbp = 0x7f87675768c0   rsp = 0x7f8767576520
rip = 0x01ce3669
Found by: previous frame's frame pointer
 6  impalad + 0x18e2adc
rbp = 0x7f87675768e0   rsp = 0x7f87675768d0
rip = 0x01ce2adc
Found by: previous frame's frame pointer
Crash reason:  SIGSEGV
Crash address: 0x7f87bd39d000
Process uptime: not available

Thread 223 (crashed)
 0  0x7f87bd39d000
rax = 0x7f87bd39d000   rdx = 0x0ec12000
rcx = 0x0f547540   rbx = 0x
rsi = 0x0f547540   rdi = 0x0ec12000
rbp = 0x7f8767575850   rsp = 0x7f8767575658
 r8 = 0x0f7255a0r9 = 0x7f87675751ef
r10 = 0x17966000   r11 = 0x0020
r12 = 0x0400   r13 = 0x0011
r14 = 0x2d2a9000   r15 = 0x0f549210
rip = 0x7f87bd39d000
Found by: given as instruction pointer in context
 1  impalad + 0x1961f63
rbp = 0x7f8767575b20   rsp = 0x7f8767575860
rip = 0x01d61f63
Found by: previous frame's frame pointer
 2  impalad + 0x195e6cf
rbp = 0x7f87675760e0   rsp = 0x7f8767575b30
rip = 0x01d5e6cf
Found by: previous frame's frame pointer
 3  impalad + 0x195c89e
rbp = 0x7f87675761a0   rsp = 0x7f87675760f0
rip = 0x01d5c89e
Found by: previous frame's frame pointer
 4  impalad + 0x18e42c8
rbp = 0x7f8767576510   rsp = 0x7f87675761b0
rip = 0x01ce42c8
Found by: previous frame's frame pointer
 5  impalad + 0x18e3669
rbp = 0x7f87675768c0   rsp = 0x7f8767576520
rip =