[ 
https://issues.apache.org/jira/browse/ARROW-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16056534#comment-16056534
 ] 

Young Park commented on ARROW-1130:
-----------------------------------

Hi Wes,

Yes, but the test passes when I set the env variables for testing hdfs (when 
built with GCC4.8) but when built with GCC5, the test passes as it just 
gracefully skips the io-hdfs-test tests. I am guessing that it may be the 
intended behavior for the tests, but I wanted to bring this up as when I traced 
the source of the error before I set the testing env variables, the error 
seemed to come from the jni_helper. And at the time, I suspected that my 
program was also running into issues with jni_helper, I was hoping that someone 
with more HDFS experience would have a better insight into what may be the root 
cause of the issue.

With respect to the project that I am working on, I am using the 
libraries(Arrow, Parquet) to connect to HDFS and read in Parquet files. 
My program successfully connects to HDFS, calls ReadBatch while iterating 
through the columns in the Row Group, constructs and returns an 
arrow::RecordBatch, but it segfaults soon afterwards.

I am suspecting an error in the use of shared_ptrs as the error log shows that 
it segfaults after calling destroy on arrow::PoolBuffer, but I wanted to rule 
out all possibilities along the way.

Attached is a error log from running my program.

{code:java}
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0xffffffff748e3af0, pid=4149, tid=0x00007f316c1d1740
#
# JRE version: OpenJDK Runtime Environment (8.0_131-b11) (build 1.8.0_131-b11)
# Java VM: OpenJDK 64-Bit Server VM (25.131-b11 mixed mode linux-amd64 
compressed oops)
# Problematic frame:
# C  0xffffffff748e3af0
#
# Failed to write core dump. Core dumps have been disabled. To enable core 
dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://www.azulsystems.com/support/
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

---------------  T H R E A D  ---------------

Current thread (0x0000000004931800):  JavaThread "main" [_thread_in_native, 
id=4149, stack(0x00007ffe747eb000,0x00007ffe748eb000)]

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 
0xffffffff748e3af0

Registers:
RAX=0xffffffff748e3af0, RBX=0x00000000066bb050, RCX=0x00007ffe748e3a90, 
RDX=0x0000000000000040
RSP=0x00007ffe748e3788, RBP=0x00007ffe748e37a0, RSI=0x0000000006681880, 
RDI=0x00007ffe748e3a90
R8 =0x00000000066815e0, R9 =0x00007f316c1d1700, R10=0x0000000000001041, 
R11=0x00007f3165c61f8a
R12=0x00007ffe748e42df, R13=0x00007ffe748e4250, R14=0x00000000042cebd8, 
R15=0x0000000000000000
RIP=0xffffffff748e3af0, EFLAGS=0x0000000000010202, CSGSFS=0x002b000000000033, 
ERR=0x0000000000000015
  TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007ffe748e3788)
0x00007ffe748e3788:   00007f3165c84cec 0000000000000000
0x00007ffe748e3798:   00000000066816f8 00007ffe748e37c0
0x00007ffe748e37a8:   00007f316636a9ed 00000000066816f8
0x00007ffe748e37b8:   00000000066816f0 00007ffe748e37e0
0x00007ffe748e37c8:   00007f316636981c 00000000066816f8
0x00007ffe748e37d8:   00000000066816f0 00007ffe748e3800
0x00007ffe748e37e8:   00007f31663688f9 00000000066816f8
0x00007ffe748e37f8:   00000000066816f0 00007ffe748e3820
0x00007ffe748e3808:   00007f31663678cb ffffffff00000000
0x00007ffe748e3818:   00000000066816e0 00007ffe748e3840
0x00007ffe748e3828:   00007f3165c66490 0000000000000000
0x00007ffe748e3838:   00000000066816e0 00007ffe748e3860
0x00007ffe748e3848:   00007f3165c62497 0000000000000000
0x00007ffe748e3858:   00000000066817e8 00007ffe748e3880
0x00007ffe748e3868:   00007f3165c613c8 00007f3165f84250
0x00007ffe748e3878:   00000000066817e0 00007ffe748e38a0
0x00007ffe748e3888:   00007f3165c613e2 00007f316c1d1700
0x00007ffe748e3898:   00000000066817e0 00007ffe748e38c0
0x00007ffe748e38a8:   00007f3165c61fb8 00000000066bb050
0x00007ffe748e38b8:   0000000006681798 00007ffe748e38e0
0x00007ffe748e38c8:   00007f3165c804ca 00007f316c1d1700
0x00007ffe748e38d8:   0000000006681798 00007ffe748e3900
0x00007ffe748e38e8:   00007f31660239e1 0000000006681798
0x00007ffe748e38f8:   0000000006681790 00007ffe748e3920
0x00007ffe748e3908:   00007f3166023314 0000000006681798
0x00007ffe748e3918:   0000000006681790 00007ffe748e3940
0x00007ffe748e3928:   00007f3166022c24 0000000006681798
0x00007ffe748e3938:   0000000006681790 00007ffe748e3960
0x00007ffe748e3948:   00007f31660214d1 ffffffff00000000
0x00007ffe748e3958:   0000000006681780 00007ffe748e3980
0x00007ffe748e3968:   00000000016d38d6 0000000000000000
0x00007ffe748e3978:   0000000006681780 00007ffe748e39a0 

Instructions: (pc=0xffffffff748e3af0)
0xffffffff748e3ad0:   
[error occurred during error reporting (printing registers, top of stack, 
instructions near pc), id 0xb]

Register to memory mapping:

RAX=0xffffffff748e3af0 is an unknown value
RBX=0x00000000066bb050 is an unknown value
RCX=0x00007ffe748e3a90 is pointing into the stack for thread: 0x0000000004931800
RDX=0x0000000000000040 is an unknown value
RSP=0x00007ffe748e3788 is pointing into the stack for thread: 0x0000000004931800
RBP=0x00007ffe748e37a0 is pointing into the stack for thread: 0x0000000004931800
RSI=0x0000000006681880 is an unknown value
RDI=0x00007ffe748e3a90 is pointing into the stack for thread: 0x0000000004931800
R8 =0x00000000066815e0 is an unknown value
R9 =0x00007f316c1d1700 is an unknown value
R10=0x0000000000001041 is an unknown value
R11=0x00007f3165c61f8a: _ZN5arrow14PrimitiveArrayD1Ev+0 in 
/parquet-cpp/arrow_ep/src/arrow_ep-install/lib/libarrow.so.0 at 
0x00007f3165b99000
R12=0x00007ffe748e42df is pointing into the stack for thread: 0x0000000004931800
R13=0x00007ffe748e4250 is pointing into the stack for thread: 0x0000000004931800
R14=0x00000000042cebd8 is an unknown value
R15=0x0000000000000000 is an unknown value


Stack: [0x00007ffe747eb000,0x00007ffe748eb000],  sp=0x00007ffe748e3788,  free 
space=993k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  0xffffffff748e3af0
C  [libparquet.so.1+0xf49ed]  void 
__gnu_cxx::new_allocator<arrow::PoolBuffer>::destroy<arrow::PoolBuffer>(arrow::PoolBuffer*)+0x23
C  [libparquet.so.1+0xf381c]  
std::enable_if<std::allocator_traits<std::allocator<arrow::PoolBuffer> 
>::__destroy_helper<arrow::PoolBuffer>::value, void>::type 
std::allocator_traits<std::allocator<arrow::PoolBuffer> 
>::_S_destroy<arrow::PoolBuffer>(std::allocator<arrow::PoolBuffer>&, 
arrow::PoolBuffer*)+0x23
C  [libparquet.so.1+0xf28f9]  void 
std::allocator_traits<std::allocator<arrow::PoolBuffer> 
>::destroy<arrow::PoolBuffer>(std::allocator<arrow::PoolBuffer>&, 
arrow::PoolBuffer*)+0x23
C  [libparquet.so.1+0xf18cb]  std::_Sp_counted_ptr_inplace<arrow::PoolBuffer, 
std::allocator<arrow::PoolBuffer>, 
(__gnu_cxx::_Lock_policy)2>::_M_dispose()+0x27
C  [libarrow.so.0+0xcd490]  
std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()+0x42
C  [libarrow.so.0+0xc9497]  
std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count()+0x27
C  [libarrow.so.0+0xc83c8]  std::__shared_ptr<arrow::Buffer, 
(__gnu_cxx::_Lock_policy)2>::~__shared_ptr()+0x1c
C  [libarrow.so.0+0xc83e2]  std::shared_ptr<arrow::Buffer>::~shared_ptr()+0x18
C  [libarrow.so.0+0xc8fb8]  arrow::PrimitiveArray::~PrimitiveArray()+0x2e
C  [libarrow.so.0+0xe74ca]  
arrow::NumericArray<arrow::FloatType>::~NumericArray()+0x2a
C  [libparquet_arrow.so.1+0x929e1]  void 
__gnu_cxx::new_allocator<arrow::NumericArray<arrow::FloatType> 
>::destroy<arrow::NumericArray<arrow::FloatType> 
>(arrow::NumericArray<arrow::FloatType>*)+0x23
C  [libparquet_arrow.so.1+0x92314]  
std::enable_if<std::allocator_traits<std::allocator<arrow::NumericArray<arrow::FloatType>
 > >::__destroy_helper<arrow::NumericArray<arrow::FloatType> >::value, 
void>::type 
std::allocator_traits<std::allocator<arrow::NumericArray<arrow::FloatType> > 
>::_S_destroy<arrow::NumericArray<arrow::FloatType> 
>(std::allocator<arrow::NumericArray<arrow::FloatType> >&, 
arrow::NumericArray<arrow::FloatType>*)+0x23
C  [libparquet_arrow.so.1+0x91c24]  void 
std::allocator_traits<std::allocator<arrow::NumericArray<arrow::FloatType> > 
>::destroy<arrow::NumericArray<arrow::FloatType> 
>(std::allocator<arrow::NumericArray<arrow::FloatType> >&, 
arrow::NumericArray<arrow::FloatType>*)+0x23
C  [libparquet_arrow.so.1+0x904d1]  
std::_Sp_counted_ptr_inplace<arrow::NumericArray<arrow::FloatType>, 
std::allocator<arrow::NumericArray<arrow::FloatType> >, 
(__gnu_cxx::_Lock_policy)2>::_M_dispose()+0x27
C  [cntk+0x12d38d6]  
std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()+0x42
C  [cntk+0x126e1c7]  
std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count()+0x27
C  [Cntk.Deserializers.DF-2.0rc2d.so+0x989a6]  std::__shared_ptr<arrow::Array, 
(__gnu_cxx::_Lock_policy)2>::~__shared_ptr()+0x1c
C  [Cntk.Deserializers.DF-2.0rc2d.so+0x989c0]  
std::shared_ptr<arrow::Array>::~shared_ptr()+0x18
C  [Cntk.Deserializers.DF-2.0rc2d.so+0x9e1bc]  void 
std::_Destroy<std::shared_ptr<arrow::Array> 
>(std::shared_ptr<arrow::Array>*)+0x18
C  [Cntk.Deserializers.DF-2.0rc2d.so+0x9d46c]  void 
std::_Destroy_aux<false>::__destroy<std::shared_ptr<arrow::Array>*>(std::shared_ptr<arrow::Array>*,
 std::shared_ptr<arrow::Array>*)+0x26
C  [Cntk.Deserializers.DF-2.0rc2d.so+0x9c5eb]  void 
std::_Destroy<std::shared_ptr<arrow::Array>*>(std::shared_ptr<arrow::Array>*, 
std::shared_ptr<arrow::Array>*)+0x23
C  [Cntk.Deserializers.DF-2.0rc2d.so+0x9afcd]  void 
std::_Destroy<std::shared_ptr<arrow::Array>*, std::shared_ptr<arrow::Array> 
>(std::shared_ptr<arrow::Array>*, std::shared_ptr<arrow::Array>*, 
std::allocator<std::shared_ptr<arrow::Array> >&)+0x27
C  [Cntk.Deserializers.DF-2.0rc2d.so+0x99a9b]  
std::vector<std::shared_ptr<arrow::Array>, 
std::allocator<std::shared_ptr<arrow::Array> > >::~vector()+0x35
C  [Cntk.Deserializers.DF-2.0rc2d.so+0x9cbce]  
arrow::RecordBatch::~RecordBatch()+0x1c
C  [Cntk.Deserializers.DF-2.0rc2d.so+0xa5568]  
std::_Sp_counted_ptr<arrow::RecordBatch*, 
(__gnu_cxx::_Lock_policy)2>::_M_dispose()+0x22
C  [cntk+0x12d38d6]  
std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()+0x42
C  [cntk+0x126e1c7]  
std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count()+0x27
C  [Cntk.Deserializers.DF-2.0rc2d.so+0x99142]  
std::__shared_ptr<arrow::RecordBatch, 
(__gnu_cxx::_Lock_policy)2>::~__shared_ptr()+0x1c
C  [Cntk.Deserializers.DF-2.0rc2d.so+0x9915c]  
std::shared_ptr<arrow::RecordBatch>::~shared_ptr()+0x18
{code}



> io-hdfs-test failure
> --------------------
>
>                 Key: ARROW-1130
>                 URL: https://issues.apache.org/jira/browse/ARROW-1130
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: C++
>         Environment: Ubuntu 16.04, GCC 4.8, Parquet-cpp
>            Reporter: Young Park
>            Priority: Blocker
>
> Hi,
> I have noticed that arrow-cpp's io-hdfs-test fails during compilation with 
> GCC 4.8, but passes when compiled with GCC 5.4 (as it just skips all tests as 
> it doesn't connect to the HDFS client).
> I went into the test output log and it seemed to want me to set the variable 
> ARROW_HDFS_TEST_USER, so I set the variable to 'root' and 
> ARROW_HDFS_TEST_PORT to '9000' (which is the port that I use to connect to my 
> local hdfs) and the test passes.
> Do I need to configure the environment and the variables in a specific way to 
> get it to work?
> I'm mainly asking as I am trying to use arrow and parquet c++ libraries in an 
> external project and I continue to run into segfaults in the libhdfs 
> jni_helper even though I successfully connect to HDFS on my local Hadoop 
> cluster and even read in a single Parquet file and am hoping that somehow 
> this will help me figure out the issue in my external project as well.
> Thank you in advance for your help.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Reply via email to