[
https://issues.apache.org/jira/browse/ORC-1288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17618685#comment-17618685
]
noirello edited comment on ORC-1288 at 10/17/22 6:57 AM:
---------------------------------------------------------
Decreasing the stripe size and the compression size from 4096 to 1024 gives
almost the same backtrace (but with an invalid pointer error):
{noformat}
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x00007ffff7a80859 in __GI_abort () at abort.c:79
#2 0x00007ffff7aeb26e in __libc_message (action=action@entry=do_abort,
fmt=fmt@entry=0x7ffff7c15298 "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#3 0x00007ffff7af32fc in malloc_printerr (str=str@entry=0x7ffff7c171e0
"munmap_chunk(): invalid pointer") at malloc.c:5347
#4 0x00007ffff7af354c in munmap_chunk (p=<optimized out>) at malloc.c:2830
#5 0x000055555557cf54 in orc::MemoryPoolImpl::free (this=0x5555559f6018
<orc::getDefaultPool()::internal>, p=0x7ffff7756010
"4\004\350\026\002\f\203\n") at
/home/noirello/workspace/orc/c++/src/MemoryPool.cc:47
#6 0x000055555557d005 in orc::DataBuffer<char>::~DataBuffer
(this=0x555555a16250, __in_chrg=<optimized out>) at
/home/noirello/workspace/orc/c++/src/MemoryPool.cc:121
#7 0x000055555557d024 in orc::DataBuffer<char>::~DataBuffer
(this=0x555555a16250, __in_chrg=<optimized out>) at
/home/noirello/workspace/orc/c++/src/MemoryPool.cc:123
#8 0x000055555559680e in std::default_delete<orc::DataBuffer<char>
>::operator() (this=0x555555a0c580, __ptr=0x555555a16250) at
/usr/include/c++/9/bits/unique_ptr.h:81{noformat}
was (Author: noirello):
Decreasing the stripe size and the compression size from 4096 to 1024 gives
almost the same backtrace (bit with an invalid pointer error):
{noformat}
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x00007ffff7a80859 in __GI_abort () at abort.c:79
#2 0x00007ffff7aeb26e in __libc_message (action=action@entry=do_abort,
fmt=fmt@entry=0x7ffff7c15298 "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#3 0x00007ffff7af32fc in malloc_printerr (str=str@entry=0x7ffff7c171e0
"munmap_chunk(): invalid pointer") at malloc.c:5347
#4 0x00007ffff7af354c in munmap_chunk (p=<optimized out>) at malloc.c:2830
#5 0x000055555557cf54 in orc::MemoryPoolImpl::free (this=0x5555559f6018
<orc::getDefaultPool()::internal>, p=0x7ffff7756010
"4\004\350\026\002\f\203\n") at
/home/noirello/workspace/orc/c++/src/MemoryPool.cc:47
#6 0x000055555557d005 in orc::DataBuffer<char>::~DataBuffer
(this=0x555555a16250, __in_chrg=<optimized out>) at
/home/noirello/workspace/orc/c++/src/MemoryPool.cc:121
#7 0x000055555557d024 in orc::DataBuffer<char>::~DataBuffer
(this=0x555555a16250, __in_chrg=<optimized out>) at
/home/noirello/workspace/orc/c++/src/MemoryPool.cc:123
#8 0x000055555559680e in std::default_delete<orc::DataBuffer<char>
>::operator() (this=0x555555a0c580, __ptr=0x555555a16250) at
/usr/include/c++/9/bits/unique_ptr.h:81{noformat}
> [C++] Invalid memory freeing with ZLIB compression
> --------------------------------------------------
>
> Key: ORC-1288
> URL: https://issues.apache.org/jira/browse/ORC-1288
> Project: ORC
> Issue Type: Bug
> Affects Versions: 1.8.0
> Reporter: noirello
> Priority: Major
>
> The simple example code ends with a segfault/munmap_chunk(): invalid pointer:
> {code:cpp}
> #include "orc/Common.hh"
> #include "orc/OrcFile.hh"
> using namespace orc;
> int main(void) {
> ORC_UNIQUE_PTR<OutputStream> outStream = writeLocalFile("test_file.orc");
> ORC_UNIQUE_PTR<Type> schema(Type::buildTypeFromString("struct<c0:int>"));
> WriterOptions options;
> options.setCompression(orc::CompressionKind_ZLIB);
> options.setStripeSize(4096);
> options.setCompressionBlockSize(4096);
> ORC_UNIQUE_PTR<Writer> writer = createWriter(*schema, outStream.get(),
> options);
> uint64_t batchSize = 65535, rowCount = 10000000;
> ORC_UNIQUE_PTR<ColumnVectorBatch> batch =
> writer->createRowBatch(batchSize);
> StructVectorBatch *root = dynamic_cast<StructVectorBatch *>(batch.get());
> LongVectorBatch *c0 = dynamic_cast<LongVectorBatch *>(root->fields[0]);
> uint64_t rows = 0;
>
> for (uint64_t i = 0; i < rowCount; ++i) {
> c0->data[rows] = i;
> rows++;
> if (rows == batchSize) {
> root->numElements = rows;
> c0->numElements = rows;
> writer->add(*batch);
> rows = 0;
> }
> }
> if (rows != 0) {
> root->numElements = rows;
> c0->numElements = rows;
> writer->add(*batch);
> rows = 0;
> }
> writer->close();
> return 0;
> }
> {code}
> The bug depends on the stripe size, compression size, and the record number
> written to the file as well. I wasn't able to reproduce the error with other
> compression strategies than ZLIB.
> It looks like to me that it's related to
> [ORC-1130|https://issues.apache.org/jira/projects/ORC/issues/ORC-1130]
> somehow, but I couldn't comprehend how. (Reverting that modification on the
> main branch solved the issue).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
