kosak commented on PR #13775:
URL: https://github.com/apache/arrow/pull/13775#issuecomment-1207130256
Hi, I have reproduced the problem and I am able to offer a clearer
explanation. On my computer I have an installation of Arrow (with its own
vendored flatbuffers) and my program is also separately vendoring its own newer
copy of flatbuffers. To be clear, this is the version where I'm not doing any
namespace renaming so I am inviting the very name conflicts that triggered this
PR in the first place. In my program I call `FlightClient::DoPut` and this is
the sequence of calls
```
(gdb) where
#0 tpnnboni () at
/home/kosak/git/deephaven-core/cpp-examples/read_table_with_arrow_flight/main.cc:26
#1 0x00005555556f9f89 in flatbuffers::FlatBufferBuilder::FlatBufferBuilder
(this=0x7fffffffcf80, initial_size=1024, allocator=0x0, own_allocator=false,
buffer_minalign=8) at
/home/kosak/git/deephaven-core/cpp-client/deephaven/client/third_party/flatbuffers/flatbuffer_builder.h:101
#2 0x0000555555f0744a in arrow::ipc::internal::WriteSchemaMessage
(schema=..., mapper=..., options=..., out=0x7fffffffd098 = {...}) at
/home/kosak/dhcpp/src/arrow/cpp/src/arrow/ipc/metadata_internal.cc:1195
#3 0x0000555555f75ecf in arrow::ipc::GetSchemaPayload (schema=...,
options=..., mapper=..., out=0x7fffffffd090) at
/home/kosak/dhcpp/src/arrow/cpp/src/arrow/ipc/writer.cc:614
#4 0x0000555555f832ae in arrow::ipc::internal::IpcFormatWriter::Start
(this=0x55555788b670) at
/home/kosak/dhcpp/src/arrow/cpp/src/arrow/ipc/writer.cc:1050
#5 0x0000555555f78eda in arrow::ipc::internal::OpenRecordBatchWriter
(sink=std::unique_ptr<arrow::ipc::internal::IpcPayloadWriter> = {...},
schema=std::shared_ptr<arrow::Schema> (use count 2, weak count 0) = {...},
options=...) at /home/kosak/dhcpp/src/arrow/cpp/src/arrow/ipc/writer.cc:1384
#6 0x000055555596452a in arrow::flight::ClientStreamWriter::Begin
(this=0x5555578c18f0, schema=std::shared_ptr<arrow::Schema> (use count 2, weak
count 0) = {...}, options=...) at
/home/kosak/dhcpp/src/arrow/cpp/src/arrow/flight/client.cc:396
#7 0x000055555595ffda in arrow::flight::FlightClient::DoPut
(this=0x555557893ba0, options=..., descriptor=...,
schema=std::shared_ptr<arrow::Schema> (use count 2, weak count 0) = {...}) at
/home/kosak/dhcpp/src/arrow/cpp/src/arrow/flight/client.cc:663
#8 0x0000555555960189 in arrow::flight::FlightClient::DoPut
(this=0x555557893ba0, options=..., descriptor=...,
schema=std::shared_ptr<arrow::Schema> (use count 2, weak count 0) = {...},
writer=0x7fffffffd3e8 = {...}, reader=0x7fffffffd3f0 = {...}) at
/home/kosak/dhcpp/src/arrow/cpp/src/arrow/flight/client.cc:672
#9 0x0000555555668367 in deephaven::client::utility::TableMaker::makeTable
(this=0x7fffffffd610, manager=...) at
/home/kosak/git/deephaven-core/cpp-client/deephaven/client/src/utility/table_maker.cc:47
#10 0x000055555563fb31 in (anonymous namespace)::makeTable (manager=...) at
/home/kosak/git/deephaven-core/cpp-examples/read_table_with_arrow_flight/main.cc:54
#11 0x000055555563f684 in main () at
/home/kosak/git/deephaven-core/cpp-examples/read_table_with_arrow_flight/main.cc:40
#12 0x00007ffff76c3083 in __libc_start_main (main=0x55555563f5e7 <main()>,
argc=1, argv=0x7fffffffd888, init=<optimized out>, fini=<optimized out>,
rtld_fini=<optimized out>, stack_end=0x7fffffffd878) at ../csu/libc-start.c:308
#13 0x000055555563f3be in _start ()
```
The terrible thing happens right at the end (that is, frame number 1),
where, deep in the stack trace, the Arrow library calls
`FlatBufferBuilder::FlatBufferBuilder` out of *MY* vendored version of
Flatbuffers, rather than out of Arrow's vendored version. (You can see this by
looking at the source file pathnames). This happened because
FlatBufferBuilder's constructor is declared inline which means the compiler
will emit a weak symbol for it and may still elect to call it outline even
though you declared it inline. It looks like the compiler did indeed choose to
call it outline, and furthermore that the linker chose my version of its
constructor rather than Arrow's.
The problem is that `FlatBufferBuilder` has changed between Flatbuffer
versions. In particular, the first field of `FlatBufferBuilder` is
`vector_downward buf_` in both versions. However the definition of
`vector_downward` has itself changed:
In the version vendored by Arrow, `vector_downward` has these fields:
```
Allocator *allocator_;
bool own_allocator_;
size_t initial_size_;
size_t buffer_minalign_;
size_t reserved_;
uint8_t *buf_;
uint8_t *cur_; // Points at location between empty (below) and used
(above).
uint8_t *scratch_; // Points to the end of the scratchpad in use.
```
In my vendored version, `vector_downward` has these fields:
```
Allocator *allocator_;
bool own_allocator_;
size_t initial_size_;
size_t buffer_minalign_;
size_t reserved_;
uoffset_t size_; // ************ THIS IS NEW ***************
uint8_t *buf_;
uint8_t *cur_; // Points at location between empty (below) and used
(above).
uint8_t *scratch_; // Points to the end of the scratchpad in use.
```
This means that the layout of `FlatBufferBuilder` is different between the
two versions, an incoherent constructor is called, and the corruption just
spreads from there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
