[
https://issues.apache.org/jira/browse/DRILL-5050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zelaine Fong reassigned DRILL-5050:
-----------------------------------
Assignee: Chunhui Shi (was: Parth Chandra)
Assigning to [~cshi] for review.
> C++ client library has symbol resolution issues when loaded by a process that
> already uses boost::asio
> ------------------------------------------------------------------------------------------------------
>
> Key: DRILL-5050
> URL: https://issues.apache.org/jira/browse/DRILL-5050
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - C++
> Affects Versions: 1.6.0
> Environment: MacOs
> Reporter: Parth Chandra
> Assignee: Chunhui Shi
> Fix For: 2.0.0
>
>
> h4. Summary
> On MacOS, the Drill ODBC driver hangs when loaded by any process that might
> also be using {{boost::asio}}. This is observed in trying to connect to Drill
> via the ODBC driver using Tableau.
> h4. Analysis
> The problem is seen in the Drill client library on MacOS. In the method
> {code}
> DrillClientImpl::recvHandshake
> .
> .
> m_io_service.reset();
> if (DrillClientConfig::getHandshakeTimeout() > 0){
>
> m_deadlineTimer.expires_from_now(boost::posix_time::seconds(DrillClientConfig::getHandshakeTimeout()));
> m_deadlineTimer.async_wait(boost::bind(
> &DrillClientImpl::handleHShakeReadTimeout,
> this,
> boost::asio::placeholders::error
> ));
> DRILL_MT_LOG(DRILL_LOG(LOG_TRACE) << "Started new handshake wait
> timer with "
> << DrillClientConfig::getHandshakeTimeout() << " seconds." <<
> std::endl;)
> }
> async_read(
> this->m_socket,
> boost::asio::buffer(m_rbuf, LEN_PREFIX_BUFLEN),
> boost::bind(
> &DrillClientImpl::handleHandshake,
> this,
> m_rbuf,
> boost::asio::placeholders::error,
> boost::asio::placeholders::bytes_transferred)
> );
> DRILL_MT_LOG(DRILL_LOG(LOG_DEBUG) << "DrillClientImpl::recvHandshake:
> async read waiting for server handshake response.\n";)
> m_io_service.run();
> .
> .
> {code}
> The call to {{io_service::run}} returns without invoking any of the handlers
> that have been registered. The {{io_service}} object has two tasks in its
> queue, the timer task, and the socket read task. However, in the run method,
> the state of the {{io_service}} object appears to change and the number of
> outstanding tasks becomes zero. The run method therefore returns immediately.
> Subsequently, any query request sent to the server hangs as data is never
> pulled off the socket.
> This is bizarre behaviour and typically points to build problems.
> More investigation revealed a more interesting thing. {{boost::asio}} is a
> header only library. In other words, there is no actual library
> {{libboost_asio}}. All the code is included into the binary that includes the
> headers of {{boost::asio}}. It so happens that the Tableau process has a
> library (libtabquery) that uses {{boost::asio}} so the code for
> {{boost::asio}} is already loaded into process memory. When the drill client
> library (via the ODBC driver) is loaded by the loader, the drill client
> library loads its own copy of the {{boost:asio}} code. At runtime, the drill
> client code jumps to an address that resolves to an address inside the
> libtabquery copy of {{boost::asio}}. And that code returns incorrectly.
> Really? How is that even allowed? Two copies of {{boost::asio}} in the same
> process? Even if that is allowed, since the code is included at compile time,
> calls to the {{boost::asio}} library should be resolved using internal
> linkage. And if the call to {{boost::asio}} is not resolved statically, the
> dynamic loader would encounter two symbols with the same name and would give
> us an error. And even if the linker picks one of the symbols, as long as the
> code is the same (for example if both libraries use the same version of
> boost) can that cause a problem? Even more importantly, how do we fix that?
> h4. Some assembly required
> The disassembled libdrillClient shows this code inside recvHandshake
> {code}
> 000000000003dd8f movq -0xb0(%rbp), %rdi
> 000000000003dd96 addq $0xc0, %rdi
> 000000000003dd9d callq 0x1bff42 ## symbol stub for:
> __ZN5boost4asio10io_service3runEv
> 000000000003dda2 movq -0xb0(%rbp), %rdi
> 000000000003dda9 cmpq $0x0, 0x190(%rdi)
> 000000000003ddb4 movq %rax, -0x158(%rbp)
> {code}
> and later in the code
> {code}
> 0000000000057216 retq
> 0000000000057217 nopw (%rax,%rax)
> __ZN5boost4asio10io_service3runEv: ## definition of
> io_service::run
> 0000000000057220 pushq %rbp
> 0000000000057221 movq %rsp, %rbp
> 0000000000057224 subq $0x30, %rsp
> 0000000000057228 leaq -0x18(%rbp), %rax
> 000000000005722c movq %rdi, -0x8(%rbp)
> 0000000000057230 movq -0x8(%rbp), %rdi
> 0000000000057234 movq %rdi, -0x28(%rbp)
> {code}
> Note that in recvHandshake the call instruction jumps to an address that is
> an offset (0x1bff42). This offset happens to be beyond the end of the
> library. It certainly isn't the offset at which the io_service::run method is
> defined (0x57220).
> The linker is definitely not resolving the address statically, but we had
> already guessed that. It is, in fact, jumping to a stub method and at
> runtime this address is being resolved to the address of the
> {{io_service::run}} method in libtabquery.
> Just to check, in the debugger, we can see the following two implementations
> of {{io_service::run}} in the process
> {code}
> libtabquery.dylib`boost::asio::io_service::run():
> 0x10d597a10: pushq %rbp
> 0x10d597a11: movq %rsp, %rbp
> 0x10d597a14: pushq %rbx
> 0x10d597a15: subq $0x18, %rsp
> 0x10d597a19: movq %rdi, %rbx
> 0x10d597a1c: movl $0x0, -0x18(%rbp)
> 0x10d597a23: callq 0x10d5b73a4 ; symbol stub for:
> boost::system::system_category()
> 0x10d597a28: movq %rax, -0x10(%rbp)
> 0x10d597a2c: movq 0x8(%rbx), %rdi
> 0x10d597a30: leaq -0x18(%rbp), %rsi
> 0x10d597a34: callq 0x10d5b71e2 ; symbol stub for:
> boost::asio::detail::task_io_service::run(boost::system::error_code&)
> 0x10d597a39: cmpl $0x0, -0x18(%rbp)
> 0x10d597a3d: jne 0x10d597a46 ;
> boost::asio::io_service::run() + 54
> 0x10d597a3f: addq $0x18, %rsp
> 0x10d597a43: popq %rbx
> 0x10d597a44: popq %rbp
> 0x10d597a45: retq
> 0x10d597a46: leaq -0x18(%rbp), %rdi
> 0x10d597a4a: callq 0x10d5b71a6 ; symbol stub for:
> boost::asio::detail::do_throw_error(boost::system::error_code const&)
> 0x10d597a4f: nop
> libdrillClient.dylib`boost::asio::io_service::run() at io_service.ipp:57:
> 0x11f158300: pushq %rbp
> 0x11f158301: movq %rsp, %rbp
> 0x11f158304: subq $0x30, %rsp
> 0x11f158308: leaq -0x18(%rbp), %rax
> 0x11f15830c: movq %rdi, -0x8(%rbp)
> 0x11f158310: movq -0x8(%rbp), %rdi
> 0x11f158314: movq %rdi, -0x28(%rbp)
> 0x11f158318: movq %rax, %rdi
> 0x11f15831b: callq 0x11f2c210c ; symbol stub for:
> boost::system::error_code::error_code()
> 0x11f158320: leaq -0x18(%rbp), %rsi
> 0x11f158324: movq -0x28(%rbp), %rax
> 0x11f158328: movq 0x8(%rax), %rdi
> 0x11f15832c: callq 0x11f2c3516 ; symbol stub for:
> boost::asio::detail::task_io_service::run(boost::system::error_code&)
> 0x11f158331: leaq -0x18(%rbp), %rdi
> 0x11f158335: movq %rax, -0x20(%rbp)
> 0x11f158339: callq 0x11f2c1bf6 ; symbol stub for:
> boost::asio::detail::throw_error(boost::system::error_code const&)
> 0x11f15833e: movq -0x20(%rbp), %rax
> 0x11f158342: addq $0x30, %rsp
> 0x11f158346: popq %rbp
> 0x11f158347: retq
> {code}
> As suspected, the code for the two versions of {{io_service::run}} is
> different, so if the code is executing the wrong version, then the behaviour
> will be, expectedly, unexpected.
> h4. What does not work
> Linking statically with boost has no effect. The code is inlined in the first
> place and is effectively part of the dynamic library already.
> Changing the load order of the libraries (by specifying
> LD_LIBRARY_PATH/DYLD_LIBRARY_PATH does not help). This is because the
> application library is already loaded into the process.
> The linker -prebind flag does not help. The prebind flag is intended to tell
> the linker to resolve all addresses at link time. Why this did not work is
> not clear.
>
> Both libtabquery.dylib and libdrillClient.dylib contain symbols (functions)
> from the {{boost::asio package}}. At runtime, the MacOs loader assigns the
> drillClient library to call the functions defined in libtabquery. This causes
> the code to behave unpredictably and eventually the ODBC driver 'hangs'
> waiting for data from the server.
>
> Because the symbol linkage is being determined at runtime, changing the
> linker settings in the Drill client build has no effect. This is true even if
> you build with static linkage (a remarkable feature of MacOS!). Also, the
> boost builds between libtabquery and libdrillClient are different even if we
> use the same boost version; the compiled code is different. This is a
> critical part of the problem because if the compiled code were the same there
> would be no problem if the code was called using the libtabquery version
> instead of the libdrillClient version.
>
> h4. Solution
> The only way to resolve this is to use a 'shaded' version of boost in the
> drill client library. Luckily for us C++ namespaces, boost's bcp tool, and
> CMake together provide a way to rename the boost namespace to any name we
> like and use it in the drill client code. This effectively renames every
> symbol from boost to a different name using a new namespace name and the
> symbol name conflict does not arise.
> Using this build of boost, and using static linking (just to make sure) in
> the Drill client library, one is able to connect to and run queries against
> Drill from Tableau.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
