[
https://issues.apache.org/jira/browse/KUDU-3334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18116138#comment-18116138
]
ASF subversion and git services commented on KUDU-3334:
-------------------------------------------------------
Commit d66c9079679a25a53f2f87523297124310299bc0 in kudu's branch
refs/heads/master from Alexey Serbin
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=d66c90796 ]
KUDU-3804 fix --exclude-libs for libkudu_client
With [1] which addressed KUDU-3334 came the --exclude-libs linker
directive for libprotobuf. It seems Kudu used the GNU gold linker
for the majority of supported platforms before relatively recent
update [2], and --exclude-libs worked as expected. BTW, the GNU gold
linker has been deprecated for some time already, and the binutils
of versions 2.44 and newer no longer contain the gold linker [3].
As mentioned in [4], GNU ld and gold allow --exclude-libs=b to hide b.a,
but ld.lld requires --exclude=libs=b.a. Since Kudu switched to
thirdparty ld.lld instead of GNU gold since [2], it makes sense to keep
the --exclude-libs linker's directive behaving as expected for all the
linkers that Kudu is able to use.
I compared the list of protobuf symbols exposed by the exported
libkudu_client.so library with and without this patch. The difference
is below:
--- /tmp/no-a-suffix.symbols 2026-09-15 15:24:38.125949226 -0700
+++ /tmp/a-suffix.symbols 2026-09-15 15:22:40.369171889 -0700
@@ -1,13 +1,5 @@
-W descriptor_table_google_2fprotobuf_2fany_2eproto_getter()
-W descriptor_table_google_2fprotobuf_2ftype_2eproto_getter()
-W descriptor_table_google_2fprotobuf_2fwrappers_2eproto_getter()
-W descriptor_table_google_2fprotobuf_2fdescriptor_2eproto_getter()
-W descriptor_table_google_2fprotobuf_2fsource_5fcontext_2eproto_getter()
W bool google_breakpad::MinidumpFileWriter::WriteStringCore<char>(char
const*, unsigned int, MDLocationDescriptor*)
W bool google_breakpad::MinidumpFileWriter::WriteStringCore<wchar_t>(wchar_t
const*, unsigned int, MDLocationDescriptor*)
-R TableStruct_google_2fprotobuf_2fany_2eproto::offsets
-R TableStruct_google_2fprotobuf_2ftype_2eproto::offsets
-R TableStruct_google_2fprotobuf_2fwrappers_2eproto::offsets
T kudu::KuduPartialRow::SetVarchar(kudu::Slice const&, kudu::Slice const&)
T kudu::KuduPartialRow::SetVarchar(int, kudu::Slice const&)
T kudu::KuduPartialRow::SetArrayBool(kudu::Slice const&, std::vector<bool,
std::allocator<bool> > const&, std::vector<bool, std::allocator<bool> > const&)
@@ -633,8 +625,6 @@
T kudu::operator!=(kudu::MonoDelta const&, kudu::MonoDelta const&)
T kudu::operator+(kudu::MonoTime const&, kudu::MonoDelta const&)
T kudu::operator+(kudu::MonoDelta const&, kudu::MonoDelta const&)
-R TableStruct_google_2fprotobuf_2fdescriptor_2eproto::offsets
-R TableStruct_google_2fprotobuf_2fsource_5fcontext_2eproto::offsets
W boost::exception::~exception()
W boost::exception::~exception()
W boost::exception::~exception()
@@ -1165,9 +1150,4 @@
V vtable for std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>
V vtable for std::bad_optional_access
V vtable for std::_Sp_counted_ptr_inplace<kudu::KuduPartialRow,
std::allocator<void>, (__gnu_cxx::_Lock_policy)2>
-D descriptor_table_google_2fprotobuf_2fany_2eproto
-D descriptor_table_google_2fprotobuf_2fdescriptor_2eproto
-D descriptor_table_google_2fprotobuf_2fsource_5fcontext_2eproto
-D descriptor_table_google_2fprotobuf_2ftype_2eproto
-D descriptor_table_google_2fprotobuf_2fwrappers_2eproto
D z_errmsg
[1] https://github.com/apache/kudu/commit/5d39a74a1
[2] https://github.com/apache/kudu/commit/1d789a87a
[3] https://www.phoronix.com/news/GNU-Gold-Linker-Deprecated
[4] https://maskray.me/blog/lld-and-gnu-linker-incompatibilities
Change-Id: I8950dc0d2c0f4b56782f201f417608acd2420025
Reviewed-on: http://gerrit.cloudera.org:8080/24859
Reviewed-by: Michael Smith <[email protected]>
Tested-by: Alexey Serbin <[email protected]>
Reviewed-by: Abhishek Chennaka <[email protected]>
> Clarify whether it's possible to hide libprotobuf symbols exported from
> libkudu_client
> --------------------------------------------------------------------------------------
>
> Key: KUDU-3334
> URL: https://issues.apache.org/jira/browse/KUDU-3334
> Project: Kudu
> Issue Type: Task
> Reporter: Alexey Serbin
> Assignee: Alexey Serbin
> Priority: Major
> Fix For: 1.16.0
>
>
> As it turned out, there are conflicts during static initialization of symbols
> related to protobuf extensions in Impala which is dynamically linked against
> {{libkudu_client}} while statically linking in {{libprotobuf}} on itself. It
> seems the issue might be caused by the static initialization of the
> descriptor pools for protobuf types, so eventually impala complains about
> that in run-time:
> {noformat}
> [libprotobuf ERROR google/protobuf/descriptor.cc:3620] Invalid proto
> descriptor for file "kudu/rpc/rpc_header.proto":
> [libprotobuf ERROR google/protobuf/descriptor.cc:3623]
> kudu.rpc.track_rpc_result: ".google.protobuf.MethodOptions" is not defined.
> [libprotobuf ERROR google/protobuf/descriptor.cc:3623]
> kudu.rpc.authz_method: ".google.protobuf.MethodOptions" is not defined.
> [libprotobuf ERROR google/protobuf/descriptor.cc:3623]
> kudu.rpc.default_authz_method: ".google.protobuf.ServiceOptions" is not
> defined.
> {noformat}
> From examining the libkudu_client, it turns out some related protobuf symbols
> are discoverable:
> {noformat}
> $ nm -g libkudu_client.so | grep protobuf | grep desc
> 00000000006469a0 R
> _ZN50TableStruct_google_2fprotobuf_2fdescriptor_2eproto7offseftsE
> 0000000000751a20 D descriptor_table_google_2fprotobuf_2fany_2eproto
> 0000000000751580 D descriptor_table_google_2fprotobuf_2fdescriptor_2eproto
> 0000000000751ac0 D
> descriptor_table_google_2fprotobuf_2fsource_5fcontext_2eproto0000000000751b60
> D descriptor_table_google_2fprotobuf_2ftype_2eproto
> 0000000000751cc0 D descriptor_table_google_2fprotobuf_2fwrappers_2eproto
> 0000000000751900 D
> scc_info_DescriptorProto_ExtensionRange_google_2fprotobuf_2fdescriptor_2eproto
> 00000000007519f0 D
> scc_info_DescriptorProto_ReservedRange_google_2fprotobuf_2fdescriptor_2eproto
> 0000000000751920 D
> scc_info_DescriptorProto_google_2fprotobuf_2fdescriptor_2eproto
> 00000000007519d0 D
> scc_info_EnumDescriptorProto_EnumReservedRange_google_2fprotobuf_2fdescriptor_2eproto
> 00000000007518c0 D
> scc_info_EnumDescriptorProto_google_2fprotobuf_2fdescriptor_2eproto
> 00000000007518a0 D scc_info_EnumOptions_google_2fprotobuf_2fdescriptor_2eproto
> 0000000000751880 D
> scc_info_EnumValueDescriptorProto_google_2fprotobuf_2fdescriptor_2eproto
> 0000000000751860 D
> scc_info_EnumValueOptions_google_2fprotobuf_2fdescriptor_2eproto
> 0000000000751840 D
> scc_info_ExtensionRangeOptions_google_2fprotobuf_2fdescriptor_2eproto
> 0000000000751820 D
> scc_info_FieldDescriptorProto_google_2fprotobuf_2fdescriptor_2eproto
> 0000000000751800 D
> scc_info_FieldOptions_google_2fprotobuf_2fdescriptor_2eproto
> 00000000007517a0 D
> scc_info_FileDescriptorProto_google_2fprotobuf_2fdescriptor_2eproto
> 0000000000751780 D
> scc_info_FileDescriptorSet_google_2fprotobuf_2fdescriptor_2eproto
> 0000000000751760 D scc_info_FileOptions_google_2fprotobuf_2fdescriptor_2eproto
> 00000000007519b0 D
> scc_info_GeneratedCodeInfo_Annotation_google_2fprotobuf_2fdescriptor_2eproto
> 0000000000751740 D
> scc_info_GeneratedCodeInfo_google_2fprotobuf_2fdescriptor_2eproto
> 0000000000751720 D
> scc_info_MessageOptions_google_2fprotobuf_2fdescriptor_2eproto
> 0000000000751700 D
> scc_info_MethodDescriptorProto_google_2fprotobuf_2fdescriptor_2eproto
> 00000000007516e0 D
> scc_info_MethodOptions_google_2fprotobuf_2fdescriptor_2eproto
> 00000000007516c0 D
> scc_info_OneofDescriptorProto_google_2fprotobuf_2fdescriptor_2eproto
> 00000000007516a0 D
> scc_info_OneofOptions_google_2fprotobuf_2fdescriptor_2eproto
> 0000000000751660 D
> scc_info_ServiceDescriptorProto_google_2fprotobuf_2fdescriptor_2eproto
> 0000000000751640 D
> scc_info_ServiceOptions_google_2fprotobuf_2fdescriptor_2eproto
> 0000000000751990 D
> scc_info_SourceCodeInfo_Location_google_2fprotobuf_2fdescriptor_2eproto
> 0000000000751620 D
> scc_info_SourceCodeInfo_google_2fprotobuf_2fdescriptor_2eproto
> 0000000000751970 D
> scc_info_UninterpretedOption_NamePart_google_2fprotobuf_2fdescriptor_2eproto
> 0000000000751600 D
> scc_info_UninterpretedOption_google_2fprotobuf_2fdescriptor_2eproto
> {noformat}
> The libprotobuf which is statically linked in by impalad doesn't contain any
> types requiring protobuf extension types, so
> {{google.protobuf.MethodOptions}} and other related types are not registered
> in the descriptor pool that impalad uses, but libkudu_client needs those.
> The theory is that libkudu_client, being dynamically linked into impalad,
> doesn't run static initializers for those extension-related types since it
> assumes that the descriptor pool created by statically linked in
> libkudu_client already ran those (though I'm not sure that's exactly what
> happens there). Probably, we could address that by hiding protobuf symbols
> exposed from impalad to avoid confusing the static initializers for
> libkudu_client. However, similar problem might happen with any other Kudu
> application, and it's easier to rather address that from libkudu_client side
> than requiring hiding protobuf symbols from all applications trying to link
> in libkudu_client. So, hopefully the issue might be addressed by hiding
> libprotobuf symbols in libkudu_client itself.
> I found this article which seems to be relevant:
> https://stackoverflow.com/questions/2222162/how-to-apply-fvisibility-option-to-symbols-in-static-libraries
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
