[
https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16981945#comment-16981945
]
Eric Badger commented on YARN-9561:
-----------------------------------
[~eyang], [[email protected]], [~ccondit], do you guys know how to add
libhadoop as a dependency for the container-executor? I'm not sure if this
needs to be done via CMakeLists.txt or a pom.xml. Right now, the compilation
code assumes that libhadoop.a exists. So obviously the problem is that the
hadoop-yarn-server-nodemanager subproject doesn't have the dependency correctly
formed to compile libhadoop if it doesn't already exist.
{noformat:title=hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml}
38 <dependency>
39 <groupId>org.apache.hadoop</groupId>
40 <artifactId>hadoop-common</artifactId>
41 <scope>provided</scope>
42 </dependency>
{noformat}
There is a dependency for hadoop-common, but it is {{provided}}. I tried
changing it to {{compile}}, but that doesn't work.
{noformat:title=hadoop-common-project/hadoop-common/src/CMakeLists.txt}
set(CMAKE_BUILD_WITH_INSTALL_RPATH TRUE)
hadoop_add_dual_library(hadoop
main/native/src/exception.c
${SRC}/io/compress/lz4/Lz4Compressor.c
${SRC}/io/compress/lz4/Lz4Decompressor.c
${SRC}/io/compress/lz4/lz4.c
${SRC}/io/compress/lz4/lz4hc.c
${ISAL_SOURCE_FILES}
${SNAPPY_SOURCE_FILES}
${ZSTD_SOURCE_FILES}
${OPENSSL_SOURCE_FILES}
${SRC}/io/compress/zlib/ZlibCompressor.c
${SRC}/io/compress/zlib/ZlibDecompressor.c
${BZIP2_SOURCE_FILES}
${SRC}/io/nativeio/NativeIO.c
${PMDK_SOURCE_FILES}
${SRC}/io/nativeio/errno_enum.c
${SRC}/io/nativeio/file_descriptor.c
${SRC}/io/nativeio/SharedFileDescriptorFactory.c
${SRC}/net/unix/DomainSocket.c
${SRC}/net/unix/DomainSocketWatcher.c
${SRC}/security/JniBasedUnixGroupsMapping.c
${SRC}/security/JniBasedUnixGroupsNetgroupMapping.c
${SRC}/security/hadoop_group_info.c
${SRC}/security/hadoop_user_info.c
${SRC}/util/NativeCodeLoader.c
${SRC}/util/NativeCrc32.c
${SRC}/util/bulk_crc32.c
${BULK_CRC_ARCH_SOURCE_FIlE}
)
if(NEED_LINK_DL)
set(LIB_DL dl)
endif()
hadoop_target_link_dual_libraries(hadoop ${LIB_DL} ${JAVA_JVM_LIBRARY})
set(LIBHADOOP_VERSION "1.0.0")
set_target_properties(hadoop PROPERTIES SOVERSION ${LIBHADOOP_VERSION})
hadoop_dual_output_directory(hadoop target/usr/local/lib)
# By embedding '$ORIGIN' into the RPATH of libhadoop.so, dlopen will look in
# the directory containing libhadoop.so. However, $ORIGIN is not supported by
# all operating systems.
if(${CMAKE_SYSTEM_NAME} MATCHES "Linux|SunOS")
set(RPATH "\$ORIGIN/")
if(EXTRA_LIBHADOOP_RPATH)
set(RPATH "${RPATH}:${EXTRA_LIBHADOOP_RPATH}/")
endif()
set_target_properties(hadoop PROPERTIES INSTALL_RPATH "${RPATH}")
endif()
{noformat}
libhadoop seems to be getting built by these lines in the hadoop-common
CMakeLists.txt. I tried simply adding {{hadoop}} instead of the full
{{libhadoop.a}} path, but that also (unsurprisingly) didn't work.
I imagine there's some special incantation that will allow me to add libhadoop
as a dependency, but I'm not sure how to do it or at what level (maven or
CMake) to do it. Any help would be appreciated.
> Add C changes for the new RuncContainerRuntime
> ----------------------------------------------
>
> Key: YARN-9561
> URL: https://issues.apache.org/jira/browse/YARN-9561
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Eric Badger
> Assignee: Eric Badger
> Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9561.001.patch, YARN-9561.002.patch,
> YARN-9561.003.patch, YARN-9561.004.patch, YARN-9561.005.patch,
> YARN-9561.006.patch, YARN-9561.007.patch, YARN-9561.008.patch,
> YARN-9561.009.patch, YARN-9561.010.patch, YARN-9561.011.patch,
> YARN-9561.012.patch, YARN-9561.013.patch, YARN-9561.014.patch
>
>
> This JIRA will be used to add the C changes to the container-executor native
> binary that are necessary for the new RuncContainerRuntime. There should be
> no changes to existing code paths.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
