[
https://issues.apache.org/jira/browse/HDDS-17155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17155:
------------------------------
Description:
h3. Problem
{{NativeLibraryLoader}} loads {{libozone_rocksdb_tools.so}} in two steps:
# It tries {{System.loadLibrary}} from {{java.library.path}}. The
distribution's {{lib/native}} only contains {{libhadoop}}, so this misses.
# It extracts the library from {{hdds-rocks-native-*.jar}} into
{{native.lib.tmp.dir}}. If that property is not set, it uses {{new
File("").getAbsoluteFile()}}, the *current working directory*.
No Ozone script sets {{native.lib.tmp.dir}}. Whenever the process cannot write
to its working directory, extraction fails and the library is not loaded:
{code}
WARN utils.NativeLibraryLoader: Unable to load library: ozone_rocksdb_tools
java.nio.file.AccessDeniedException:
/opt/hadoop/ozone_rocksdb_tools7373108702252003876
WARN snapshot.SnapshotDiffManager: Native Library for raw sst file reading
loading failed. Fallback to performing a full diff instead. ...
{code}
This was hit in the compose runner image: the OM runs in {{/opt/hadoop}}, the
mounted distribution, and that directory was not writable by the container
user. The fallback is silent apart from the WARN, so snapshot diff quietly
loses the native path. {{ozone debug checknative}} then reports {{rocks-tools:
false}} for this reason, which would also mask the libstdc++ problem in
HDDS-17153.
For comparison:
* Hadoop's {{NativeCodeLoader}} only uses {{System.loadLibrary}} from
{{lib/native}} and never extracts.
* RocksJava's own {{NativeLibraryLoader}} extracts {{librocksdbjni}} into
{{java.io.tmpdir}} by default ({{File.createTempFile}}), or into
{{ROCKSDB_SHAREDLIB_DIR}}.
h3. Fix
Keep the working directory as the default extraction location, and fall back to
{{java.io.tmpdir}} ({{Files.createTempDirectory(libraryName)}}) only when the
extraction directory cannot be created under the working directory. Deployments
that load the library today are unaffected, including ones where {{/tmp}} is
mounted {{noexec}}. An explicit {{native.lib.tmp.dir}} behaves as before and
does not fall back. Cleanup is unchanged: the existing shutdown hook deletes
the directory.
h3. Testing done
* {{TestNativeLibraryLoader#testDummyLibrary}} covers the working directory
default (property {{null}} or empty), an explicit {{native.lib.tmp.dir}}, and
the fallback to {{java.io.tmpdir}} when the working directory cannot hold the
extraction directory (property {{null}} or empty). The two fallback cases fail
without the fix.
* Checkstyle is clean.
was:
h3. Problem
{{NativeLibraryLoader}} loads {{libozone_rocksdb_tools.so}} in two steps:
# It tries {{System.loadLibrary}} from {{java.library.path}}. The
distribution's {{lib/native}} only contains {{libhadoop}}, so this misses.
# It extracts the library from {{hdds-rocks-native-*.jar}} into
{{native.lib.tmp.dir}}. If that property is not set, it uses {{new
File("").getAbsoluteFile()}}, the *current working directory*.
No Ozone script sets {{native.lib.tmp.dir}}. Whenever the process cannot write
to its working directory, extraction fails and the library is not loaded:
{code}
WARN utils.NativeLibraryLoader: Unable to load library: ozone_rocksdb_tools
java.nio.file.AccessDeniedException:
/opt/hadoop/ozone_rocksdb_tools7373108702252003876
WARN snapshot.SnapshotDiffManager: Native Library for raw sst file reading
loading failed. Fallback to performing a full diff instead. ...
{code}
This was hit in the compose runner image: the OM runs in {{/opt/hadoop}}, the
mounted distribution, and that directory was not writable by the container
user. The fallback is silent apart from the WARN, so snapshot diff quietly
loses the native path. {{ozone debug checknative}} then reports {{rocks-tools:
false}} for this reason, which would also mask the libstdc++ problem in
HDDS-17153.
For comparison:
* Hadoop's {{NativeCodeLoader}} only uses {{System.loadLibrary}} from
{{lib/native}} and never extracts.
* RocksJava's own {{NativeLibraryLoader}} extracts {{librocksdbjni}} into
{{java.io.tmpdir}} by default ({{File.createTempFile}}), or into
{{ROCKSDB_SHAREDLIB_DIR}}.
h3. Fix
When {{native.lib.tmp.dir}} is not set (or empty), create the extraction
directory with {{Files.createTempDirectory(libraryName)}}, i.e. under
{{java.io.tmpdir}}. An explicit {{native.lib.tmp.dir}} behaves as before.
Cleanup is unchanged: the existing shutdown hook deletes the directory.
h3. Testing done
* {{TestNativeLibraryLoader#testDummyLibrary}} now expects the library under
{{java.io.tmpdir}} when the property is {{null}} or empty. Previously it
expected the working directory.
** Without the fix, those two cases fail ({{Expecting actual not to be
empty}}). With the fix, they pass.
** The explicit-directory case still passes.
* Native build ({{-Drocks_tools_native}}) on master: TestNativeLibraryLoader
(6, including the real {{ozone_rocksdb_tools}} load),
TestManagedRawSSTFileIterator (28), TestLatestVersionedKWayMergeIterator (10)
and TestLatestVersionedKWayMergeIteratorOverSst (2) all pass. Nothing was
extracted into the module directory.
* Checkstyle is clean.
> NativeLibraryLoader falls back to java.io.tmpdir when the working directory
> is not writable
> -------------------------------------------------------------------------------------------
>
> Key: HDDS-17155
> URL: https://issues.apache.org/jira/browse/HDDS-17155
> Project: Apache Ozone
> Issue Type: Bug
> Components: Native
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
> Attachments: HDDS-17155.001.patch
>
>
> h3. Problem
> {{NativeLibraryLoader}} loads {{libozone_rocksdb_tools.so}} in two steps:
> # It tries {{System.loadLibrary}} from {{java.library.path}}. The
> distribution's {{lib/native}} only contains {{libhadoop}}, so this misses.
> # It extracts the library from {{hdds-rocks-native-*.jar}} into
> {{native.lib.tmp.dir}}. If that property is not set, it uses {{new
> File("").getAbsoluteFile()}}, the *current working directory*.
> No Ozone script sets {{native.lib.tmp.dir}}. Whenever the process cannot
> write to its working directory, extraction fails and the library is not
> loaded:
> {code}
> WARN utils.NativeLibraryLoader: Unable to load library: ozone_rocksdb_tools
> java.nio.file.AccessDeniedException:
> /opt/hadoop/ozone_rocksdb_tools7373108702252003876
> WARN snapshot.SnapshotDiffManager: Native Library for raw sst file reading
> loading failed. Fallback to performing a full diff instead. ...
> {code}
> This was hit in the compose runner image: the OM runs in {{/opt/hadoop}}, the
> mounted distribution, and that directory was not writable by the container
> user. The fallback is silent apart from the WARN, so snapshot diff quietly
> loses the native path. {{ozone debug checknative}} then reports
> {{rocks-tools: false}} for this reason, which would also mask the libstdc++
> problem in HDDS-17153.
> For comparison:
> * Hadoop's {{NativeCodeLoader}} only uses {{System.loadLibrary}} from
> {{lib/native}} and never extracts.
> * RocksJava's own {{NativeLibraryLoader}} extracts {{librocksdbjni}} into
> {{java.io.tmpdir}} by default ({{File.createTempFile}}), or into
> {{ROCKSDB_SHAREDLIB_DIR}}.
> h3. Fix
> Keep the working directory as the default extraction location, and fall back
> to {{java.io.tmpdir}} ({{Files.createTempDirectory(libraryName)}}) only when
> the extraction directory cannot be created under the working directory.
> Deployments that load the library today are unaffected, including ones where
> {{/tmp}} is mounted {{noexec}}. An explicit {{native.lib.tmp.dir}} behaves as
> before and does not fall back. Cleanup is unchanged: the existing shutdown
> hook deletes the directory.
> h3. Testing done
> * {{TestNativeLibraryLoader#testDummyLibrary}} covers the working directory
> default (property {{null}} or empty), an explicit {{native.lib.tmp.dir}}, and
> the fallback to {{java.io.tmpdir}} when the working directory cannot hold the
> extraction directory (property {{null}} or empty). The two fallback cases
> fail without the fix.
> * Checkstyle is clean.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
