[
https://issues.apache.org/jira/browse/HDFS-16128?focusedWorklogId=627368&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-627368
]
ASF GitHub Bot logged work on HDFS-16128:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 24/Jul/21 05:23
Start Date: 24/Jul/21 05:23
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3201:
URL: https://github.com/apache/hadoop/pull/3201#issuecomment-886002374
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
|||| _ fgl Compile Tests _ |
| +0 :ok: | mvndep | 13m 19s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 57s | | fgl passed |
| +1 :green_heart: | compile | 20m 42s | | fgl passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 18m 26s | | fgl passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 50s | | fgl passed |
| +1 :green_heart: | mvnsite | 3m 20s | | fgl passed |
| +1 :green_heart: | javadoc | 2m 15s | | fgl passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 22s | | fgl passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | spotbugs | 2m 28s |
[/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3201/7/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html)
| hadoop-common-project/hadoop-common in fgl has 3 extant spotbugs warnings.
|
| -1 :x: | spotbugs | 3m 32s |
[/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3201/7/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html)
| hadoop-hdfs-project/hadoop-hdfs in fgl has 2 extant spotbugs warnings. |
| -1 :x: | shadedclient | 16m 49s | | branch has errors when building
and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 11s | | the patch passed |
| +1 :green_heart: | compile | 22m 42s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 22m 42s | | the patch passed |
| +1 :green_heart: | compile | 19m 34s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | javac | 19m 34s |
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3201/7/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 1 new + 1792
unchanged - 1 fixed = 1793 total (was 1793) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 1s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3201/7/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 2 new + 121 unchanged - 0 fixed = 123 total (was
121) |
| +1 :green_heart: | mvnsite | 3m 22s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 8s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 27s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 38s | |
hadoop-common-project/hadoop-common generated 0 new + 2 unchanged - 1 fixed = 2
total (was 3) |
| +1 :green_heart: | spotbugs | 3m 31s | |
hadoop-hdfs-project/hadoop-hdfs generated 0 new + 0 unchanged - 2 fixed = 0
total (was 2) |
| +1 :green_heart: | shadedclient | 17m 34s | | patch has no errors
when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 17m 22s | | hadoop-common in the patch
passed. |
| -1 :x: | unit | 284m 44s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3201/7/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 14s | | The patch does not
generate ASF License warnings. |
| | | 492m 37s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.TestErasureCodingExerciseAPIs |
| | hadoop.hdfs.TestDistributedFileSystem |
| | hadoop.hdfs.TestErasureCodingPolicyWithSnapshot |
| | hadoop.hdfs.server.namenode.TestNameNodeRecovery |
| |
hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForStoragePolicy |
| | hadoop.hdfs.server.namenode.TestFSImageWithAcl |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap |
| | hadoop.hdfs.server.namenode.TestFSNamesystem |
| | hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer |
| | hadoop.fs.TestFcHdfsSetUMask |
| | hadoop.hdfs.server.namenode.TestFSImage |
| | hadoop.hdfs.server.namenode.snapshot.TestOrderedSnapshotDeletion |
| | hadoop.hdfs.server.namenode.TestStorageRestore |
| | hadoop.hdfs.server.namenode.TestPersistentStoragePolicySatisfier |
| | hadoop.security.TestPermission |
| | hadoop.hdfs.TestRollingUpgradeRollback |
| | hadoop.hdfs.TestErasureCodingPolicyWithSnapshotWithRandomECPolicy |
| |
hadoop.hdfs.server.namenode.snapshot.TestFSImageWithOrderedSnapshotDeletion |
| | hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints |
| | hadoop.hdfs.TestDFSUpgradeFromImage |
| | hadoop.hdfs.server.namenode.snapshot.TestAclWithSnapshot |
| | hadoop.cli.TestAclCLI |
| | hadoop.hdfs.server.namenode.snapshot.TestXAttrWithSnapshot |
| | hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot |
| | hadoop.hdfs.server.namenode.TestNameNodeAcl |
| | hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics |
| | hadoop.hdfs.server.namenode.snapshot.TestCheckpointsWithSnapshots |
| | hadoop.hdfs.TestDFSPermission |
| | hadoop.hdfs.TestViewDistributedFileSystem |
| | hadoop.hdfs.server.namenode.TestFileContextAcl |
| | hadoop.hdfs.TestExtendedAcls |
| | hadoop.hdfs.server.namenode.snapshot.TestRandomOpsWithSnapshots |
| | hadoop.hdfs.TestRollingUpgradeDowngrade |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapRootDescendantDiff |
| | hadoop.hdfs.server.namenode.TestStartup |
| | hadoop.hdfs.server.namenode.TestCacheDirectives |
| | hadoop.hdfs.server.namenode.TestSecondaryNameNodeUpgrade |
| | hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshot |
| | hadoop.cli.TestAclCLIWithPosixAclInheritance |
| | hadoop.hdfs.server.namenode.TestFSNamesystemLock |
| | hadoop.hdfs.server.namenode.TestNameEditsConfigs |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport |
| | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.server.namenode.TestFSImageWithSnapshot |
| | hadoop.hdfs.web.TestWebHDFSAcl |
| | hadoop.hdfs.server.namenode.TestCommitBlockSynchronization |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.namenode.TestFileTruncate |
| | hadoop.hdfs.server.namenode.TestCacheDirectivesWithViewDFS |
| | hadoop.hdfs.server.namenode.TestFSNamesystemLockReport |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3201/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3201 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 19c2698267da 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | fgl / 5cd228b75554124dc635053ee00b8e7542752fd5 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3201/7/testReport/ |
| Max. process+thread count | 3066 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3201/7/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 627368)
Remaining Estimate: 0h
Time Spent: 10m
> [FGL] Add support for saving/loading an FS Image for PartitionedGSet
> --------------------------------------------------------------------
>
> Key: HDFS-16128
> URL: https://issues.apache.org/jira/browse/HDFS-16128
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs, namenode
> Reporter: Xing Lin
> Assignee: Xing Lin
> Priority: Major
> Labels: pull-request-available
> Fix For: Fine-Grained Locking
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Add support to save Inodes stored in PartitionedGSet when saving an FS image
> and load Inodes into PartitionedGSet from a saved FS image.
> h1. Saving FSImage
> *Original HDFS design*: iterate every inode in inodeMap and save them into
> the FSImage file.
> *FGL*: no change is needed here, since PartitionedGSet also provides an
> iterator interface, to iterate over inodes stored in partitions.
> h1. Loading an HDFS
> *Original HDFS design*: it first loads the FSImage files and then loads edit
> logs for recent changes. FSImage files contain different sections, including
> INodeSections and INodeDirectorySections. An InodeSection contains serialized
> Inodes objects and the INodeDirectorySection contains the parent inode for an
> Inode. When loading an FSImage, the system first loads INodeSections and then
> load the INodeDirectorySections, to set the parent inode for each inode.
> After FSImage files are loaded, edit logs are then loaded. Edit log contains
> recent changes to the filesystem, including Inodes creation/deletion. For a
> newly created INode, the parent inode is set before it is added to the
> inodeMap.
> *FGL*: when adding an Inode into the partitionedGSet, we need the parent
> inode of an inode, in order to determine which partition to store that inode,
> when NAMESPACE_KEY_DEPTH = 2. Thus, in FGL, when loading FSImage files, we
> used a temporary LightweightGSet (inodeMapTemp), to store inodes. When
> LoadFSImage is done, the parent inode for all existing inodes in FSImage
> files is set. We can now move the inodes into a partitionedGSet. Load edit
> logs can work as usual, as the parent inode for an inode is set before it is
> added to the inodeMap.
> In theory, PartitionedGSet can support to store inodes without setting its
> parent inodes. All these inodes will be stored in the 0th partition. However,
> we decide to use a temporary LightweightGSet (inodeMapTemp) to store these
> inodes, to make this case more transparent.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
