eric-badger edited a comment on pull request #2513: URL: https://github.com/apache/hadoop/pull/2513#issuecomment-761225895
Still haven't found an issue with the squashfs creation yet, but there is some inherent parsing that needs to happen to convert OCI images into squashfs filesystems that can be read correctly by overlayFS. It looks like whiteout files and opaque directories are not implemented in this PR. The issue is that OCI images handle whiteouts/opaque directories in an annoyingly different way than overlayFS does. OCI uses '.wh.' and '.wh..wh..opq' files while OverlayFS uses a character devices and directory extended attributes. OCI Standard: https://github.com/opencontainers/image-spec/blob/master/layer.md#whiteouts > Whiteouts > - A whiteout file is an empty file with a special filename that signifies a path should be deleted. > - A whiteout filename consists of the prefix .wh. plus the basename of the path to be deleted. > - As files prefixed with .wh. are special whiteout markers, it is not possible to create a filesystem which has a file or directory with a name beginning with .wh.. > - Once a whiteout is applied, the whiteout itself MUST also be hidden. > - Whiteout files MUST only apply to resources in lower/parent layers. > - Files that are present in the same layer as a whiteout file can only be hidden by whiteout files in subsequent layers. > Opaque Whiteout > - In addition to expressing that a single entry should be removed from a lower layer, layers may remove all of the children using an opaque whiteout entry. > - An opaque whiteout entry is a file with the name .wh..wh..opq indicating that all siblings are hidden in the lower layer. OverlayFS: https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt > whiteouts and opaque directories > -------------------------------- > > In order to support rm and rmdir without changing the lower > filesystem, an overlay filesystem needs to record in the upper filesystem > that files have been removed. This is done using whiteouts and opaque > directories (non-directories are always opaque). > > A whiteout is created as a character device with 0/0 device number. > When a whiteout is found in the upper level of a merged directory, any > matching name in the lower level is ignored, and the whiteout itself > is also hidden. > > A directory is made opaque by setting the xattr "trusted.overlay.opaque" > to "y". Where the upper filesystem contains an opaque directory, any > directory in the lower filesystem with the same name is ignored. There's some discussion [here](https://github.com/opencontainers/image-spec/issues/24) as to why the OCI spec didn't choose to use the overlayFS method of whiteouts. Mostly it appears to be inconsistent behavior in tar that may or may not support the extended attributes and that they didn't want to depend on. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
