Hello Community Members ,
I would like to propose a change to how we handle sparse image generation for
ext* file-systems in OpenEmbedded,
specifically moving away from the generic img2simg tool toward the specialised
ext2simg in the e2fsprogs [contrib/android]
*1) Problem*
Currently, image_types_sparse.bbclass uses img2simg for all file-system types.
img2simg is a tool that reads the raw image to find zero-filled blocks.
For ext* file-systems, this is inefficient compared to ext2simg from e2fsprogs
[contrib/android]
which understands the file-system structure and uses the block bitmap to
determine sparse areas.
*2) History*
Previously, 'ext2simg' was bundled within android-tools (v5.1 and older).
The utility has since migrated to the e2fsprogs project (contrib/android).
However, upstream e2fsprogs does not provide a build system for this tool,
and it requires libsparse from android-tools to compile.
*3) Proposed Changes*
To implement this, I am looking at a cross-layer approach:
a. meta-oe (android-tools):
Export libsparse, libbase, and liblog headers and libraries to the sysroot.
Currently, these are often internal to the build.
Reorganise header installation to ${includedir}/sparse to match the expected
include paths for e2fsprogs contrib tools.
b. oe-core (e2fsprogs):
Add a conditional compilation step in e2fsprogs-native to build the ext4sparse
utility located in contrib/android/.
This would link against the exported libsparse from android-tools-native.
c. meta-oe (image_types_sparse.bbclass):
Update CONVERSION_CMD:sparse to detect ext* types.
Use ext2simg from the e2fsprogs [contrib/android] for these types while
falling back to img2simg for others (like f2fs).
*4) Points for Discussion / Feasibility*
I am seeking feedback on the following:
*a. Layer Dependency:*
This introduces a tighter coupling between oe-core and meta-oe.
Is it acceptable for e2fsprogs-native (core) to optionally depend on
android-tools-native (meta-oe),
or i can create an bbapend file in meta-oe called e2fsprogs_%.bbappend.
*b. Build Integration:*
In my PoC, I've used a manual ${CC} call in do_compile:append.
Would it be preferred to patch the e2fsprogs Makefile to handle this "contrib"
tool more natively?
*c. Library Handling:*
To ensure the native binary finds libsparse at image creation time, I am
currently using LD_LIBRARY_PATH in the class.
I suspect a better approach would be ensuring proper RPATH during the e2fsprogs
build.
If members can help for any better way of doing this
*d. I have a working proof of concept using RPI4 build*
Attached alon are the patches
0001-e2fsprogs-creates-ext4sparse-using-android-tools-lib.patch
0001-meta-oe-image_types_sparse-use-ext4sparse-for-ext-im.patch
*5) Logical layout of current POC*
This commit bridges the gap between these two recipes:
1. android-tools:
- Updated to v29.0.6 and masked v5.1.1 to provide modern libraries.
- Modified do_install to export libsparse, libbase, and liblog
headers and libraries to the sysroot, enabling external linking.
2. image_types_sparse.bbclass:
- Updated CONVERSION_CMD to branch by filesystem type, calling
ext4sparse specifically for ext* images.
- Exported LD_LIBRARY_PATH to the native sysroot to ensure the
ext4sparse binary can locate its shared dependencies at runtime.
3. Layer Configuration:
- Updated layer.conf to support dynamic-layers for SELinux and
synchronized BBMASK entries to prioritize the updated tools.
*6) Benefit :*
This reduces the time and cost involved while flashing Android RFS / Fastboot
etc
Even in CI/CD , which all adds to ease of usage and have monetary benefit
involved with continous CI/CD
Looking forward to thoughts / feedback from community members
Please do let me know if any specific platform or group needs to be addressed
to have such discussion
If there is any specific meeting involved where i can join and explain , please
do let me know
I can connect to team members and works as per suggestion from community to
fine tune / rework if any specific changes community suggest
Thanks ,
Ashish Kumar Mishra
From 533648b1208131d4adab15c48e5c7eb5c0772d73 Mon Sep 17 00:00:00 2001
From: AshishKumar Mishra <[email protected]>
Date: Mon, 9 Feb 2026 11:11:08 +0530
Subject: [PATCH] meta-oe: image_types_sparse: use ext4sparse for ext* image
conversion
Update the sparse image conversion process for ext* filesystems to use
ext4sparse (ext2simg) instead of the generic img2simg tool.
Unlike img2simg, which treats all zero-blocks identically, ext4sparse leverages
e2fsprogs knowledge of the filesystem.
This allows it to mark zero-blocks outside of the ext-filesystem as
"don't care" (skip chunks) rather than "fill-with-zeros" (raw chunks)
resulting in more optimized and smaller sparse images.
Previously, 'ext2simg' was bundled within android-tools (v5.1 and older).
The utility has since migrated to the e2fsprogs project (contrib/android).
However, upstream e2fsprogs does not provide a build system for this tool,
and it requires libsparse from android-tools to compile.
This commit bridges the gap between these two recipes:
1. android-tools:
- Updated to v29.0.6 and masked v5.1.1 to provide modern libraries.
- Modified do_install to export libsparse, libbase, and liblog
headers and libraries to the sysroot, enabling external linking.
2. image_types_sparse.bbclass:
- Updated CONVERSION_CMD to branch by filesystem type, calling
ext4sparse specifically for ext* images.
- Exported LD_LIBRARY_PATH to the native sysroot to ensure the
ext4sparse binary can locate its shared dependencies at runtime.
3. Layer Configuration:
- Updated layer.conf to support dynamic-layers for SELinux and
synchronized BBMASK entries to prioritize the updated tools.
Upstream-Status: Pending
Signed-off-by: AshishKumar Mishra <[email protected]>
---
meta-oe/classes/image_types_sparse.bbclass | 26 +++++++---
meta-oe/conf/layer.conf | 13 ++++-
.../android-tools/android-tools_29.0.6.r14.bb | 51 +++++++++++++++++++
3 files changed, 81 insertions(+), 9 deletions(-)
diff --git a/meta-oe/classes/image_types_sparse.bbclass b/meta-oe/classes/image_types_sparse.bbclass
index 5416c2a019..03e0ecd6c3 100644
--- a/meta-oe/classes/image_types_sparse.bbclass
+++ b/meta-oe/classes/image_types_sparse.bbclass
@@ -12,12 +12,22 @@ CONVERSIONTYPES += "sparse"
DELETE_RAWIMAGE_AFTER_SPARSE_CMD ??= "0"
CONVERSION_CMD:sparse = " \
- truncate --no-create --size=%${SPARSE_BLOCK_SIZE} "${IMAGE_NAME}.${type}"; \
- img2simg -s "${IMAGE_NAME}.${type}" "${IMAGE_NAME}.${type}.sparse" ${SPARSE_BLOCK_SIZE}; \
- if [ "${DELETE_RAWIMAGE_AFTER_SPARSE_CMD}" = "1" ]; then \
- rm -f ${IMAGE_NAME}.${type};\
- bbwarn "Raw file ${IMAGE_NAME}.${type} removed" ;\
- fi;\
- "
+ truncate --no-create --size=%${SPARSE_BLOCK_SIZE} '${IMAGE_NAME}.${type}'; \
+ case '${type}' in \
+ ext*) \
+ export LD_LIBRARY_PATH='${STAGING_LIBDIR_NATIVE}:${LD_LIBRARY_PATH}'; \
+ \
+ bbwarn 'Running e2fsprogs-derived ext4sparse...' ; \
+ ext4sparse '${IMAGE_NAME}.${type}' '${IMAGE_NAME}.simg' || bberror 'ext4sparse failed' \
+ ;; \
+ *) \
+ bbwarn 'Generating sparse image for non-ext filesystem...'; \
+ img2simg -s '${IMAGE_NAME}.${type}' '${IMAGE_NAME}.${type}.sparse' ${SPARSE_BLOCK_SIZE}; \
+ ;; \
+ esac; \
+ if [ '${DELETE_RAWIMAGE_AFTER_SPARSE_CMD}' = '1' ]; then \
+ rm -f '${IMAGE_NAME}.${type}'; \
+ fi; \
+"
-CONVERSION_DEPENDS_sparse = "android-tools-native"
+CONVERSION_DEPENDS:sparse = "android-tools-native e2fsprogs-native"
diff --git a/meta-oe/conf/layer.conf b/meta-oe/conf/layer.conf
index 186ff9a488..9ce2cdb66c 100644
--- a/meta-oe/conf/layer.conf
+++ b/meta-oe/conf/layer.conf
@@ -12,7 +12,16 @@
BBPATH .= ":${LAYERDIR}"
# We have a recipes directory, add to BBFILES
-BBFILES += "${LAYERDIR}/recipes-*/*/*.bb ${LAYERDIR}/recipes-*/*/*.bbappend"
+# Adding dynamic layers recipes
+BBFILES += "\
+ ${LAYERDIR}/recipes-*/*/*.bb \
+ ${LAYERDIR}/recipes-*/*/*.bbappend \
+ ${LAYERDIR}/dynamic-layers/selinux/recipes-*/*/*.bb \
+ ${LAYERDIR}/dynamic-layers/selinux/recipes-*/*/*.bbappend \
+"
+
+# Manually removing default android-tools recipes
+BBMASK += "meta-openembedded/meta-oe/recipes-devtools/android-tools/android-tools_5.1.1.r37.bb"
BBFILE_COLLECTIONS += "openembedded-layer"
BBFILE_PATTERN_openembedded-layer := "^${LAYERDIR}/"
@@ -40,6 +49,8 @@ BBFILES_DYNAMIC += " \
perl-layer:${LAYERDIR}/dynamic-layers/perl-layer/recipes-*/*/*.bbappend \
selinux:${LAYERDIR}/dynamic-layers/selinux/recipes-*/*/*.bb \
selinux:${LAYERDIR}/dynamic-layers/selinux/recipes-*/*/*.bbappend \
+ selinux-layer:${LAYERDIR}/dynamic-layers/selinux/recipes-*/*/*.bb \
+ selinux-layer:${LAYERDIR}/dynamic-layers/selinux/recipes-*/*/*.bbappend \
"
# This should only be incremented on significant changes that will
diff --git a/meta-oe/dynamic-layers/selinux/recipes-devtool/android-tools/android-tools_29.0.6.r14.bb b/meta-oe/dynamic-layers/selinux/recipes-devtool/android-tools/android-tools_29.0.6.r14.bb
index 74928ed171..0a8518a1dd 100644
--- a/meta-oe/dynamic-layers/selinux/recipes-devtool/android-tools/android-tools_29.0.6.r14.bb
+++ b/meta-oe/dynamic-layers/selinux/recipes-devtool/android-tools/android-tools_29.0.6.r14.bb
@@ -190,3 +190,54 @@ FILES:${PN}-fstools = "\
FILES:${PN} += "${libdir}/android ${libdir}/android/*"
BBCLASSEXTEND = "native"
+
+android_tools_enable_devmode() {
+ touch ${IMAGE_ROOTFS}/etc/usb-debugging-enabled
+}
+
+ROOTFS_POSTPROCESS_COMMAND_${PN}-adbd += "${@bb.utils.contains("USB_DEBUGGING_ENABLED", "1", "android_tools_enable_devmode;", "", d)}"
+
+# e2fsprogs expecting headers in sparse/ subdirectory
+do_install:append() {
+ # to resolve the "Multiple shlib providers" conflict.
+ rm -f ${D}${libdir}/android/libsparse.so*
+ rm -f ${D}${libdir}/android/libbase.so*
+ rm -f ${D}${libdir}/android/liblog.so*
+
+ if [ -d "${S}/system/core/libsparse/include/sparse" ]; then
+ install -d ${D}${includedir}/sparse
+ cp -r ${S}/system/core/libsparse/include/sparse/* ${D}${includedir}/sparse/
+ else
+ bberror "Sparse headers not found in ${S}/system/core/libsparse/include/sparse"
+ fi
+
+ install -d ${D}${libdir}
+ for lib in libsparse libbase liblog; do
+ if [ -f "${S}/debian/out/system/core/${lib}.so" ]; then
+ bbwarn "Installing ${lib} to sysroot"
+ install -m 0755 ${S}/debian/out/system/core/${lib}.so ${D}${libdir}/${lib}.so.0
+ ln -sf ${lib}.so.0 ${D}${libdir}/${lib}.so
+ fi
+ done
+
+ install -d ${D}${libdir}/android
+ for lib in libsparse libbase liblog; do
+ if [ -f "${D}${libdir}/${lib}.so.0" ]; then
+ ln -sf ../${lib}.so.0 ${D}${libdir}/android/${lib}.so.0
+ ln -sf ../${lib}.so ${D}${libdir}/android/${lib}.so
+ fi
+ done
+}
+
+FILES:${PN}-dev += " \
+ ${includedir}/sparse \
+ ${libdir}/lib*.so \
+ ${libdir}/android/lib*.so \
+"
+
+FILES:${PN} += " \
+ ${libdir}/lib*.so.* \
+ ${libdir}/android/lib*.so.* \
+"
+
+SYSROOT_DIRS:append:class-native = " ${includedir} ${libdir}"
--
2.34.1
From d43f0747a57a976578f0bfa259fae11063b6b12d Mon Sep 17 00:00:00 2001
From: AshishKumar Mishra <[email protected]>
Date: Mon, 9 Feb 2026 11:14:33 +0530
Subject: [PATCH] e2fsprogs: creates ext4sparse using android-tools library
Add a custom do_compile step to build the ext4sparse utility (ext2simg)
when building for the native class using android-tools-native.
Includes debug logging for build verification
Upstream-Status: Pending
Signed-off-by: AshishKumar Mishra <[email protected]>
---
.../e2fsprogs/e2fsprogs_1.47.3.bb | 23 +++++++++++++++++++
1 file changed, 23 insertions(+)
diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.47.3.bb b/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.47.3.bb
index 40658399d9..090dbd1282 100644
--- a/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.47.3.bb
+++ b/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.47.3.bb
@@ -151,3 +151,26 @@ do_install_ptest() {
install -d ${D}${PTEST_PATH}/data
install -m 0644 ${B}/tests/test_data.tmp ${D}${PTEST_PATH}/data/
}
+
+DEPENDS:append:class-native = " android-tools-native "
+
+do_compile:append:class-native() {
+ bbwarn "=============== DEBUG ======================="
+ bbwarn "Compiling ext2simg.c with native toolchain"
+ # Source directory for ext2simg.c in the e2fsprogs tree
+ SRC_EXT2SIMG="${S}/contrib/android"
+
+ INCLUDES="-I${S}/lib -I${B}/lib -I${SRC_EXT2SIMG}/lib"
+
+ # STAGING_LIBDIR_NATIVE: where android-tools-native installed libsparse
+ LIBS="-L${B}/lib -L${STAGING_LIBDIR_NATIVE} -lsparse -lext2fs -lz -lcom_err"
+
+ ${CC} ${CFLAGS} ${LDFLAGS} -o "${B}/ext4sparse" "${SRC_EXT2SIMG}/ext2simg.c" ${INCLUDES} ${LIBS}
+ bbwarn "ext4sparse compilation finished"
+ bbwarn "=============== DEBUG ======================="
+}
+
+do_install:append:class-native() {
+ install -d ${D}${bindir}
+ install -m 0755 ${B}/ext4sparse ${D}${bindir}/ext4sparse
+}
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#230708):
https://lists.openembedded.org/g/openembedded-core/message/230708
Mute This Topic: https://lists.openembedded.org/mt/117715350/21656
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
