On Thu, May 23, 2024 at 10:28 PM Gao Xiang wrote:
>
> Hi Sandeep,
>
Hi Gao,
> On 2024/5/24 05:01, Sandeep Dhavale wrote:
> > Depending on size of the filesystem being built there can be huge number
> > of elements in the hashmap. Currently we call hashmap_iter_first() in
> > while loop to iterate
I got a report from AOSP user that performance of mkfs.erofs with dedupe
option that mkfs.erofs time increased to very high number. For example
creation of 8GB uncompressed erofs image increased from 36seconds to
27minutes when dedupe was enabled. After profiling mkfs.erofs for sample
data, I
Depending on size of the filesystem being built there can be huge number
of elements in the hashmap. Currently we call hashmap_iter_first() in
while loop to iterate and free the elements. However technically
correct, this is inefficient in 2 aspects.
- As we are iterating the elements for
This helper sets hasmap.shrink_at to 0. This is helpful to iterate over
hashmap using hashmap_iter_next() and use hashmap_remove() in single
pass efficeintly.
Signed-off-by: Sandeep Dhavale
---
include/erofs/hashmap.h | 4
1 file changed, 4 insertions(+)
diff --git
On Mon, May 20, 2024 at 2:06 AM Gao Xiang wrote:
>
> Currently, each DEFLATE stream takes one 32 KiB permanent internal
> window buffer even if there is no running instance which uses DEFLATE
> algorithm.
>
> It's unexpected and wasteful on embedded devices with limited resources
> and servers
mkfs.erofs supports creating filesystem images with different
blocksizes. Add filesystem blocksize in super block dump so
its easier to inspect the filesystem.
The field is added after FS magic, so the output now looks like:
Filesystem magic number: 0xE0F5E1E2
Filesystem
Add optimization to treat data blocks filled with 0s as a hole.
Even though diskspace savings are comparable to chunk based or dedupe,
having no block assigned saves us redundant disk IOs during read.
To detect blocks filled with zeros during chunking, we insert block
filled with zeros
mkfs.erofs supports creating filesystem images with different
blocksizes. Add filesystem blocksize in super block dump so
its easier to inspect the filesystem.
The field is added after FS magic, so the output now looks like:
Filesystem magic number: 0xE0F5E1E2
Filesystem
On Fri, Apr 12, 2024 at 5:09 PM Gao Xiang wrote:
>
> Hi Sandeep,
>
> On 2024/4/13 06:51, Sandeep Dhavale wrote:
> > mkfs.erofs supports creating filesystem images with different
> > blocksizes. Add filesystem blocksize in super block dump so
> > its easier to inspect the filesystem.
> >
> > The
On Fri, Apr 12, 2024 at 5:07 PM Gao Xiang wrote:
>
> Hi Sandeep,
>
> On 2024/4/10 06:14, Sandeep Dhavale wrote:
> > Add optimization to treat data blocks filled with 0s as a hole.
> > Even though diskspace savings are comparable to chunk based or dedupe,
> > having no block assigned saves us
mkfs.erofs supports creating filesystem images with different
blocksizes. Add filesystem blocksize in super block dump so
its easier to inspect the filesystem.
The filed is added at last, so the output now looks like:
Filesystem magic number: 0xE0F5E1E2
Filesystem blocks:
Looks good.
Reviewed-by: Sandeep Dhavale
Thanks,
Sandeep.
Add optimization to treat data blocks filled with 0s as a hole.
Even though diskspace savings are comparable to chunk based or dedupe,
having no block assigned saves us redundant disk IOs during read.
To detect blocks filled with zeros during chunking, we insert block
filled with zeros
>
> Thanks for catching this, since the original patch is
> for next upstream cycle, may I fold this fix in the
> original patch?
>
Hi Gao,
Sounds good. As the fix is simple, it makes sense to fold it into the
original one.
Thanks,
Sandeep.
erofs will decompress in the preemptible context (kworker or per cpu
thread). As smp_processor_id() cannot be used in preemptible contexts,
use raw_smp_processor_id() instead to index into global buffer pool.
Reported-by: syzbot+27cc650ef45b379df...@syzkaller.appspotmail.com
Fixes: 7a7513292cc6
On Thu, Apr 4, 2024 at 7:00 AM Gao Xiang wrote:
>
> Hi Sandeep,
>
> On 2024/4/4 07:57, Sandeep Dhavale wrote:
> > Add optimization to treat data blocks filled with 0s as a hole.
> > Even though diskspace savings are comparable to chunk based or dedupe,
> > having no block assigned saves us
Add optimization to treat data blocks filled with 0s as a hole.
Even though diskspace savings are comparable to chunk based or dedupe,
having no block assigned saves us redundant disk IOs during read.
This patch detects if the block is filled with zeros and marks
chunk as erofs_holechunk so there
Hi,
We noticed that in android if you build erofs images with ELFs which
have higher alignment say 16K or 64K, there was a considerable increase
in the size of the uncompressed erofs image. The size increase could be
mitigated with -Ededupe or --chunksize=4096 but that still results in
lot of
When we work with sparse files (files with holes), we need to consider
when the contiguous data block starts after each hole to correctly calculate
minextblks so we can merge consecutive chunks later.
Now that we need to recalculate minextblks multiple places, put the logic
in helper function for
Hi Gao,
Thank you for the review. I will update and send the v2.
Thanks,
Sandeep.
When we work with sparse files (files with holes), we need to consider
when the contiguous data block starts after each hole to correctly calculate
minextblks so we can merge consecutive chunks later.
Now that we need to recalculate minextblks multiple places, put the logic
in helper function for
Hi,
While working on a enhancement where blocks filled with zeros can be treated
as a hole (sparse file), I stumbled upon a bug. I found during mkfs.erofs,
the utility handles the sparse files by looking at the holes and marking them
as a `erofs_holechunk`. However the calculation of minextblks
I have been contributing to erofs for sometime and I would like to help
with code reviews as well.
Signed-off-by: Sandeep Dhavale
---
MAINTAINERS | 1 +
1 file changed, 1 insertion(+)
diff --git a/MAINTAINERS b/MAINTAINERS
index 4f298c4187fb..b130340d71bb 100644
--- a/MAINTAINERS
+++
On Sun, Mar 3, 2024 at 7:54 PM Gao Xiang wrote:
>
> syzbot reports a KMSAN reproducer [1] which generates a crafted
> filesystem image and causes IMA to read uninitialized page cache.
>
> Later, (rq->outputsize > rq->inputsize) will be formally supported
> after either large uncompressed
In erofs_find_target_block() when erofs_dirnamecmp() returns 0,
we do not assign the target metabuf. This causes the caller
erofs_namei()'s erofs_put_metabuf() at the end to be not effective
leaving the refcount on the page.
As the page from metabuf (buf->page) is never put, such page cannot be
>
> If it looks good to you, could you resend a formal patch? Thanks!
>
Hi Gao,
This looks better and more readable. I will send a v2.
Thanks,
Sandeep.
> Thanks,
> Gao Xiang
In erofs_find_target_block() when erofs_dirnamecmp() returns 0,
we do not assign the target metabuf. This causes the caller
erofs_namei()'s erofs_put_metabuf() at the end to be not effective
leaving the refcount on the page.
As the page from metabuf (buf->page) is never put, such page cannot be
On Thu, Jan 25, 2024 at 4:01 AM Gao Xiang wrote:
>
> I encountered a race issue after lengthy (~594647 sec) stress tests on
> a 64k-page arm64 VM with several 4k-block EROFS images. The timing
> is like below:
>
> z_erofs_try_inplace_io z_erofs_fill_bio_vec
>
On Sat, Oct 7, 2023 at 1:25 AM Gao Xiang wrote:
>
>
>
> On 2023/10/7 15:16, Sandeep Dhavale wrote:
> > On Fri, Oct 6, 2023 at 8:27 PM Gao Xiang
> > wrote:
> >>
> >> Hi Sandeep!
> >>
> >> On 2023/10/6 06:40, Sandeep Dhavale wrote:
> >>> AC_RUN_IFELSE expects the action if cross compiling. If not
On Fri, Oct 6, 2023 at 8:27 PM Gao Xiang wrote:
>
> Hi Sandeep!
>
> On 2023/10/6 06:40, Sandeep Dhavale wrote:
> > AC_RUN_IFELSE expects the action if cross compiling. If not provided
> > cross compilation fails with error "configure: error: cannot run test
> > program while cross compiling".
> >
AC_RUN_IFELSE expects the action if cross compiling. If not provided
cross compilation fails with error "configure: error: cannot run test
program while cross compiling".
Use 4096 as the buest guess PAGESIZE if cross compiling as it is still
the most common page size.
Reported-in:
On Tue, Sep 19, 2023 at 7:07 PM Gao Xiang wrote:
>
> Hi Sandeep,
>
> On 2023/9/20 05:02, Sandeep Dhavale wrote:
> > We move `idx` pointer as we iterate through for loop based on `count`. If
> > we error out from the loop, restore the pointer to allocated memory
> > before calling free().
> >
> >
We move `idx` pointer as we iterate through for loop based on `count`. If
we error out from the loop, restore the pointer to allocated memory
before calling free().
Fixes: 39147b48b76d ("erofs-utils: lib: add erofs_rebuild_load_tree() helper")
Signed-off-by: Sandeep Dhavale
---
lib/rebuild.c |
The value in variable 'e' is checked without initializing and can
wrongly signify end of tar 2 empty blocks.
Signed-off-by: Sandeep Dhavale
---
lib/tar.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/tar.c b/lib/tar.c
index b58bfd5..c4c89ec 100644
--- a/lib/tar.c
+++
The intended assignment is already part of the next for loop
initialization.
Signed-off-by: Sandeep Dhavale
---
lib/namei.c | 1 -
1 file changed, 1 deletion(-)
diff --git a/lib/namei.c b/lib/namei.c
index 45dbcd3..294d7a3 100644
--- a/lib/namei.c
+++ b/lib/namei.c
@@ -262,7 +262,6 @@ static
If call to inflateInit2() fails, release the memory allocated for buff
before returning.
Signed-off-by: Sandeep Dhavale
---
lib/decompress.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/lib/decompress.c b/lib/decompress.c
index 01f0141..fe8a40c 100644
---
If z_erofs_pack_file_from_fd() fails, take the error path to free up the
allocated resources.
Signed-off-by: Sandeep Dhavale
---
lib/compress.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/lib/compress.c b/lib/compress.c
index 8c79418..81f277a 100644
--- a/lib/compress.c
+++
Need to free allocated buffer_head bh if __erofs_battach() fails.
Signed-off-by: Sandeep Dhavale
---
lib/cache.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/lib/cache.c b/lib/cache.c
index 5205d57..a9948f0 100644
--- a/lib/cache.c
+++ b/lib/cache.c
@@ -282,8 +282,10
padding is only used in the next if() block. Remove the redundant call.
Signed-off-by: Sandeep Dhavale
---
lib/blobchunk.c | 1 -
1 file changed, 1 deletion(-)
diff --git a/lib/blobchunk.c b/lib/blobchunk.c
index 86b29c1..fcc71b8 100644
--- a/lib/blobchunk.c
+++ b/lib/blobchunk.c
@@ -91,7
Free the memory allocated for 'entry' if strdup() fails.
Signed-off-by: Sandeep Dhavale
---
fsck/main.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/fsck/main.c b/fsck/main.c
index 7f78513..3f86da4 100644
--- a/fsck/main.c
+++ b/fsck/main.c
@@ -585,8 +585,10 @@ static
Hi,
Below series contains few small misc fixes found after running through clang
static checker.
Sandeep Dhavale (7):
erofs-utils: fsck: Fix potential memory leak in error path
erofs-utils: lib: Remove redundant line to get padding
erofs-utils: lib: Fix memory leak if __erofs_battach()
This basically follows the fix in kernel commit
001b8ccd0650 (erofs: fix compact 4B support for 16k
block size).
Without this patch fsck on images with 16k block size
reports corruption which is not correct.
Fixes: 76b822726ff8 ("erofs-utils: introduce compacted compression
indexes")
On Wed, Aug 30, 2023 at 4:34 PM Gao Xiang wrote:
>
> Hi Sandeep,
>
> On 2023/8/31 07:16, Sandeep Dhavale wrote:
> > As erofs-utils supports different block sizes upto
> > EROFS_MAX_BLOCK_SIZE, relax the checks so same tools
> > can be used to create images for platforms where
> > page size can be
Set mkfs default blocksize to current platform pagesize.
This means mkfs with default options will work on current
platform. If we are building image for a platform for a different
blocksize, we can override default with -b option up to
EROFS_MAX_BLOCK_SIZE.
Signed-off-by: Sandeep Dhavale
---
As erofs-utils supports different block sizes upto
EROFS_MAX_BLOCK_SIZE, relax the checks so same tools
can be used to create images for platforms where
page size can be greater than 4096.
Signed-off-by: Sandeep Dhavale
---
lib/namei.c | 2 --
mkfs/main.c | 9 +
2 files changed, 5
>
> Sandeep, thoughts?
>
I prefer to modify erofs check and contain the fix there as erofs
cares about it and it's
least invasive. For contexts where erofs cannot tell for sure, it will
always schedule kworker
(like CONFIG_PREEMPT_COUNT=n).
I will also do measurements to see if erofs should
>
> Sorry, but the current lockdep-support functions need to stay focused
> on lockdep. They are not set up for general use, as we already saw
> with rcu_is_watching().
>
Ok, understood.
> If you get a z_erofs_wq_needed() (or whatever) upstream, and if it turns
> out that there is an
> Thank you for the background.
>
> > Paul, Joel,
> > Shall we fix the rcu_read_lock_*held() regardless of how erofs
> > improves the check?
>
> Help me out here. Exactly what is broken with rcu_read_lock_*held(),
> keeping in mind their intended use for lockdep-based debugging?
>
Hi Paul,
With
Hello All,
Let me answer some of the questions raised here.
* Performance aspect
EROFS is one of the popular filesystem of choice in Android for read
only partitions
as it provides 30%+ storage space savings with compression.
In addition to microbenchmarks, boot times and cold app launch
Hi Joel,
Thank you for the review!
> Actually even the original code is incorrect (the lockdep version)
> since preemptible() cannot be relied upon if CONFIG_PREEMPT_COUNT is
> not set. However, that's debug code. In this case, it is a real
> user (the fs code). In non-preemptible kernels, we are
Currently if CONFIG_DEBUG_LOCK_ALLOC is not set
- rcu_read_lock_held() always returns 1
- rcu_read_lock_any_held() may return 0 with CONFIG_PREEMPT_RCU
This is inconsistent and it was discovered when trying a fix
for problem reported [1] with CONFIG_DEBUG_LOCK_ALLOC is not
set and
Current check for atomic context is not sufficient as
z_erofs_decompressqueue_endio can be called under rcu lock
from blk_mq_flush_plug_list(). See the stacktrace [1]
In such case we should hand off the decompression work for async
processing rather than trying to do sync decompression in current
On Tue, Jun 20, 2023 at 11:23 PM Gao Xiang wrote:
>
>
>
> On 2023/6/21 14:18, Sandeep Dhavale wrote:
> > On Tue, Jun 20, 2023 at 5:38 PM Gao Xiang
> > wrote:
>
> ...
>
> >>
> > I think this looks good. rcu_read_lock_any_held() can detect this.
>
> Okay, could you submit a formal patch with
On Tue, Jun 20, 2023 at 5:38 PM Gao Xiang wrote:
>
> Hi Sandeep,
>
> On 2023/6/21 04:38, Sandeep Dhavale wrote:
> > Hi,
> > I think we are under RCU read lock in the stack at
> > blk_mq_flush_plug_list+0x2b0/0x354
> >
> > blk_mq_flush_plug_list calls blk_mq_run_dispatch_ops
> > which is a macro
Hi,
I think we are under RCU read lock in the stack at
blk_mq_flush_plug_list+0x2b0/0x354
blk_mq_flush_plug_list calls blk_mq_run_dispatch_ops
which is a macro in block/blk-mq.h
/* run the code block in @dispatch_ops with rcu/srcu read lock held */
#define __blk_mq_run_dispatch_ops(q,
Hi,
One of the partners reported that they are seeing the warning log
which points out that the sleeping function is being called from
invalid context. We have not seen this in internal testing and the
partner does not have a repro yet. Below is the log.
[17185.137395] [T701615] CpuMonitorServi:
On Mon, May 22, 2023 at 2:21 AM Gao Xiang wrote:
>
> As Sandeep shown [1], high priority RT per-cpu kthreads are
> typically helpful for Android scenarios to minimize the scheduling
> latencies.
>
> Switch EROFS_FS_PCPU_KTHREAD_HIPRI on by default if
> EROFS_FS_PCPU_KTHREAD is on since it's the
On Mon, Feb 27, 2023 at 9:01 PM Gao Xiang wrote:
>
> Hi Sandeep,
>
> On 2023/2/28 12:47, Sandeep Dhavale via Linux-erofs wrote:
> > Hi all,
> > I completed the tests and the results are consistent with
> > our previous observation. We can see that removing WQ_UNBOUN
Hi all,
I completed the tests and the results are consistent with
our previous observation. We can see that removing WQ_UNBOUND
helps but the scheduling latency by using high priority per cpu
kthreads is even lower. Below is the table.
|-+---+---+--+---|
|
On Thu, Feb 23, 2023 at 11:08 AM Gao Xiang wrote:
>
>
>
> On 2023/2/24 02:52, Gao Xiang wrote:
> > Hi Eric,
> >
> > On 2023/2/24 02:29, Eric Biggers wrote:
> >> Hi,
> >>
> >> On Wed, Feb 08, 2023 at 05:33:22PM +0800, Gao Xiang wrote:
> >>> From: Sandeep Dhavale
> >>>
> >>> Using per-cpu thread
On Mon, Feb 6, 2023 at 6:55 PM Gao Xiang wrote:
>
>
>
> On 2023/2/7 03:41, Sandeep Dhavale wrote:
> > On Mon, Feb 6, 2023 at 2:01 AM Gao Xiang wrote:
> >>
> >> Hi Sandeep,
> >>
> >> On Fri, Jan 06, 2023 at 07:35:01AM +, Sandeep Dhavale wrote:
> >>> Using per-cpu thread pool we can reduce the
On Mon, Feb 6, 2023 at 2:01 AM Gao Xiang wrote:
>
> Hi Sandeep,
>
> On Fri, Jan 06, 2023 at 07:35:01AM +, Sandeep Dhavale wrote:
> > Using per-cpu thread pool we can reduce the scheduling latency compared
> > to workqueue implementation. With this patch scheduling latency and
> > variation is
On Sat, Jan 14, 2023 at 1:01 PM Nathan Huckleberry wrote:
>
> On Fri, Jan 13, 2023 at 6:20 PM Gao Xiang wrote:
> >
> > Hi Nathan!
> >
> > On 2023/1/14 05:07, Nathan Huckleberry wrote:
> > > Add a WQ flag that allows workqueues to use SCHED_FIFO with the least
> > > imporant RT priority. This
Using per-cpu thread pool we can reduce the scheduling latency compared
to workqueue implementation. With this patch scheduling latency and
variation is reduced as per-cpu threads are high priority kthread_workers.
The results were evaluated on arm64 Android devices running 5.10 kernel.
The
On Thu, Jan 5, 2023 at 10:02 PM Gao Xiang wrote:
>
> Hi Sandeep,
>
> On 2023/1/6 13:41, Sandeep Dhavale wrote:
> > Using per-cpu thread pool we can reduce the scheduling latency compared
> > to workqueue implementation. With this patch scheduling latency and
> > variation is reduced as per-cpu
Using per-cpu thread pool we can reduce the scheduling latency compared
to workqueue implementation. With this patch scheduling latency and
variation is reduced as per-cpu threads are high priority kthread_workers.
The results were evaluated on arm64 Android devices running 5.10 kernel.
The
Using per-cpu thread pool we can reduce the scheduling latency compared
to workqueue implementation. With this patch scheduling latency and
variation is reduced as per-cpu threads are high priority kthread_workers.
The results were evaluated on arm64 Android devices running 5.10 kernel.
The
On Wed, Dec 21, 2022 at 6:48 PM Gao Xiang wrote:
>
> Hi Sandeen,
>
> On Thu, Dec 22, 2022 at 01:15:29AM +, Sandeep Dhavale wrote:
> > Using per-cpu thread pool we can reduce the scheduling latency compared
> > to workqueue implementation. With this patch scheduling latency and
> > variation
Using per-cpu thread pool we can reduce the scheduling latency compared
to workqueue implementation. With this patch scheduling latency and
variation is reduced as per-cpu threads are SCHED_FIFO kthread_workers.
The results were evaluated on arm64 Android devices running 5.10 kernel.
The table
69 matches
Mail list logo