Re: [PATCH 5/7] btrfs-progs: volumes: Allow find_free_dev_extent() to return maximum hole size

[Nikolay Borisov]

On  2.02.2018 13:49, Qu Wenruo wrote:
> 
> 
> On 2018年02月02日 19:41, Nikolay Borisov wrote:
>>
>>
>> On  2.02.2018 10:19, Qu Wenruo wrote:
>>> Just as kernel find_free_dev_extent(), allow it to return maximum hole
>>> size for us to build device list for later chunk allocator rework.
>>>
>>> Signed-off-by: Qu Wenruo 
>>> ---
>>>  volumes.c | 6 +++---
>>>  1 file changed, 3 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/volumes.c b/volumes.c
>>> index b47ff1f392b5..f4009ffa7c9e 100644
>>> --- a/volumes.c
>>> +++ b/volumes.c
>>> @@ -516,10 +516,10 @@ out:
>>>  }
>>>  
>>>  static int find_free_dev_extent(struct btrfs_device *device, u64 num_bytes,
>>> -   u64 *start)
>>> +   u64 *start, u64 *len)
>>>  {
>>> /* FIXME use last free of some kind */
>>> -   return find_free_dev_extent_start(device, num_bytes, 0, start, NULL);
>>> +   return find_free_dev_extent_start(device, num_bytes, 0, start, len);
>>
>> Why do we need the free_dev_extent() wrapper over free_dev_extent_start
>> at all?
> 
> Because kernel part still needs find_free_dev_extent(), in extent-tree.c.
> 
> I'd prefer to keep the function across kernel and btrfs-progs.

You are right, I missed it while I was grepping the first time.

Reviewed-by: Nikolay Borisov 

> 
> Thanks,
> Qu
> 
>>>  }
>>>  
>>>  static int btrfs_alloc_dev_extent(struct btrfs_trans_handle *trans,
>>> @@ -543,7 +543,7 @@ static int btrfs_alloc_dev_extent(struct 
>>> btrfs_trans_handle *trans,
>>>  * is responsible to make sure it's free.
>>>  */
>>> if (!convert) {
>>> -   ret = find_free_dev_extent(device, num_bytes, start);
>>> +   ret = find_free_dev_extent(device, num_bytes, start, NULL);
>>> if (ret)
>>> goto err;
>>> }
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majord...@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 25/26] btrfs-progs: deprecate libbtrfs helpers with libbtrfsutil equivalents

[David Sterba]

On Fri, Jan 26, 2018 at 10:41:13AM -0800, Omar Sandoval wrote:
> From: Omar Sandoval 
> 
> The old libbtrfs defines some helpers which do the same thing as some
> libbtrfsutil helpers. Reimplement the libbtrfs helpers in terms of the
> libbtrfsutil APIs and mark the libbtrfs versions as deprecated, which we
> could ideally get rid of eventually.

Can we keep both libraries separate, ie. no runtime dependency? The
libbtrfs still has some use internally, the deprecation should happen on
the application level.

If there are some exported functions in libbtrfs that are (and should)
be provided by the util library, then we can use the same .o for
building as a temporary solution.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[josef-btrfs:current-work 3/3] block/blk-wbt.c:876:10: error: implicit declaration of function 'blkg_lookup_create'; did you mean 'blk_lookup_devt'?

[kbuild test robot]

tree:   https://git.kernel.org/pub/scm/linux/kernel/git/josef/btrfs-next.git 
current-work
head:   1ac686f6d1b04b480cd245fb927180e1238bd3ac
commit: 1ac686f6d1b04b480cd245fb927180e1238bd3ac [3/3] current-work
config: x86_64-randconfig-x005-201804 (attached as .config)
compiler: gcc-7 (Debian 7.2.0-12) 7.2.1 20171025
reproduce:
git checkout 1ac686f6d1b04b480cd245fb927180e1238bd3ac
# save the attached .config to linux build tree
make ARCH=x86_64 

All error/warnings (new ones prefixed by >>):

   include/linux/jiffies.h:105:18: note: in definition of macro 'time_after'
  ((long)((b) - (a)) < 0))
 ^
   block/blk-wbt.c:463:3: note: in expansion of macro 'time_before'
  time_before(now, rwb->last_comp + HZ / 10);
  ^~~
   block/blk-wbt.c: In function 'get_limit':
   block/blk-wbt.c:481:16: error: 'struct rq_wb' has no member named 'wb_max'; 
did you mean 'wb_normal'?
  limit = rwb->wb_max;
   ^~
   wb_normal
   block/blk-wbt.c: In function 'wbt_wait':
   block/blk-wbt.c:587:28: error: 'struct rq_wb' has no member named 
'last_issue'; did you mean 'sync_issue'?
   wb_timestamp(rwb, >last_issue);
   ^~
   sync_issue
   block/blk-wbt.c:593:29: error: 'struct rq_wb' has no member named 'cb'
 if (!blk_stat_is_active(rwb->cb))
^~
   block/blk-wbt.c: In function 'wbt_set_queue_depth':
   block/blk-wbt.c:634:8: error: 'struct rq_wb' has no member named 
'queue_depth'; did you mean 'rq_depth'?
  rwb->queue_depth = depth;
   ^~~
   rq_depth
   block/blk-wbt.c: In function 'wbt_set_write_cache':
   block/blk-wbt.c:642:6: error: 'struct rq_wb' has no member named 'wc'
  rwb->wc = write_cache_on;
 ^~
   block/blk-wbt.c: In function 'wbt_init':
   block/blk-wbt.c:704:5: error: 'struct rq_wb' has no member named 'cb'
 rwb->cb = blk_stat_alloc_callback(wb_timer_fn, wbt_data_dir, 2, rwb);
^~
   block/blk-wbt.c:705:10: error: 'struct rq_wb' has no member named 'cb'
 if (!rwb->cb) {
 ^~
   block/blk-wbt.c:713:5: error: 'struct rq_wb' has no member named 'last_comp'
 rwb->last_comp = rwb->last_issue = jiffies;
^~
   block/blk-wbt.c:713:24: error: 'struct rq_wb' has no member named 
'last_issue'; did you mean 'sync_issue'?
 rwb->last_comp = rwb->last_issue = jiffies;
   ^~
   sync_issue
   block/blk-wbt.c:715:5: error: 'struct rq_wb' has no member named 'win_nsec'
 rwb->win_nsec = RWB_WINDOW_NSEC;
^~
   block/blk-wbt.c:723:30: error: 'struct rq_wb' has no member named 'cb'
 blk_stat_add_callback(q, rwb->cb);
 ^~
   block/blk-wbt.c:725:5: error: 'struct rq_wb' has no member named 
'min_lat_nsec'
 rwb->min_lat_nsec = wbt_default_latency_nsec(q);
^~
   block/blk-wbt.c: In function 'blk_qos_exit':
   block/blk-wbt.c:743:12: error: 'struct request_queue' has no member named 
'rwb'; did you mean 'rq_wb'?
 while (q->rwb) {
   ^~~
   rq_wb
   block/blk-wbt.c:744:26: error: 'struct request_queue' has no member named 
'rq_rwb'; did you mean 'rq_wb'?
  struct rq_wb *rwb = q->rq_rwb;
 ^~
 rq_wb
   block/blk-wbt.c:745:17: error: 'struct rq_wb' has no member named 'next'
  q->rq_wb = rwb->next;
^~
   block/blk-wbt.c:746:34: error: 'struct rq_wb' has no member named 'cb'
  blk_stat_remove_callback(q, rwb->cb);
 ^~
   block/blk-wbt.c:747:29: error: 'struct rq_wb' has no member named 'cb'
  blk_stat_free_callback(rwb->cb);
^~
   block/blk-wbt.c: In function 'blkg_to_qg':
   block/blk-wbt.c:783:9: error: implicit declaration of function 'pd_to_tg'; 
did you mean 'pd_to_qg'? [-Werror=implicit-function-declaration]
 return pd_to_tg(blkg_to_pd(blkg, _policy_qos));
^~~~
pd_to_qg
   block/blk-wbt.c:783:9: warning: return makes pointer from integer without a 
cast [-Wint-conversion]
 return pd_to_tg(blkg_to_pd(blkg, _policy_qos));
^
   block/blk-wbt.c: In function 'check_scale_change':
   block/blk-wbt.c:853:14: error: passing argument 1 of 'scale_down' from 
incompatible pointer type [-Werror=incompatible-pointer-types]
  scale_down(>rq_wait, false);
 ^
   block/blk-wbt.c:334:13: note: expected 'struct rq_depth *' but argument is 
of type 'struct rq_wait *'
static void scale_down(struct rq_depth *rqd, bool hard_throttle)
^~
   block/blk-wbt.c:855:12: error: passing argument 1 of 'scale_up' from 
incompatible pointer type [-Werror=incompatible-pointer-types]
  scale_up(>rq_wait);
   ^
   block/blk-wbt.c:312:13: 

[josef-btrfs:current-work 2/3] include/linux/bio.h:521:55: warning: 'struct blkcg_gq' declared inside parameter list will not be visible outside of this definition or declaration

[kbuild test robot]

tree:   https://git.kernel.org/pub/scm/linux/kernel/git/josef/btrfs-next.git 
current-work
head:   1ac686f6d1b04b480cd245fb927180e1238bd3ac
commit: 4ece201cfafab298cd89083567403a53f0971635 [2/3] block: add bi_blkg to 
the bio for cgroups
config: x86_64-randconfig-x018-201804 (attached as .config)
compiler: gcc-7 (Debian 7.2.0-12) 7.2.1 20171025
reproduce:
git checkout 4ece201cfafab298cd89083567403a53f0971635
# save the attached .config to linux build tree
make ARCH=x86_64 

All warnings (new ones prefixed by >>):

   In file included from include/linux/blkdev.h:21:0,
from block//partitions/check.h:3,
from block//partitions/check.c:22:
>> include/linux/bio.h:521:55: warning: 'struct blkcg_gq' declared inside 
>> parameter list will not be visible outside of this definition or declaration
static int bio_associate_blkg(struct bio *bio, struct blkcg_gq *blkg) { 
return 0; }
  ^~~~
   include/linux/bio.h:521:12: warning: 'bio_associate_blkg' defined but not 
used [-Wunused-function]
static int bio_associate_blkg(struct bio *bio, struct blkcg_gq *blkg) { 
return 0; }
   ^~

vim +521 include/linux/bio.h

   506  
   507  #define bio_dev(bio) \
   508  disk_devt((bio)->bi_disk)
   509  
   510  #define bio_devname(bio, buf) \
   511  __bdevname(bio_dev(bio), (buf))
   512  
   513  #ifdef CONFIG_BLK_CGROUP
   514  int bio_associate_blkcg(struct bio *bio, struct cgroup_subsys_state 
*blkcg_css);
   515  int bio_associate_blkg(struct bio *bio, struct blkcg_gq *blkg);
   516  void bio_disassociate_task(struct bio *bio);
   517  void bio_clone_blkcg_association(struct bio *dst, struct bio *src);
   518  #else   /* CONFIG_BLK_CGROUP */
   519  static inline int bio_associate_blkcg(struct bio *bio,
   520  struct cgroup_subsys_state *blkcg_css) { return 
0; }
 > 521  static int bio_associate_blkg(struct bio *bio, struct blkcg_gq *blkg) { 
 > return 0; }
   522  static inline void bio_disassociate_task(struct bio *bio) { }
   523  static inline void bio_clone_blkcg_association(struct bio *dst,
   524  struct bio *src) { }
   525  #endif  /* CONFIG_BLK_CGROUP */
   526  

---
0-DAY kernel test infrastructureOpen Source Technology Center
https://lists.01.org/pipermail/kbuild-all   Intel Corporation


.config.gz
Description: application/gzip

[josef-btrfs:current-work 2/3] include/linux/bio.h:521:55: warning: 'struct blkcg_gq' declared inside parameter list

[kbuild test robot]

tree:   https://git.kernel.org/pub/scm/linux/kernel/git/josef/btrfs-next.git 
current-work
head:   1ac686f6d1b04b480cd245fb927180e1238bd3ac
commit: 4ece201cfafab298cd89083567403a53f0971635 [2/3] block: add bi_blkg to 
the bio for cgroups
config: i386-randconfig-a1-201804 (attached as .config)
compiler: gcc-4.9 (Debian 4.9.4-2) 4.9.4
reproduce:
git checkout 4ece201cfafab298cd89083567403a53f0971635
# save the attached .config to linux build tree
make ARCH=i386 

All warnings (new ones prefixed by >>):

   In file included from include/linux/blkdev.h:21:0,
from include/linux/backing-dev.h:15,
from fs/open.c:15:
>> include/linux/bio.h:521:55: warning: 'struct blkcg_gq' declared inside 
>> parameter list
static int bio_associate_blkg(struct bio *bio, struct blkcg_gq *blkg) { 
return 0; }
  ^
>> include/linux/bio.h:521:55: warning: its scope is only this definition or 
>> declaration, which is probably not what you want
   include/linux/bio.h:521:12: warning: 'bio_associate_blkg' defined but not 
used [-Wunused-function]
static int bio_associate_blkg(struct bio *bio, struct blkcg_gq *blkg) { 
return 0; }
   ^

vim +521 include/linux/bio.h

   506  
   507  #define bio_dev(bio) \
   508  disk_devt((bio)->bi_disk)
   509  
   510  #define bio_devname(bio, buf) \
   511  __bdevname(bio_dev(bio), (buf))
   512  
   513  #ifdef CONFIG_BLK_CGROUP
   514  int bio_associate_blkcg(struct bio *bio, struct cgroup_subsys_state 
*blkcg_css);
   515  int bio_associate_blkg(struct bio *bio, struct blkcg_gq *blkg);
   516  void bio_disassociate_task(struct bio *bio);
   517  void bio_clone_blkcg_association(struct bio *dst, struct bio *src);
   518  #else   /* CONFIG_BLK_CGROUP */
   519  static inline int bio_associate_blkcg(struct bio *bio,
   520  struct cgroup_subsys_state *blkcg_css) { return 
0; }
 > 521  static int bio_associate_blkg(struct bio *bio, struct blkcg_gq *blkg) { 
 > return 0; }
   522  static inline void bio_disassociate_task(struct bio *bio) { }
   523  static inline void bio_clone_blkcg_association(struct bio *dst,
   524  struct bio *src) { }
   525  #endif  /* CONFIG_BLK_CGROUP */
   526  

---
0-DAY kernel test infrastructureOpen Source Technology Center
https://lists.01.org/pipermail/kbuild-all   Intel Corporation


.config.gz
Description: application/gzip

[josef-btrfs:current-work 3/3] block/blk-wbt.c:866:32: error: 'struct rq_qos' has no member named 'q'

[kbuild test robot]

tree:   https://git.kernel.org/pub/scm/linux/kernel/git/josef/btrfs-next.git 
current-work
head:   1ac686f6d1b04b480cd245fb927180e1238bd3ac
commit: 1ac686f6d1b04b480cd245fb927180e1238bd3ac [3/3] current-work
config: x86_64-randconfig-x008-201804 (attached as .config)
compiler: gcc-7 (Debian 7.2.0-12) 7.2.1 20171025
reproduce:
git checkout 1ac686f6d1b04b480cd245fb927180e1238bd3ac
# save the attached .config to linux build tree
make ARCH=x86_64 

All error/warnings (new ones prefixed by >>):

from include/linux/gfp.h:6,
from include/linux/slab.h:15,
from block/blk-wbt.c:23:
   block/blk-wbt.c:463:23: error: 'struct rq_wb' has no member named 'last_comp'
  time_before(now, rwb->last_comp + HZ / 10);
  ^
   include/linux/jiffies.h:105:18: note: in definition of macro 'time_after'
  ((long)((b) - (a)) < 0))
 ^
   block/blk-wbt.c:463:3: note: in expansion of macro 'time_before'
  time_before(now, rwb->last_comp + HZ / 10);
  ^~~
   block/blk-wbt.c: In function 'get_limit':
   block/blk-wbt.c:481:16: error: 'struct rq_wb' has no member named 'wb_max'; 
did you mean 'wb_normal'?
  limit = rwb->wb_max;
   ^~
   wb_normal
   block/blk-wbt.c: In function 'wbt_wait':
   block/blk-wbt.c:587:28: error: 'struct rq_wb' has no member named 
'last_issue'; did you mean 'sync_issue'?
   wb_timestamp(rwb, >last_issue);
   ^~
   sync_issue
   block/blk-wbt.c:593:29: error: 'struct rq_wb' has no member named 'cb'
 if (!blk_stat_is_active(rwb->cb))
^~
   block/blk-wbt.c: In function 'wbt_set_queue_depth':
   block/blk-wbt.c:634:8: error: 'struct rq_wb' has no member named 
'queue_depth'; did you mean 'rq_depth'?
  rwb->queue_depth = depth;
   ^~~
   rq_depth
   block/blk-wbt.c: In function 'wbt_set_write_cache':
   block/blk-wbt.c:642:6: error: 'struct rq_wb' has no member named 'wc'
  rwb->wc = write_cache_on;
 ^~
   block/blk-wbt.c: In function 'wbt_init':
   block/blk-wbt.c:704:5: error: 'struct rq_wb' has no member named 'cb'
 rwb->cb = blk_stat_alloc_callback(wb_timer_fn, wbt_data_dir, 2, rwb);
^~
   block/blk-wbt.c:705:10: error: 'struct rq_wb' has no member named 'cb'
 if (!rwb->cb) {
 ^~
   block/blk-wbt.c:713:5: error: 'struct rq_wb' has no member named 'last_comp'
 rwb->last_comp = rwb->last_issue = jiffies;
^~
   block/blk-wbt.c:713:24: error: 'struct rq_wb' has no member named 
'last_issue'; did you mean 'sync_issue'?
 rwb->last_comp = rwb->last_issue = jiffies;
   ^~
   sync_issue
   block/blk-wbt.c:715:5: error: 'struct rq_wb' has no member named 'win_nsec'
 rwb->win_nsec = RWB_WINDOW_NSEC;
^~
   block/blk-wbt.c:723:30: error: 'struct rq_wb' has no member named 'cb'
 blk_stat_add_callback(q, rwb->cb);
 ^~
   block/blk-wbt.c:725:5: error: 'struct rq_wb' has no member named 
'min_lat_nsec'
 rwb->min_lat_nsec = wbt_default_latency_nsec(q);
^~
   block/blk-wbt.c: In function 'blk_qos_exit':
   block/blk-wbt.c:743:12: error: 'struct request_queue' has no member named 
'rwb'; did you mean 'rq_wb'?
 while (q->rwb) {
   ^~~
   rq_wb
   block/blk-wbt.c:744:26: error: 'struct request_queue' has no member named 
'rq_rwb'; did you mean 'rq_wb'?
  struct rq_wb *rwb = q->rq_rwb;
 ^~
 rq_wb
   block/blk-wbt.c:745:17: error: 'struct rq_wb' has no member named 'next'
  q->rq_wb = rwb->next;
^~
   block/blk-wbt.c:746:34: error: 'struct rq_wb' has no member named 'cb'
  blk_stat_remove_callback(q, rwb->cb);
 ^~
   block/blk-wbt.c:747:29: error: 'struct rq_wb' has no member named 'cb'
  blk_stat_free_callback(rwb->cb);
^~
   block/blk-wbt.c: In function 'blkg_to_qg':
   block/blk-wbt.c:783:9: error: implicit declaration of function 'pd_to_tg'; 
did you mean 'pd_to_qg'? [-Werror=implicit-function-declaration]
 return pd_to_tg(blkg_to_pd(blkg, _policy_qos));
^~~~
pd_to_qg
   block/blk-wbt.c:783:9: warning: return makes pointer from integer without a 
cast [-Wint-conversion]
 return pd_to_tg(blkg_to_pd(blkg, _policy_qos));
^
   block/blk-wbt.c: In function 'check_scale_change':
   block/blk-wbt.c:853:14: error: passing argument 1 of 'scale_down' from 
incompatible pointer type [-Werror=incompatible-pointer-types]
  scale_down(>rq_wait, false);
 ^
   block/blk-wbt.c:334:13: note: expected 'struct rq_depth *' but argument is 
of type 'struct rq_wait *'
   

Re: [PATCH 4/7] btrfs-progs: Introduce btrfs_raid_array and related infrastructures

[Qu Wenruo]

On 2018年02月02日 19:37, Nikolay Borisov wrote:
> 
> 
> On  2.02.2018 10:19, Qu Wenruo wrote:
>> As part of the effort to unify code and behavior between btrfs-progs and
>> kernel, copy the btrfs_raid_array from kernel to btrfs-progs.
>>
>> So later we can use the btrfs_raid_array[] to get needed raid info other
>> than manually do if-else branches.
>>
>> Signed-off-by: Qu Wenruo 
>> ---
>>  ctree.h   | 12 +++-
>>  volumes.c | 66 
>> +++
>>  volumes.h | 30 +
>>  3 files changed, 107 insertions(+), 1 deletion(-)
>>
>> diff --git a/ctree.h b/ctree.h
>> index 17cdac76c58c..c76849d8deb7 100644
>> --- a/ctree.h
>> +++ b/ctree.h
>> @@ -958,7 +958,17 @@ struct btrfs_csum_item {
>>  #define BTRFS_BLOCK_GROUP_RAID5 (1ULL << 7)
>>  #define BTRFS_BLOCK_GROUP_RAID6 (1ULL << 8)
>>  #define BTRFS_BLOCK_GROUP_RESERVED  BTRFS_AVAIL_ALLOC_BIT_SINGLE
>  
> 
> BTRFS_BLOCK_GROUP_RESERVED differs in userspace than in kernel space. 
> The btrfs_Tree header has it defined as: 
> 
> #define BTRFS_BLOCK_GROUP_RESERVED  (BTRFS_AVAIL_ALLOC_BIT_SINGLE | \ 
>   
>  BTRFS_SPACE_INFO_GLOBAL_RSV)

Another good place to cleanup. :)

Thanks,
Qu

> 
> 
> Otherwise LGTM: 
> 
> Reviewed-by: Nikolay Borisov 
> 
>> -#define BTRFS_NR_RAID_TYPES 7
>> +
>> +enum btrfs_raid_types {
>> +BTRFS_RAID_RAID10,
>> +BTRFS_RAID_RAID1,
>> +BTRFS_RAID_DUP,
>> +BTRFS_RAID_RAID0,
>> +BTRFS_RAID_SINGLE,
>> +BTRFS_RAID_RAID5,
>> +BTRFS_RAID_RAID6,
>> +BTRFS_NR_RAID_TYPES
>> +};
>>  
>>  #define BTRFS_BLOCK_GROUP_TYPE_MASK (BTRFS_BLOCK_GROUP_DATA |\
>>   BTRFS_BLOCK_GROUP_SYSTEM |  \
>> diff --git a/volumes.c b/volumes.c
>> index a9dc8c939dc5..b47ff1f392b5 100644
>> --- a/volumes.c
>> +++ b/volumes.c
>> @@ -30,6 +30,72 @@
>>  #include "utils.h"
>>  #include "kernel-lib/raid56.h"
>>  
>> +const struct btrfs_raid_attr btrfs_raid_array[BTRFS_NR_RAID_TYPES] = {
>> +[BTRFS_RAID_RAID10] = {
>> +.sub_stripes= 2,
>> +.dev_stripes= 1,
>> +.devs_max   = 0,/* 0 == as many as possible */
>> +.devs_min   = 4,
>> +.tolerated_failures = 1,
>> +.devs_increment = 2,
>> +.ncopies= 2,
>> +},
>> +[BTRFS_RAID_RAID1] = {
>> +.sub_stripes= 1,
>> +.dev_stripes= 1,
>> +.devs_max   = 2,
>> +.devs_min   = 2,
>> +.tolerated_failures = 1,
>> +.devs_increment = 2,
>> +.ncopies= 2,
>> +},
>> +[BTRFS_RAID_DUP] = {
>> +.sub_stripes= 1,
>> +.dev_stripes= 2,
>> +.devs_max   = 1,
>> +.devs_min   = 1,
>> +.tolerated_failures = 0,
>> +.devs_increment = 1,
>> +.ncopies= 2,
>> +},
>> +[BTRFS_RAID_RAID0] = {
>> +.sub_stripes= 1,
>> +.dev_stripes= 1,
>> +.devs_max   = 0,
>> +.devs_min   = 2,
>> +.tolerated_failures = 0,
>> +.devs_increment = 1,
>> +.ncopies= 1,
>> +},
>> +[BTRFS_RAID_SINGLE] = {
>> +.sub_stripes= 1,
>> +.dev_stripes= 1,
>> +.devs_max   = 1,
>> +.devs_min   = 1,
>> +.tolerated_failures = 0,
>> +.devs_increment = 1,
>> +.ncopies= 1,
>> +},
>> +[BTRFS_RAID_RAID5] = {
>> +.sub_stripes= 1,
>> +.dev_stripes= 1,
>> +.devs_max   = 0,
>> +.devs_min   = 2,
>> +.tolerated_failures = 1,
>> +.devs_increment = 1,
>> +.ncopies= 2,
>> +},
>> +[BTRFS_RAID_RAID6] = {
>> +.sub_stripes= 1,
>> +.dev_stripes= 1,
>> +.devs_max   = 0,
>> +.devs_min   = 3,
>> +.tolerated_failures = 2,
>> +.devs_increment = 1,
>> +.ncopies= 3,
>> +},
>> +};
>> +
>>  struct stripe {
>>  struct btrfs_device *dev;
>>  u64 physical;
>> diff --git a/volumes.h b/volumes.h
>> index 7bbdf615d31a..612a0a7586f4 100644
>> --- a/volumes.h
>> +++ b/volumes.h
>> @@ -108,6 +108,36 @@ struct map_lookup {
>>  struct btrfs_bio_stripe stripes[];
>>  };
>>  
>> +struct btrfs_raid_attr {
>> +int sub_stripes;/* sub_stripes info for map */
>> +int dev_stripes;/* stripes per dev */
>> +int devs_max;   /* max devs to use */
>> +int devs_min;   /* min devs needed */
>> +int tolerated_failures; /* max tolerated fail devs */
>> +int devs_increment; /* ndevs has to be a multiple of this */
>> +int ncopies;/* 

Re: [PATCH 2/2] btrfs: add read_mirror_policy parameter devid

[Austin S. Hemmelgarn]

On 2018-02-01 18:46, Edmund Nadolski wrote:



On 02/01/2018 01:12 AM, Anand Jain wrote:



On 02/01/2018 01:26 PM, Edmund Nadolski wrote:

On 1/31/18 7:36 AM, Anand Jain wrote:



On 01/31/2018 09:42 PM, Nikolay Borisov wrote:



So usually this should be functionality handled by the raid/san
controller I guess, > but given that btrfs is playing the role of a
controller here at what point are we drawing the line of not
implementing block-level functionality into the filesystem ?


Don't worry this is not invading into the block layer. How
can you even build this functionality in the block layer ?
Block layer even won't know that disks are mirrored. RAID
does or BTRFS in our case.



By block layer I guess I meant the storage driver of a particular raid
card. Because what is currently happening is re-implementing
functionality that will generally sit in the driver. So my question was
more generic and high-level - at what point do we draw the line of
implementing feature that are generally implemented in hardware devices
(be it their drivers or firmware).


   Not all HW configs use RAID capable HBAs. A server connected to a SATA
   JBOD using a SATA HBA without MD will relay on BTRFS to provide all
the
   features and capabilities that otherwise would have provided by such a
   presumable HW config.


That does sort of sound like means implementing some portion of the
HBA features/capabilities in the filesystem.

To me it seems this this could be workable at the fs level, provided it
deals just with policies and remains hardware-neutral.


  Thanks. Ok.


However most
of the use cases appear to involve some hardware-dependent knowledge
or assumptions.



What happens when someone sets this on a virtual disk,
or say a (persistent) memory-backed block device?


  Do you have any policy in particular ?


No, this is your proposal ;^)

You've said cases #3 thru #6 are illustrative only. However they make
assumptions about the underlying storage, and/or introduce potential for
unexpected behaviors. Plus they could end up replicating functionality
from other layers as Nikolay pointed out. Seems unlikely these would be
practical to implement.
The I/O one would actually be rather nice to have and wouldn't really be 
duplicating anything (at least, not duplicating anything we consistently 
run on top of).  The pid-based selector works fine for cases where the 
only thing on the disks is a single BTRFS filesystem.  When there's more 
than that, it can very easily result in highly asymmetrical load on the 
disks because it doesn't account for current I/O load when picking a 
copy to read.  Last I checked, both MD and DM-RAID have at least the 
option to use I/O load in determining where to send reads for RAID1 
setups, and they do a far better job than BTRFS at balancing load in 
these cases.


Case #2 seems concerning if it exposes internal,
implementation-dependent filesystem data into a de facto user-level
interface. (Do we ensure the devid is unique, and cannot get changed or
re-assigned internally to a different device, etc?)
The devid gets assigned when a device is added to a filesystem, it's a 
monotonically increasing number that gets incremented for every new 
device, and never changes for a given device as long as it remains in 
the filesystem (it will change if you remove the device and then re-add 
it).  The only exception to this is that the replace command will assign 
the new device the same devid that the device it is replacing had (which 
I would argue leads to consistent behavior here).  Given that, I think 
it's sufficiently safe to use it for something like this.

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RESEND PATCH] btrfs: Fix UAF when cleaning up fs_devs with a single stale device

[David Sterba]

On Tue, Jan 30, 2018 at 04:07:37PM +0200, Nikolay Borisov wrote:
> Commit 4fde46f0cc71 ("Btrfs: free the stale device") introduced
> btrfs_free_stale_device which iterates the device lists for all
> registered btrfs filesystems and deletes those devices which aren't
> mounted. In a btrfs_devices structure has only 1 device attached to it
> and it is unused then btrfs_free_stale_devices will proceed to also
> free the btrfs_fs_devices struct itself. Currently this leads to a UAF
> since list_for_each_entry will try to perform a check on the already-
> freed memory to see if it has to terminated the loop.
> 
> The fix is to use 'break' when we know we are freeing the current
> fs_devs.
> 
> Fixes: 4fde46f0cc71 ("Btrfs: free the stale device")
> Signed-off-by: Nikolay Borisov 

Added to next, thanks.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] btrfs: Remove btrfs_inode::delayed_iput_count

[David Sterba]

On Tue, Jan 16, 2018 at 09:31:58AM +0200, Nikolay Borisov wrote:
> delayed_iput_count wa supposed to be used to implement, well, delayed
> iput. The idea is that we keep accumulating the number of iputs we do
> until eventually the inode is deleted. Turns out we never really
> switched the delayed_iput_count from 0 to 1, hence all conditional
> code relying on the value of that member being different than 0 was
> never executed. This, as it turns out, didn't cause any problem due
> to the simple fact that the generic inode's i_count member was always
> used to count the number of iputs. So let's just remove the unused
> member and all unused code. This patch essentially provides no
> functional changes. While at it, also add proper documentation for 
> btrfs_add_delayed_iput 
> 
> Signed-off-by: Nikolay Borisov 

Reviewed-by: David Sterba 
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 2/2] btrfs: Remove custom crc32c init code

[David Sterba]

On Mon, Jan 08, 2018 at 11:45:05AM +0200, Nikolay Borisov wrote:
> The custom crc32 init code was introduced in
> 14a958e678cd ("Btrfs: fix btrfs boot when compiled as built-in") to
> enable using btrfs as a built-in. However, later as pointed out by
> 60efa5eb2e88 ("Btrfs: use late_initcall instead of module_init") this
> wasn't enough and finally btrfs was switched to late_initcall which
> comes after the generic crc32c implementation is initiliased. The
> latter commit superseeded the former. Now that we don't have to
> maintain our own code let's just remove it and switch to using the
> generic implementation.
> 
> Signed-off-by: Nikolay Borisov 
> ---
> 
> Despite touching a lot of files the patch is really simple. Here is the gist 
> of the changes: 
> 
> 1. Select LIBCRC32C rather than the low-level modules. 
> 2. s/btrfs_crc32c/crc32c/g
> 3. replace hash.h with linux/crc32c.h
> 4. Move the btrfs namehash funcs to ctree.h and change the tree accordingly. 
> 
> I've tested this with btrfs being both a module and a built-in and xfstest
> doesn't complain. 

This text could be also part of the changelog, it's useful to understand
how the change is done.

So this patches looks good to me and does seem to fix the longstanding
problem of not automatically selectiong the crc32c module when btrfs is
used. IIRC this has some workaround in dracut.

The modinfo confirms that now all the module dependencies are there:

before:
depends:zstd_compress,zstd_decompress,raid6_pq,xor,zlib_deflate

after:
depends:
libcrc32c,zstd_compress,zstd_decompress,raid6_pq,xor,zlib_deflate
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH RESEND 1/4] btrfs: Remove userspace transaction ioctls

[David Sterba]

On Wed, Jan 10, 2018 at 06:32:10PM +0200, Nikolay Borisov wrote:
> Commit 3558d4f88ec8 ("btrfs: Deprecate userspace transaction ioctls")
> marked the beginning of the end of userspace transaction. This commit
> finishes the job!
> 
> Signed-off-by: Nikolay Borisov 

This does not compile without the other patches, the last use of
btrfs_ioctl_trans_end is in patch 3/4. All patches should compile and
work separately, not just as the whole series.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[josef-btrfs:current-work 3/3] block/blk-wbt.c:919:1-10: alloc with no test, possible model on line 924

[kbuild test robot]

tree:   https://git.kernel.org/pub/scm/linux/kernel/git/josef/btrfs-next.git 
current-work
head:   1ac686f6d1b04b480cd245fb927180e1238bd3ac
commit: 1ac686f6d1b04b480cd245fb927180e1238bd3ac [3/3] current-work


coccinelle warnings: (new ones prefixed by >>)

>> block/blk-wbt.c:919:1-10: alloc with no test, possible model on line 924

vim +919 block/blk-wbt.c

   910  
   911  static int blkcg_qos_init(struct request_queue *q)
   912  {
   913  struct blkcg_qos *blkcg_qos;
   914  struct rq_qos *rqos;
   915  int i;
   916  
   917  BUILD_BUG_ON(WBT_NR_BITS > BLK_STAT_RES_BITS);
   918  
 > 919  blkcg_qos = kzalloc(sizeof(*blkcg_qos), GFP_KERNEL);
   920  if (!rwb)
   921  return -ENOMEM;
   922  
   923  rqos = _qos->rqos;
 > 924  rqos->cb = blk_stat_alloc_callback(qos_timer_fn, wbt_data_dir, 
 > 2,
   925 blkcg_qos);
   926  if (!rqos->cb) {
   927  kfree(blkcg_qos);
   928  return -ENOMEM;
   929  }
   930  
   931  rq_qos_add(q, rqos);
   932  blk_stat_add_callback(q, rqos->cb);
   933  atomic_set(_qos->scale_cookie, DEFAULT_SCALE_COOKIE);
   934  blkcg_qos->scale_lat = 0;
   935  #if 0
   936  for (i = 0; i < WBT_NUM_RWQ; i++) {
   937  atomic_set(>rq_wait[i].inflight, 0);
   938  init_waitqueue_head(>rq_wait[i].wait);
   939  }
   940  
   941  rwb->enable_state = WBT_STATE_ON_DEFAULT;
   942  wbt_update_limits(rwb);
   943  
   944  /*
   945   * Assign rwb and add the stats callback.
   946   */
   947  if (q->rq_wb)
   948  q->rq_wb->next = rwb;
   949  else (q->rq_wb)
   950  q->rq_wb = rwb;
   951  blk_stat_add_callback(q, rwb->cb);
   952  
   953  rwb->min_lat_nsec = wbt_default_latency_nsec(q);
   954  
   955  wbt_set_queue_depth(rwb, blk_queue_depth(q));
   956  wbt_set_write_cache(rwb, test_bit(QUEUE_FLAG_WC, 
>queue_flags));
   957  #endif
   958  return 0;
   959  }
   960  

---
0-DAY kernel test infrastructureOpen Source Technology Center
https://lists.01.org/pipermail/kbuild-all   Intel Corporation
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH RESEND 3/4] btrfs: Remove transaction handle from btrfs_file_private

[David Sterba]

On Wed, Jan 10, 2018 at 06:32:12PM +0200, Nikolay Borisov wrote:
> Since we no longer support userspace transaction there is no need to
> keep this member variable, so remove it.
> 
> Signed-off-by: Nikolay Borisov 

Reviewed-by: David Sterba 
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] btrfs: print error if primary super block write fails

[David Sterba]

On Fri, Jan 26, 2018 at 02:47:09PM -0800, Howard McLauchlan wrote:
> Presently, failing a primary super block write but succeeding in at
> least one super block write in general will appear to users as if
> nothing important went wrong. However, upon unmounting and re-mounting,
> the file system will be in a rolled back state. This was discovered
> with a BCC program that uses bpf_override_return() to fail super block
> writes.
> 
> This patch outputs an error clarifying that the primary super block
> write has failed, so users can expect potentially erroneous behaviour.
> It also forces wait_dev_supers() to return an error to its caller if
> the primary super block write fails.
> 
> Signed-off-by: Howard McLauchlan 
> ---
>  fs/btrfs/disk-io.c | 17 ++---
>  1 file changed, 14 insertions(+), 3 deletions(-)
> 
> diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c
> index 5da18ebc9222..8f96e1e4c613 100644
> --- a/fs/btrfs/disk-io.c
> +++ b/fs/btrfs/disk-io.c
> @@ -3293,11 +3293,13 @@ static int write_dev_supers(struct btrfs_device 
> *device,
>   * Return number of errors when buffer head is not found or not marked up to
>   * date.
>   */
> -static int wait_dev_supers(struct btrfs_device *device, int max_mirrors)
> +static int wait_dev_supers(struct btrfs_fs_info *fs_info,
> +struct btrfs_device *device, int max_mirrors)
>  {
>   struct buffer_head *bh;
>   int i;
>   int errors = 0;
> + bool primary_failed = false;
>   u64 bytenr;
>  
>   if (max_mirrors == 0)
> @@ -3314,11 +3316,14 @@ static int wait_dev_supers(struct btrfs_device 
> *device, int max_mirrors)
> BTRFS_SUPER_INFO_SIZE);
>   if (!bh) {
>   errors++;
> + primary_failed = (i == 0) || primary_failed;

if (i == 0)
primary_failed = true;

looks more readable to me.

>   continue;
>   }
>   wait_on_buffer(bh);
> - if (!buffer_uptodate(bh))
> + if (!buffer_uptodate(bh)) {
>   errors++;
> + primary_failed = (i == 0) || primary_failed;
> + }
>  
>   /* drop our reference */
>   brelse(bh);
> @@ -3327,6 +3332,12 @@ static int wait_dev_supers(struct btrfs_device 
> *device, int max_mirrors)
>   brelse(bh);
>   }
>  
> + /* log error, force error return */
> + if (primary_failed) {
> + btrfs_err(fs_info, "error encountered writing primary super 
> block");

As Qu pointed out, the device id is desired here. The errno is probably
-EIO in all cases, so does not need to be part of the message.

> + return -1;
> + }
> +
>   return errors < i ? 0 : -1;
>  }
>  
> @@ -3557,7 +3568,7 @@ int write_all_supers(struct btrfs_fs_info *fs_info, int 
> max_mirrors)
>   if (!dev->in_fs_metadata || !dev->writeable)
>   continue;
>  
> - ret = wait_dev_supers(dev, max_mirrors);
> + ret = wait_dev_supers(fs_info, dev, max_mirrors);
>   if (ret)
>   total_errors++;
>   }
> -- 
> 2.14.1
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH RESEND 1/4] btrfs: Remove userspace transaction ioctls

[David Sterba]

On Wed, Jan 10, 2018 at 06:32:10PM +0200, Nikolay Borisov wrote:
> Commit 3558d4f88ec8 ("btrfs: Deprecate userspace transaction ioctls")
> marked the beginning of the end of userspace transaction. This commit
> finishes the job!
> 
> Signed-off-by: Nikolay Borisov 

Reviewed-by: David Sterba 
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH RESEND 4/4] btrfs: Remove btrfs_fs_info::open_ioctl_trans

[David Sterba]

On Wed, Jan 10, 2018 at 06:32:13PM +0200, Nikolay Borisov wrote:
> Since userspace transaction have been removed we no longer have use
> for this field so delete it.
> 
> Signed-off-by: Nikolay Borisov 

Reviewed-by: David Sterba 
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] btrfs: Streamline btrfs_delalloc_reserve_metadata initial operations

[David Sterba]

On Tue, Jan 30, 2018 at 04:47:54PM +0200, Nikolay Borisov wrote:
> On 30.01.2018 16:34, David Sterba wrote:
> > On Fri, Jan 12, 2018 at 04:21:05PM +0200, Nikolay Borisov wrote:
> >> @@ -6062,19 +6062,19 @@ int btrfs_delalloc_reserve_metadata(struct 
> >> btrfs_inode *inode, u64 num_bytes)
> >> * If we have a transaction open (can happen if we call truncate_block
> >> * from truncate), then we need FLUSH_LIMIT so we don't deadlock.
> >> */
> >> +
> >>if (btrfs_is_free_space_inode(inode)) {
> >>flush = BTRFS_RESERVE_NO_FLUSH;
> >>delalloc_lock = false;
> >> -  } else if (current->journal_info) {
> >> -  flush = BTRFS_RESERVE_FLUSH_LIMIT;
> >> -  }
> >> +  } else {
> >> +  if (current->journal_info)
> >> +  flush = BTRFS_RESERVE_FLUSH_LIMIT;
> >>  
> >> -  if (flush != BTRFS_RESERVE_NO_FLUSH &&
> >> -  btrfs_transaction_in_commit(fs_info))
> >> -  schedule_timeout(1);
> >> +  if (btrfs_transaction_in_commit(fs_info))
> >> +  schedule_timeout(1);
> >>  
> >> -  if (delalloc_lock)
> >>mutex_lock(>delalloc_mutex);
> >> +  }
> > 
> > Squeezing the condition branches makes the code more readable, I have
> > only one objection and it's the mutex_lock. It IMHO looks better when
> > it's a separate branch as it pairs with the unlock:
> > 
> > if (delalloc_lock)
> > mutex_lock(...);
> > 
> > ...
> > 
> > if (delalloc_lock)
> > mutex_unlock(...);
> > 
> > In your version it's implied by the first if that checks
> > btrfs_is_free_space_inode and delalloc_lock is hidden there.
> > 
> 
> My line of thought when developing the patch was that delalloc is
> another level of indirection and. What I wanted to achieve in the end is
> to make it clear that delalloc_mutex really depends on whether we are
> reserving for the freespace inode or not.

Well, I almost overlooked the mutex on first top-down reading the code.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC PATCH v2 2/4] btrfs-progs: Add EXEC represent path of executable file

[David Sterba]

On Fri, Feb 02, 2018 at 04:34:03PM +0800, Gu Jinxiang wrote:
> Use EXEC instead of TOP to represent the path of excutable file.
> EXEC is set to TOP by default, but when there is no excutable file
> in TOP, use the path where btrfs is install as EXEC.

What if we just allow to change TOP (ie. do not overwrite it in the test
driver scripts)? The logic will be the same as with EXEC, but we won't
have to rewrite essentailly all paths in the testsuite.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/2] libcrc32c: Add crc32c_impl function

[David Sterba]

Adding linux-cry...@vger.kernel.org to CC

Link to the 2/2 patch https://patchwork.kernel.org/patch/10149203/

On Mon, Jan 08, 2018 at 11:45:04AM +0200, Nikolay Borisov wrote:
> This function returns a string with the currently in-use implementation
> of the crc32c algorithm, i.e crc32c-generic (for unoptimised, generic
> implementation) or crc32c-intel for the sse optimised version. This
> will be used by btrfs.
> 
> Signed-off-by: Nikolay Borisov 
> ---
>  include/linux/crc32c.h | 1 +
>  lib/libcrc32c.c| 6 ++
>  2 files changed, 7 insertions(+)
> 
> diff --git a/include/linux/crc32c.h b/include/linux/crc32c.h
> index 357ae4611a45..bd21af828ff6 100644
> --- a/include/linux/crc32c.h
> +++ b/include/linux/crc32c.h
> @@ -5,6 +5,7 @@
>  #include 
>  
>  extern u32 crc32c(u32 crc, const void *address, unsigned int length);
> +extern const char *crc32c_impl(void);
>  
>  /* This macro exists for backwards-compatibility. */
>  #define crc32c_le crc32c
> diff --git a/lib/libcrc32c.c b/lib/libcrc32c.c
> index 9f79547d1b97..eaf71e0e04be 100644
> --- a/lib/libcrc32c.c
> +++ b/lib/libcrc32c.c
> @@ -71,6 +71,12 @@ static void __exit libcrc32c_mod_fini(void)
>   crypto_free_shash(tfm);
>  }
>  
> +const char *crc32c_impl(void)
> +{
> + return crypto_tfm_alg_driver_name(crypto_shash_tfm(tfm));
> +}
> +EXPORT_SYMBOL(crc32c_impl);

Crypto maintainers, would it be still possible to squeeze this patch to
4.16? It's quite trivial, but as it is not in a code I maintain I'm not
comfortable to add it to my tree (unless I get an ACK).

The linked patch depends on that change and would let us get rid of some
custom crypto wrappers around crc32c.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH RESEND 2/4] btrfs: Remove code referencing unused TRANS_USERSPACE

[David Sterba]

On Wed, Jan 10, 2018 at 06:32:11PM +0200, Nikolay Borisov wrote:
> Now that the userspace transaction ioctls have been removed,
> TRANS_USERSPACE is no longer used hence we can remove it.
> 
> Signed-off-by: Nikolay Borisov 

Reviewed-by: David Sterba 

> -struct btrfs_trans_handle *btrfs_start_ioctl_transaction(struct btrfs_root 
> *root)

The declaration removal from transaction.h is missing, I'll fix that.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC PATCH v2 3/4] btrfs-progs: Modify rootdir files when use mkfs.btrfs

[David Sterba]

On Fri, Feb 02, 2018 at 04:34:04PM +0800, Gu Jinxiang wrote:
> Since there is no $TOP/Documentation file after package the testsuite.
> So use a common file that is usually exsits.
> 
> Signed-off-by: Gu Jinxiang 
> ---
>  tests/misc-tests/002-uuid-rewrite/test.sh   | 4 ++--
>  tests/misc-tests/003-zero-log/test.sh   | 2 +-
>  tests/mkfs-tests/004-rootdir-keeps-size/test.sh | 2 +-
>  3 files changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/tests/misc-tests/002-uuid-rewrite/test.sh 
> b/tests/misc-tests/002-uuid-rewrite/test.sh
> index 158bda90..27341322 100755
> --- a/tests/misc-tests/002-uuid-rewrite/test.sh
> +++ b/tests/misc-tests/002-uuid-rewrite/test.sh
> @@ -25,7 +25,7 @@ test_uuid_random()
>  
>   run_check $SUDO_HELPER $EXEC/mkfs.btrfs -f \
>   --uuid $origuuid \
> - --rootdir $TOP/Documentation \
> + --rootdir /lib/modules/`uname -r`/ \

As an easy source of filesystem data it's ok, maybe in the log term
something more deterministic should be implemented. The size of the
modules is unpredictable and may not fit on the filesystem and the test
would fail.

The size of the Documentation/ is 388KiB on my system, the modules are
+260MiB, that also makes the test run longer for no reason.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3] btrfs: print error if primary super block write fails

[Howard McLauchlan]

On 02/02/2018 11:09 AM, Howard McLauchlan wrote:
> Presently, failing a primary super block write but succeeding in at
> least one super block write in general will appear to users as if
> nothing important went wrong. However, upon unmounting and re-mounting,
> the file system will be in a rolled back state. This was discovered
> with a BCC program that uses bpf_override_return() to fail super block
> writes.
> 
> This patch outputs an error clarifying that the primary super block
> write has failed, so users can expect potentially erroneous behaviour.
> It also forces wait_dev_supers() to return an error to its caller if
> the primary super block write fails.
> 
> Signed-off-by: Howard McLauchlan 
Reviewed-by: Qu Wenruo 
> ---
> V3: Rewrote boolean setting logic for clarity
> V2: Added devid to output, removed unnecessary fs_info parameter
> 
>  fs/btrfs/disk-io.c | 15 ++-
>  1 file changed, 14 insertions(+), 1 deletion(-)
> 
> diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c
> index 5da18ebc9222..6ef32eb94669 100644
> --- a/fs/btrfs/disk-io.c
> +++ b/fs/btrfs/disk-io.c
> @@ -3298,6 +3298,7 @@ static int wait_dev_supers(struct btrfs_device *device, 
> int max_mirrors)
>   struct buffer_head *bh;
>   int i;
>   int errors = 0;
> + bool primary_failed = false;
>   u64 bytenr;
>  
>   if (max_mirrors == 0)
> @@ -3314,11 +3315,16 @@ static int wait_dev_supers(struct btrfs_device 
> *device, int max_mirrors)
> BTRFS_SUPER_INFO_SIZE);
>   if (!bh) {
>   errors++;
> + if (i == 0)
> + primary_failed = true;
>   continue;
>   }
>   wait_on_buffer(bh);
> - if (!buffer_uptodate(bh))
> + if (!buffer_uptodate(bh)) {
>   errors++;
> + if (i == 0)
> + primary_failed = true;
> + }
>  
>   /* drop our reference */
>   brelse(bh);
> @@ -3327,6 +,13 @@ static int wait_dev_supers(struct btrfs_device 
> *device, int max_mirrors)
>   brelse(bh);
>   }
>  
> + /* log error, force error return */
> + if (primary_failed) {
> + btrfs_err(device->fs_info, "error writing primary super block 
> to device %llu",
> +   device->devid);
> + return -1;
> + }
> +
>   return errors < i ? 0 : -1;
>  }
>  
> 
Forgot to add Qu's reviewed by
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 2/2] btrfs: Remove custom crc32c init code

[Andy Shevchenko]

On Mon, Jan 8, 2018 at 11:45 AM, Nikolay Borisov  wrote:
> The custom crc32 init code was introduced in
> 14a958e678cd ("Btrfs: fix btrfs boot when compiled as built-in") to
> enable using btrfs as a built-in. However, later as pointed out by
> 60efa5eb2e88 ("Btrfs: use late_initcall instead of module_init") this
> wasn't enough and finally btrfs was switched to late_initcall which
> comes after the generic crc32c implementation is initiliased. The
> latter commit superseeded the former. Now that we don't have to
> maintain our own code let's just remove it and switch to using the
> generic implementation.

>  fs/btrfs/hash.c| 54 
> --
>  fs/btrfs/hash.h| 43 

IIRC Adding -D to git format-patch will make it shorter and nicer.
But please, double check that git am actually detects removals.

-- 
With Best Regards,
Andy Shevchenko
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC PATCH v2 0/4] Add support for export testsuits

[David Sterba]

On Fri, Feb 02, 2018 at 04:34:01PM +0800, Gu Jinxiang wrote:
> Achieved:
> 1. export testsuit by:
>  $ make EXPORT=/where/you/want/to/generate/tests.tar.gz package
> relative path and absolute path both be ok.
> Besides tests itself, fssum and btrfs-corrupt-block will also be
> included in tests.tar.gz, since misc and fsck tests depend on
> btrfs-corrupt-block, and misc tests depend on fssum. 

My idea was:

$ make testuite

that will build all dependencies, ie. fssum and btrfs-corrupt-block, and
create btrfs-progs-tests.tar.gz in the current directory, ie. no need
for the EXPORT variable.

The tar command you used would add anything that's under tessts/ but we
don't need everything. So I'd expect there's a file that lists all files
and directories that we want in the testsuite tarball.

Some sort of identification should be put to the tarball, eg. what git
describe says and timestamp of generation.

> 2. after decompress tests.tar.gz, run test by:
>  $ TEST=`MASK` ./mkfs-tests.sh
> and, without MASK also be ok.
> replenish:
> The directory Structure after decompress tests.tar.gz is:
>  $ tar -xzvf ./tests.tar.gz
>  $ ls
>tests fssum btrfs-corrupt-block

The structure could be reduced by one directory level if the contents of
tests is moved one directory up. That way all the test driver scritps
are in the same place as fssum and btrfs-corrupt-block. This would need
to be reflected in the test scripts.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 04/26] libbtrfsutil: add btrfs_util_is_subvolume() and btrfs_util_subvolume_id()

[Omar Sandoval]

On Thu, Feb 01, 2018 at 05:28:28PM +0100, David Sterba wrote:
> On Tue, Jan 30, 2018 at 08:54:08AM +0200, Nikolay Borisov wrote:
> > 
> > 
> > On 29.01.2018 23:43, Omar Sandoval wrote:
> > > On Mon, Jan 29, 2018 at 12:24:26PM +0200, Nikolay Borisov wrote:
> > >> On 26.01.2018 20:40, Omar Sandoval wrote:
> > > [snip]
> > >>> +/*
> > >>> + * This intentionally duplicates btrfs_util_f_is_subvolume() instead 
> > >>> of opening
> > >>> + * a file descriptor and calling it, because fstat() and fstatfs() 
> > >>> don't accept
> > >>> + * file descriptors opened with O_PATH on old kernels (before v3.6 and 
> > >>> before
> > >>> + * v3.12, respectively), but stat() and statfs() can be called on a 
> > >>> path that
> > >>> + * the user doesn't have read or write permissions to.
> > >>> + */
> > >>> +__attribute__((visibility("default")))
> > >>
> > >> Why do we need to explicitly set the attribute visibility to default,
> > >> isn't it implicitly default already?
> > > 
> > > Ah, I forgot to add -fvisibility=hidden to the build rule when I ported
> > > this to the btrfs-progs Makefile, that's why that's there. I'll add
> > > -fvisibility=hidden to the Makefile.
> > 
> > Right, it could be a good idea to hide the visibility attribute behind
> > an eloquent macro i.e. (PUBLIC|LIBRARY)_FUNC or some such.
> 
> Macro would be better (but is not needed for the initial version).
> 
> Alternatively the library .sym file can externally track the exported
> symbols and also track versioning.

I'll add a macro for this, thanks.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[josef-btrfs:current-work 3/3] block/blk-wbt.h:96:1: sparse: two or more data types in declaration specifiers

[kbuild test robot]

tree:   https://git.kernel.org/pub/scm/linux/kernel/git/josef/btrfs-next.git 
current-work
head:   1ac686f6d1b04b480cd245fb927180e1238bd3ac
commit: 1ac686f6d1b04b480cd245fb927180e1238bd3ac [3/3] current-work
reproduce:
# apt-get install sparse
git checkout 1ac686f6d1b04b480cd245fb927180e1238bd3ac
make ARCH=x86_64 allmodconfig
make C=1 CF=-D__CHECK_ENDIAN__


sparse warnings: (new ones prefixed by >>)

>> block/blk-wbt.h:96:1: sparse: two or more data types in declaration 
>> specifiers
>> block/blk-wbt.h:96:1: sparse: Trying to use reserved word 'struct' as 
>> identifier
   block/blk-wbt.h:96:8: sparse: Expected ; at end of declaration
   block/blk-wbt.h:96:8: sparse: got rq_qos
   block/blk-wbt.h:100:1: sparse: Expected ; at the end of type declaration
   block/blk-wbt.h:100:1: sparse: got }
   In file included from block/elevator.c:44:0:
   block/blk-wbt.h:89:36: warning: 'struct rq_qos' declared inside parameter 
list will not be visible outside of this definition or declaration
enum wbt_flags rq_qos struct bio
^~
   block/blk-wbt.h:91:23: warning: 'struct rq_qos' declared inside parameter 
list will not be visible outside of this definition or declaration
void rq_qos struct blk_issue_stat
^~
   block/blk-wbt.h:92:25: warning: 'struct rq_qos' declared inside parameter 
list will not be visible outside of this definition or declaration
void rq_qos struct blk_issue_stat
^~
   block/blk-wbt.h:93:26: warning: 'struct rq_qos' declared inside parameter 
list will not be visible outside of this definition or declaration
void rq_qos struct blk_issue_stat
^~
   block/blk-wbt.h:96:1: error: expected ';', identifier or '(' before 'struct'
struct rq_qos {
^~
   block/blk-wbt.h:123:8: error: redefinition of 'struct rq_qos'
struct rq_qos {
^~
   block/blk-wbt.h:96:8: note: originally defined here
struct rq_qos {
^~
   block/blk-wbt.h: In function 'rq_qos_add':
   block/blk-wbt.h:169:16: error: 'struct request_queue' has no member named 
'rqos'
rqos->next = q->rqos;
^~
   block/blk-wbt.h:170:3: error: 'struct request_queue' has no member named 
'rqos'
q->rqos = rqos;
^~
--
>> block/blk-wbt.h:96:1: sparse: two or more data types in declaration 
>> specifiers
>> block/blk-wbt.h:96:1: sparse: Trying to use reserved word 'struct' as 
>> identifier
   block/blk-wbt.h:96:8: sparse: Expected ; at end of declaration
   block/blk-wbt.h:96:8: sparse: got rq_qos
   block/blk-wbt.h:100:1: sparse: Expected ; at the end of type declaration
   block/blk-wbt.h:100:1: sparse: got }
   block/blk-sysfs.c:432:56: sparse: no member 'min_lat_nsec' in struct rq_wb
   block/blk-sysfs.c:457:20: sparse: no member 'min_lat_nsec' in struct rq_wb
   block/blk-sysfs.c:459:20: sparse: no member 'min_lat_nsec' in struct rq_wb
   In file included from block/blk-sysfs.c:18:0:
   block/blk-wbt.h:89:36: warning: 'struct rq_qos' declared inside parameter 
list will not be visible outside of this definition or declaration
enum wbt_flags rq_qos struct bio
^~
   block/blk-wbt.h:91:23: warning: 'struct rq_qos' declared inside parameter 
list will not be visible outside of this definition or declaration
void rq_qos struct blk_issue_stat
^~
   block/blk-wbt.h:92:25: warning: 'struct rq_qos' declared inside parameter 
list will not be visible outside of this definition or declaration
void rq_qos struct blk_issue_stat
^~
   block/blk-wbt.h:93:26: warning: 'struct rq_qos' declared inside parameter 
list will not be visible outside of this definition or declaration
void rq_qos struct blk_issue_stat
^~
   block/blk-wbt.h:96:1: error: expected ';', identifier or '(' before 'struct'
struct rq_qos {
^~
   block/blk-wbt.h:123:8: error: redefinition of 'struct rq_qos'
struct rq_qos {
^~
   block/blk-wbt.h:96:8: note: originally defined here
struct rq_qos {
^~
   block/blk-wbt.h: In function 'rq_qos_add':
   block/blk-wbt.h:169:16: error: 'struct request_queue' has no member named 
'rqos'
rqos->next = q->rqos;
^~
   block/blk-wbt.h:170:3: error: 'struct request_queue' has no member named 
'rqos'
q->rqos = rqos;
^~
   block/blk-sysfs.c: In function 'queue_wb_lat_show':
   block/blk-sysfs.c:432:49: error: 'struct rq_wb' has no member named 
'min_lat_nsec'
return sprintf(page, "%llun", div_u64(q->rq_wb->min_lat_nsec, 1000));
^~
   block/blk-sysfs.c: In function 'queue_wb_lat_store':
   block/blk-sysfs.c:457:6: error: 'struct rq_wb' has no member named 
'min_lat_nsec'
rwb->min_lat_nsec = wbt_default_latency_nsec(q);
^~
   block/blk-sysfs.c:459:6: error: 'struct rq_wb' has no member named 
'min_lat_nsec'
rwb->min_lat_nsec = val COPYING CREDITS Documentation Kbuild Kconfig 
MAINTAINERS Makefile README arch block certs crypto drivers firmware fs include 
init ipc kernel lib mm net samples scripts security sound tools usr virt 

[PATCH v3] btrfs: print error if primary super block write fails

[Howard McLauchlan]

Presently, failing a primary super block write but succeeding in at
least one super block write in general will appear to users as if
nothing important went wrong. However, upon unmounting and re-mounting,
the file system will be in a rolled back state. This was discovered
with a BCC program that uses bpf_override_return() to fail super block
writes.

This patch outputs an error clarifying that the primary super block
write has failed, so users can expect potentially erroneous behaviour.
It also forces wait_dev_supers() to return an error to its caller if
the primary super block write fails.

Signed-off-by: Howard McLauchlan 
---
V3: Rewrote boolean setting logic for clarity
V2: Added devid to output, removed unnecessary fs_info parameter

 fs/btrfs/disk-io.c | 15 ++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c
index 5da18ebc9222..6ef32eb94669 100644
--- a/fs/btrfs/disk-io.c
+++ b/fs/btrfs/disk-io.c
@@ -3298,6 +3298,7 @@ static int wait_dev_supers(struct btrfs_device *device, 
int max_mirrors)
struct buffer_head *bh;
int i;
int errors = 0;
+   bool primary_failed = false;
u64 bytenr;
 
if (max_mirrors == 0)
@@ -3314,11 +3315,16 @@ static int wait_dev_supers(struct btrfs_device *device, 
int max_mirrors)
  BTRFS_SUPER_INFO_SIZE);
if (!bh) {
errors++;
+   if (i == 0)
+   primary_failed = true;
continue;
}
wait_on_buffer(bh);
-   if (!buffer_uptodate(bh))
+   if (!buffer_uptodate(bh)) {
errors++;
+   if (i == 0)
+   primary_failed = true;
+   }
 
/* drop our reference */
brelse(bh);
@@ -3327,6 +,13 @@ static int wait_dev_supers(struct btrfs_device *device, 
int max_mirrors)
brelse(bh);
}
 
+   /* log error, force error return */
+   if (primary_failed) {
+   btrfs_err(device->fs_info, "error writing primary super block 
to device %llu",
+ device->devid);
+   return -1;
+   }
+
return errors < i ? 0 : -1;
 }
 
-- 
2.14.1

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] btrfs: Move qgroup rescan on quota enable to btrfs_quota_enable

[Qu Wenruo]

On 2018年01月31日 16:52, Nikolay Borisov wrote:
> Currently btrfs_run_qgroups is doing a bit too much. Not only is it
> responsible for synchronizing in-memory state of qgroups to disk but
> it also contains code to trigger the initial qgroup rescan when
> quota is enabled initially. This condition is detected by checking that
> BTRFS_FS_QUOTA_ENABLED is not set and BTRFS_FS_QUOTA_ENABLING is set.
> Nothing really requires fro the code to be structured (and scattered)
> the way it is so let's streamline things. First move the quota rescan
> code into btrfs_quota_enable, where its invocation is closer to the
> user. This also makes the FS_QUOTA_ENABLING flag redundant so let's
> remove it as well.i
> 
> This has been tested with a full xfstest run with qgroups enabled on
> the scratch device of every xfstest and no regressions were observed.
> 
> Signed-off-by: Nikolay Borisov 

Looks good.

Since rescan work is async, initializing it at enable looks pretty
reasonable.

No delaying makes it easier to read.

> ---
>  fs/btrfs/ctree.h  |  1 -
>  fs/btrfs/qgroup.c | 35 ++-
>  2 files changed, 10 insertions(+), 26 deletions(-)

And indeed reduces codes.

Reviewed-by: Qu Wenruo 

Thanks,
Qu

> 
> diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
> index 1a462ab85c49..83b645920aba 100644
> --- a/fs/btrfs/ctree.h
> +++ b/fs/btrfs/ctree.h
> @@ -707,7 +707,6 @@ struct btrfs_delayed_root;
>  #define BTRFS_FS_LOG_RECOVERING  4
>  #define BTRFS_FS_OPEN5
>  #define BTRFS_FS_QUOTA_ENABLED   6
> -#define BTRFS_FS_QUOTA_ENABLING  7
>  #define BTRFS_FS_UPDATE_UUID_TREE_GEN9
>  #define BTRFS_FS_CREATING_FREE_SPACE_TREE10
>  #define BTRFS_FS_BTREE_ERR   11
> diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c
> index f822152693a4..5b446d1bb22d 100644
> --- a/fs/btrfs/qgroup.c
> +++ b/fs/btrfs/qgroup.c
> @@ -826,10 +826,8 @@ int btrfs_quota_enable(struct btrfs_trans_handle *trans,
>   int slot;
>  
>   mutex_lock(_info->qgroup_ioctl_lock);
> - if (fs_info->quota_root) {
> - set_bit(BTRFS_FS_QUOTA_ENABLING, _info->flags);
> + if (fs_info->quota_root)
>   goto out;
> - }
>  
>   fs_info->qgroup_ulist = ulist_alloc(GFP_KERNEL);
>   if (!fs_info->qgroup_ulist) {
> @@ -923,8 +921,15 @@ int btrfs_quota_enable(struct btrfs_trans_handle *trans,
>   }
>   spin_lock(_info->qgroup_lock);
>   fs_info->quota_root = quota_root;
> - set_bit(BTRFS_FS_QUOTA_ENABLING, _info->flags);
> + set_bit(BTRFS_FS_QUOTA_ENABLED, _info->flags);
>   spin_unlock(_info->qgroup_lock);
> + ret = qgroup_rescan_init(fs_info, 0, 1);
> + if (!ret) {
> + qgroup_rescan_zero_tracking(fs_info);
> + btrfs_queue_work(fs_info->qgroup_rescan_workers,
> +  _info->qgroup_rescan_work);
> + }
> +
>  out_free_path:
>   btrfs_free_path(path);
>  out_free_root:
> @@ -2077,17 +2082,9 @@ int btrfs_run_qgroups(struct btrfs_trans_handle *trans,
>  {
>   struct btrfs_root *quota_root = fs_info->quota_root;
>   int ret = 0;
> - int start_rescan_worker = 0;
>  
>   if (!quota_root)
> - goto out;
> -
> - if (!test_bit(BTRFS_FS_QUOTA_ENABLED, _info->flags) &&
> - test_bit(BTRFS_FS_QUOTA_ENABLING, _info->flags))
> - start_rescan_worker = 1;
> -
> - if (test_and_clear_bit(BTRFS_FS_QUOTA_ENABLING, _info->flags))
> - set_bit(BTRFS_FS_QUOTA_ENABLED, _info->flags);
> + return ret;
>  
>   spin_lock(_info->qgroup_lock);
>   while (!list_empty(_info->dirty_qgroups)) {
> @@ -2116,18 +2113,6 @@ int btrfs_run_qgroups(struct btrfs_trans_handle *trans,
>   if (ret)
>   fs_info->qgroup_flags |= BTRFS_QGROUP_STATUS_FLAG_INCONSISTENT;
>  
> - if (!ret && start_rescan_worker) {
> - ret = qgroup_rescan_init(fs_info, 0, 1);
> - if (!ret) {
> - qgroup_rescan_zero_tracking(fs_info);
> - btrfs_queue_work(fs_info->qgroup_rescan_workers,
> -  _info->qgroup_rescan_work);
> - }
> - ret = 0;
> - }
> -
> -out:
> -
>   return ret;
>  }
>  
> 



signature.asc
Description: OpenPGP digital signature

Re: [PATCH 00/26] btrfs-progs: introduce libbtrfsutil, "btrfs-progs as a library"

[Hans van Kranenburg]

Hi,

On 01/26/2018 07:51 PM, Hugo Mills wrote:
> On Fri, Jan 26, 2018 at 10:40:48AM -0800, Omar Sandoval wrote:
>> From: Omar Sandoval 
>>
>> One of the features requests I get most often is a library to do the
>> sorts of operations that we do with btrfs-progs. We can shell out to
>> btrfs-progs, but the output format isn't always easily parsasble, and
>> shelling out isn't always ideal. There's libbtrfs, but it's very
>> incomplete, doesn't have a well thought out API, and is licensed under
>> the GPL, making it hard to use for many projects.
>>
>> libbtrfsutil is a new library written from scratch to address these
>> issues. The implementation is completely independent of the existing
>> btrfs-progs code, including kerncompat.h, and has a clean API and naming
>> scheme. It is licensed under the LGPL. It also includes Python bindings
>> by default. I will maintain the library code.

Insert mandatory picture here:

https://memegenerator.net/img/instances/12076366/python-all-the-things.jpg

>*speechless*
> 
>That's awesome, Omar (although with the python bindings, you've
> probably just ruined Hans' day ;) ).

Not quite, or should I say, quite the opposite, actually.

I can think of a few different python libs related to btrfs right now...

1. This one, which works on the same level as the btrfs-progs command
line utils and provides the same set of functionality, with the same
input and output, only way easier to handle when automating things. The
audience is the same audience as who would else try to 'shell out' to
btrfs-progs and then do painful parsing of the output.

2. A lib for advanced btrfs users (more of a niche audience) who want to
get a peek behind the curtains of what is going on in their online
filesystem. Directly do searches in metadata, directly call an ioctl
etc, either for educational purposes, or to e.g. do things like write a
small special purpose dedupe program, visualize things etc. This is the
python3-btrfs lib that I've been working on for some time now.

3. A lib for btrfs developers who want to script working on offline
fileystems. This one does not exist yet. Well, it's a bit of an unborn
baby now, my first brain dump:
  https://github.com/knorrie/btrfs-progs/blob/python/python3/README.md
This would be the lib that exposes the (to be revised and improved)
internal libbtrfs (I guess).

All of these are not competitors. They serve different purpose and a
different audience. So I'm happy to see number 1 also happening. :-)

>> Patch 1 is a preparation cleanup which can go in independently. Patch 2
>> adds the build system stuff for the library, and patch 3 does the same
>> for the Python bindings. Patches 4-14 implement the library helpers,
>> currently subvolume helpers and the sync ioctls. Patches 15-26 replace
>> the btrfs-progs and libbtrfs code to use libbtrfsutil instead. I took
>> care to preserve backwards-compatibility. `btrfs subvol list` in
>> particular had some buggy behaviors for -o and -a that I emulated in the
>> new code, see the comments in the code.

+1 for starting to clean up that code. Especially the whole subvolume
list part looks like a great improvement.

>> These patches are also available on my GitHub:
>> https://github.com/osandov/btrfs-progs/tree/libbtrfsutil. That branch
>> will rebase as I update this series.
>>
>> Please share feedback regarding the API, implementation, or anything
>> else.

Great work, and I'm of course also interested in what you think about
the lib nr.3 idea, since I didn't throw it around on the list before
yet. :-)

-- 
Hans van Kranenburg
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 21/26] btrfs-progs: use libbtrfsutil for subvol show

[Omar Sandoval]

On Sat, Feb 03, 2018 at 12:18:02AM +0100, Hans van Kranenburg wrote:
> On 01/26/2018 07:41 PM, Omar Sandoval wrote:
> > From: Omar Sandoval 
> > 
> > Now implemented with btrfs_util_subvolume_path(),
> > btrfs_util_subvolume_info(), and subvolume iterators.
> > 
> > Signed-off-by: Omar Sandoval 
> > ---
> >  cmds-subvolume.c | 150 
> > ---
> >  utils.c  | 118 ---
> >  utils.h  |   5 --
> >  3 files changed, 99 insertions(+), 174 deletions(-)
> > 
> > diff --git a/cmds-subvolume.c b/cmds-subvolume.c
> > index c5e03011..b969fc88 100644
> > --- a/cmds-subvolume.c
> > +++ b/cmds-subvolume.c
> >
> >
> > [...]
> > } else
> > strcpy(tstr, "-");
> > printf("\tCreation time: \t\t%s\n", tstr);
> >  
> > -   printf("\tSubvolume ID: \t\t%llu\n", get_ri.root_id);
> > -   printf("\tGeneration: \t\t%llu\n", get_ri.gen);
> > -   printf("\tGen at creation: \t%llu\n", get_ri.ogen);
> > -   printf("\tParent ID: \t\t%llu\n", get_ri.ref_tree);
> > -   printf("\tTop level ID: \t\t%llu\n", get_ri.top_id);
> > +   printf("\tSubvolume ID: \t\t%" PRIu64 "\n", subvol.id);
> > +   printf("\tGeneration: \t\t%" PRIu64 "\n", subvol.generation);
> > +   printf("\tGen at creation: \t%" PRIu64 "\n", subvol.otransid);
> > +   printf("\tParent ID: \t\t%" PRIu64 "\n", subvol.parent_id);
> > +   printf("\tTop level ID: \t\t%" PRIu64 "\n", subvol.parent_id);
> >  
> > -   if (get_ri.flags & BTRFS_ROOT_SUBVOL_RDONLY)
> > +   if (subvol.flags & BTRFS_ROOT_SUBVOL_RDONLY)
> > printf("\tFlags: \t\t\treadonly\n");
> > else
> > printf("\tFlags: \t\t\t-\n");
> >  
> > /* print the snapshots of the given subvol if any*/
> > printf("\tSnapshot(s):\n");
> > -   filter_set = btrfs_list_alloc_filter_set();
> > -   btrfs_list_setup_filter(_set, BTRFS_LIST_FILTER_BY_PARENT,
> > -   (u64)(unsigned long)get_ri.uuid);
> > -   btrfs_list_setup_print_column(BTRFS_LIST_PATH);
> >  
> > -   fd = open_file_or_dir(fullpath, );
> > -   if (fd < 0) {
> > -   fprintf(stderr, "ERROR: can't access '%s'\n", fullpath);
> > -   goto out;
> > +   err = btrfs_util_f_create_subvolume_iterator(fd,
> > +BTRFS_FS_TREE_OBJECTID,
> > +0, );
> > +
> > [...]
> When you have enough subvolumes in a filesystem, let's say 10 (yes,
> that sometimes happens), the current btrfs sub list is quite unusable,
> which is kind of expected. But, currently, sub show is also unusable
> because it still starts loading a list of all subvolumes to be able to
> print the 'snapshots, if any'.
> 
> So I guess that this new sub show will at least print the info first and
> then use the iterator and start crawling through the list, which can be
> interrupted? At least you get the relevant info first then. :-)

Right, and since we don't load everything into memory all at once, both
show and list will be able to output subvolumes one by one.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 21/26] btrfs-progs: use libbtrfsutil for subvol show

[Hans van Kranenburg]

On 01/26/2018 07:41 PM, Omar Sandoval wrote:
> From: Omar Sandoval 
> 
> Now implemented with btrfs_util_subvolume_path(),
> btrfs_util_subvolume_info(), and subvolume iterators.
> 
> Signed-off-by: Omar Sandoval 
> ---
>  cmds-subvolume.c | 150 
> ---
>  utils.c  | 118 ---
>  utils.h  |   5 --
>  3 files changed, 99 insertions(+), 174 deletions(-)
> 
> diff --git a/cmds-subvolume.c b/cmds-subvolume.c
> index c5e03011..b969fc88 100644
> --- a/cmds-subvolume.c
> +++ b/cmds-subvolume.c
>
>
> [...]
>   } else
>   strcpy(tstr, "-");
>   printf("\tCreation time: \t\t%s\n", tstr);
>  
> - printf("\tSubvolume ID: \t\t%llu\n", get_ri.root_id);
> - printf("\tGeneration: \t\t%llu\n", get_ri.gen);
> - printf("\tGen at creation: \t%llu\n", get_ri.ogen);
> - printf("\tParent ID: \t\t%llu\n", get_ri.ref_tree);
> - printf("\tTop level ID: \t\t%llu\n", get_ri.top_id);
> + printf("\tSubvolume ID: \t\t%" PRIu64 "\n", subvol.id);
> + printf("\tGeneration: \t\t%" PRIu64 "\n", subvol.generation);
> + printf("\tGen at creation: \t%" PRIu64 "\n", subvol.otransid);
> + printf("\tParent ID: \t\t%" PRIu64 "\n", subvol.parent_id);
> + printf("\tTop level ID: \t\t%" PRIu64 "\n", subvol.parent_id);
>  
> - if (get_ri.flags & BTRFS_ROOT_SUBVOL_RDONLY)
> + if (subvol.flags & BTRFS_ROOT_SUBVOL_RDONLY)
>   printf("\tFlags: \t\t\treadonly\n");
>   else
>   printf("\tFlags: \t\t\t-\n");
>  
>   /* print the snapshots of the given subvol if any*/
>   printf("\tSnapshot(s):\n");
> - filter_set = btrfs_list_alloc_filter_set();
> - btrfs_list_setup_filter(_set, BTRFS_LIST_FILTER_BY_PARENT,
> - (u64)(unsigned long)get_ri.uuid);
> - btrfs_list_setup_print_column(BTRFS_LIST_PATH);
>  
> - fd = open_file_or_dir(fullpath, );
> - if (fd < 0) {
> - fprintf(stderr, "ERROR: can't access '%s'\n", fullpath);
> - goto out;
> + err = btrfs_util_f_create_subvolume_iterator(fd,
> +  BTRFS_FS_TREE_OBJECTID,
> +  0, );
> +
> [...]
When you have enough subvolumes in a filesystem, let's say 10 (yes,
that sometimes happens), the current btrfs sub list is quite unusable,
which is kind of expected. But, currently, sub show is also unusable
because it still starts loading a list of all subvolumes to be able to
print the 'snapshots, if any'.

So I guess that this new sub show will at least print the info first and
then use the iterator and start crawling through the list, which can be
interrupted? At least you get the relevant info first then. :-)

-- 
Hans van Kranenburg
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 3/7] btrfs-progs: Make btrfs_alloc_chunk to handle block group creation

[Nikolay Borisov]

On  2.02.2018 10:19, Qu Wenruo wrote:
> Before this patch, chunk allocation is split into 2 parts:
> 
> 1) Chunk allocation
>Handled by btrfs_alloc_chunk(), which will insert chunk and device
>extent items.
> 
> 2) Block group allocation
>Handled by btrfs_make_block_group(), which will insert block group
>item and update space info.
> 
> However for chunk allocation, we don't really need to split these
> operations as all btrfs_alloc_chunk() has btrfs_make_block_group()
> followed.
> 
> So it's reasonable to merge btrfs_make_block_group() call into
> btrfs_alloc_chunk() to save several lines, and provides the basis for
> later btrfs_alloc_chunk() rework.
> 
> Signed-off-by: Qu Wenruo 
> ---
>  convert/main.c |  4 
>  extent-tree.c  | 10 ++
>  mkfs/main.c| 19 ---
>  volumes.c  | 10 ++
>  4 files changed, 8 insertions(+), 35 deletions(-)
> 
> diff --git a/convert/main.c b/convert/main.c
> index b2444bb2ff21..240d3aa46db9 100644
> --- a/convert/main.c
> +++ b/convert/main.c
> @@ -915,10 +915,6 @@ static int make_convert_data_block_groups(struct 
> btrfs_trans_handle *trans,
>   BTRFS_BLOCK_GROUP_DATA, true);
>   if (ret < 0)
>   break;
> - ret = btrfs_make_block_group(trans, fs_info, 0,
> - BTRFS_BLOCK_GROUP_DATA, cur, len);
> - if (ret < 0)
> - break;
>   cur += len;
>   }
>   }
> diff --git a/extent-tree.c b/extent-tree.c
> index b085ab0352b3..bccd83d1bae6 100644
> --- a/extent-tree.c
> +++ b/extent-tree.c
> @@ -1909,15 +1909,9 @@ static int do_chunk_alloc(struct btrfs_trans_handle 
> *trans,
>   space_info->flags, false);
>   if (ret == -ENOSPC) {
>   space_info->full = 1;
> - return 0;
> + return ret;
>   }
> -
> - BUG_ON(ret);
> -
> - ret = btrfs_make_block_group(trans, fs_info, 0, space_info->flags,
> -  start, num_bytes);
> - BUG_ON(ret);
> - return 0;
> + return ret;
>  }
>  
>  static int update_block_group(struct btrfs_root *root,
> diff --git a/mkfs/main.c b/mkfs/main.c
> index 358395ca0250..49159ea533b9 100644
> --- a/mkfs/main.c
> +++ b/mkfs/main.c
> @@ -87,12 +87,6 @@ static int create_metadata_block_groups(struct btrfs_root 
> *root, int mixed,
>   error("no space to allocate data/metadata chunk");
>   goto err;
>   }
> - if (ret)
> - return ret;
> - ret = btrfs_make_block_group(trans, fs_info, 0,
> -  BTRFS_BLOCK_GROUP_METADATA |
> -  BTRFS_BLOCK_GROUP_DATA,
> -  chunk_start, chunk_size);
>   if (ret)
>   return ret;
>   allocation->mixed += chunk_size;
> @@ -106,12 +100,7 @@ static int create_metadata_block_groups(struct 
> btrfs_root *root, int mixed,
>   }
>   if (ret)
>   return ret;
> - ret = btrfs_make_block_group(trans, fs_info, 0,
> -  BTRFS_BLOCK_GROUP_METADATA,
> -  chunk_start, chunk_size);
>   allocation->metadata += chunk_size;
> - if (ret)
> - return ret;
>   }
>  
>   root->fs_info->system_allocs = 0;
> @@ -140,12 +129,7 @@ static int create_data_block_groups(struct 
> btrfs_trans_handle *trans,
>   }
>   if (ret)
>   return ret;
> - ret = btrfs_make_block_group(trans, fs_info, 0,
> -  BTRFS_BLOCK_GROUP_DATA,
> -  chunk_start, chunk_size);
>   allocation->data += chunk_size;
> - if (ret)
> - return ret;
>   }
>  
>  err:
> @@ -249,9 +233,6 @@ static int create_one_raid_group(struct 
> btrfs_trans_handle *trans,
>   if (ret)
>   return ret;
>  
> - ret = btrfs_make_block_group(trans, fs_info, 0,
> -  type, chunk_start, chunk_size);
> -
>   type &= BTRFS_BLOCK_GROUP_TYPE_MASK;
>   if (type == BTRFS_BLOCK_GROUP_DATA) {
>   allocation->data += chunk_size;
> diff --git a/volumes.c b/volumes.c
> index 9ee4650351c3..a9dc8c939dc5 100644
> --- a/volumes.c
> +++ b/volumes.c
> @@ -837,10 +837,9 @@ error:
>   / sizeof(struct btrfs_stripe) + 1)
>  
>  /*
> - * Alloc a chunk, will insert dev extents, chunk item.
> - * NOTE: This function will not insert block group item nor mark newly
> - * allocated chunk available for later allocation.
> - * Block group 

Re: [PATCH 4/7] btrfs-progs: Introduce btrfs_raid_array and related infrastructures

[Nikolay Borisov]

On  2.02.2018 10:19, Qu Wenruo wrote:
> As part of the effort to unify code and behavior between btrfs-progs and
> kernel, copy the btrfs_raid_array from kernel to btrfs-progs.
> 
> So later we can use the btrfs_raid_array[] to get needed raid info other
> than manually do if-else branches.
> 
> Signed-off-by: Qu Wenruo 
> ---
>  ctree.h   | 12 +++-
>  volumes.c | 66 
> +++
>  volumes.h | 30 +
>  3 files changed, 107 insertions(+), 1 deletion(-)
> 
> diff --git a/ctree.h b/ctree.h
> index 17cdac76c58c..c76849d8deb7 100644
> --- a/ctree.h
> +++ b/ctree.h
> @@ -958,7 +958,17 @@ struct btrfs_csum_item {
>  #define BTRFS_BLOCK_GROUP_RAID5  (1ULL << 7)
>  #define BTRFS_BLOCK_GROUP_RAID6  (1ULL << 8)
>  #define BTRFS_BLOCK_GROUP_RESERVED   BTRFS_AVAIL_ALLOC_BIT_SINGLE
 

BTRFS_BLOCK_GROUP_RESERVED differs in userspace than in kernel space. 
The btrfs_Tree header has it defined as: 

#define BTRFS_BLOCK_GROUP_RESERVED  (BTRFS_AVAIL_ALLOC_BIT_SINGLE | \   
 BTRFS_SPACE_INFO_GLOBAL_RSV)


Otherwise LGTM: 

Reviewed-by: Nikolay Borisov 

> -#define BTRFS_NR_RAID_TYPES 7
> +
> +enum btrfs_raid_types {
> + BTRFS_RAID_RAID10,
> + BTRFS_RAID_RAID1,
> + BTRFS_RAID_DUP,
> + BTRFS_RAID_RAID0,
> + BTRFS_RAID_SINGLE,
> + BTRFS_RAID_RAID5,
> + BTRFS_RAID_RAID6,
> + BTRFS_NR_RAID_TYPES
> +};
>  
>  #define BTRFS_BLOCK_GROUP_TYPE_MASK  (BTRFS_BLOCK_GROUP_DATA |\
>BTRFS_BLOCK_GROUP_SYSTEM |  \
> diff --git a/volumes.c b/volumes.c
> index a9dc8c939dc5..b47ff1f392b5 100644
> --- a/volumes.c
> +++ b/volumes.c
> @@ -30,6 +30,72 @@
>  #include "utils.h"
>  #include "kernel-lib/raid56.h"
>  
> +const struct btrfs_raid_attr btrfs_raid_array[BTRFS_NR_RAID_TYPES] = {
> + [BTRFS_RAID_RAID10] = {
> + .sub_stripes= 2,
> + .dev_stripes= 1,
> + .devs_max   = 0,/* 0 == as many as possible */
> + .devs_min   = 4,
> + .tolerated_failures = 1,
> + .devs_increment = 2,
> + .ncopies= 2,
> + },
> + [BTRFS_RAID_RAID1] = {
> + .sub_stripes= 1,
> + .dev_stripes= 1,
> + .devs_max   = 2,
> + .devs_min   = 2,
> + .tolerated_failures = 1,
> + .devs_increment = 2,
> + .ncopies= 2,
> + },
> + [BTRFS_RAID_DUP] = {
> + .sub_stripes= 1,
> + .dev_stripes= 2,
> + .devs_max   = 1,
> + .devs_min   = 1,
> + .tolerated_failures = 0,
> + .devs_increment = 1,
> + .ncopies= 2,
> + },
> + [BTRFS_RAID_RAID0] = {
> + .sub_stripes= 1,
> + .dev_stripes= 1,
> + .devs_max   = 0,
> + .devs_min   = 2,
> + .tolerated_failures = 0,
> + .devs_increment = 1,
> + .ncopies= 1,
> + },
> + [BTRFS_RAID_SINGLE] = {
> + .sub_stripes= 1,
> + .dev_stripes= 1,
> + .devs_max   = 1,
> + .devs_min   = 1,
> + .tolerated_failures = 0,
> + .devs_increment = 1,
> + .ncopies= 1,
> + },
> + [BTRFS_RAID_RAID5] = {
> + .sub_stripes= 1,
> + .dev_stripes= 1,
> + .devs_max   = 0,
> + .devs_min   = 2,
> + .tolerated_failures = 1,
> + .devs_increment = 1,
> + .ncopies= 2,
> + },
> + [BTRFS_RAID_RAID6] = {
> + .sub_stripes= 1,
> + .dev_stripes= 1,
> + .devs_max   = 0,
> + .devs_min   = 3,
> + .tolerated_failures = 2,
> + .devs_increment = 1,
> + .ncopies= 3,
> + },
> +};
> +
>  struct stripe {
>   struct btrfs_device *dev;
>   u64 physical;
> diff --git a/volumes.h b/volumes.h
> index 7bbdf615d31a..612a0a7586f4 100644
> --- a/volumes.h
> +++ b/volumes.h
> @@ -108,6 +108,36 @@ struct map_lookup {
>   struct btrfs_bio_stripe stripes[];
>  };
>  
> +struct btrfs_raid_attr {
> + int sub_stripes;/* sub_stripes info for map */
> + int dev_stripes;/* stripes per dev */
> + int devs_max;   /* max devs to use */
> + int devs_min;   /* min devs needed */
> + int tolerated_failures; /* max tolerated fail devs */
> + int devs_increment; /* ndevs has to be a multiple of this */
> + int ncopies;/* how many copies to data has */
> +};
> +
> +extern const struct btrfs_raid_attr btrfs_raid_array[BTRFS_NR_RAID_TYPES];
> +
> +static inline enum btrfs_raid_types 

Re: [PATCH 5/7] btrfs-progs: volumes: Allow find_free_dev_extent() to return maximum hole size

[Nikolay Borisov]

On  2.02.2018 10:19, Qu Wenruo wrote:
> Just as kernel find_free_dev_extent(), allow it to return maximum hole
> size for us to build device list for later chunk allocator rework.
> 
> Signed-off-by: Qu Wenruo 
> ---
>  volumes.c | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/volumes.c b/volumes.c
> index b47ff1f392b5..f4009ffa7c9e 100644
> --- a/volumes.c
> +++ b/volumes.c
> @@ -516,10 +516,10 @@ out:
>  }
>  
>  static int find_free_dev_extent(struct btrfs_device *device, u64 num_bytes,
> - u64 *start)
> + u64 *start, u64 *len)
>  {
>   /* FIXME use last free of some kind */
> - return find_free_dev_extent_start(device, num_bytes, 0, start, NULL);
> + return find_free_dev_extent_start(device, num_bytes, 0, start, len);

Why do we need the free_dev_extent() wrapper over free_dev_extent_start
at all?
>  }
>  
>  static int btrfs_alloc_dev_extent(struct btrfs_trans_handle *trans,
> @@ -543,7 +543,7 @@ static int btrfs_alloc_dev_extent(struct 
> btrfs_trans_handle *trans,
>* is responsible to make sure it's free.
>*/
>   if (!convert) {
> - ret = find_free_dev_extent(device, num_bytes, start);
> + ret = find_free_dev_extent(device, num_bytes, start, NULL);
>   if (ret)
>   goto err;
>   }
>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] btrfs: qgroup: Fix root item corruption when multiple same source snapshiots are created with quota enabled

[Filipe Manana]

On Tue, Dec 19, 2017 at 7:44 AM, Qu Wenruo  wrote:
> When multiple pending snapshots referring the same source subvolume are
> executed, enabled quota will cause root item corruption, where root
> items are using old bytenr (no backref in extent tree).
>
> This can be triggered by fstests btrfs/152.
>
> The cause is when source subvolume is still dirty, extra commit
> (simplied transaction commit) of qgroup_account_snapshot() can skip
> dirty roots not recorded in current transaction, making root item of
> source subvolume not updated.
>
> Fix it by forcing recording source subvolume in current transaction
> before qgroup sub-transaction commit.
>
> Reported-by: Justin Maggard 
> Signed-off-by: Qu Wenruo 
Reviewed-by: Filipe Manana 

Looks good.

> ---
>  fs/btrfs/transaction.c | 10 +-
>  1 file changed, 9 insertions(+), 1 deletion(-)
>
> diff --git a/fs/btrfs/transaction.c b/fs/btrfs/transaction.c
> index ddae813c01dd..f645e5de5fa5 100644
> --- a/fs/btrfs/transaction.c
> +++ b/fs/btrfs/transaction.c
> @@ -319,7 +319,7 @@ static int record_root_in_trans(struct btrfs_trans_handle 
> *trans,
> if ((test_bit(BTRFS_ROOT_REF_COWS, >state) &&
> root->last_trans < trans->transid) || force) {
> WARN_ON(root == fs_info->extent_root);
> -   WARN_ON(root->commit_root != root->node);
> +   WARN_ON(!force && root->commit_root != root->node);
>
> /*
>  * see below for IN_TRANS_SETUP usage rules
> @@ -1366,6 +1366,14 @@ static int qgroup_account_snapshot(struct 
> btrfs_trans_handle *trans,
> if (!test_bit(BTRFS_FS_QUOTA_ENABLED, _info->flags))
> return 0;
>
> +   /*
> +* Ensure dirty @src will be commited.
> +* Or after comming commit_fs_roots() and switch_commit_roots(),
> +* any dirty but not recorded root will never be updated again.
> +* Causing outdated root item.
> +*/
> +   record_root_in_trans(trans, src, 1);
> +
> /*
>  * We are going to commit transaction, see btrfs_commit_transaction()
>  * comment for reason locking tree_log_mutex
> --
> 2.15.1
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Filipe David Manana,

“Whether you think you can, or you think you can't — you're right.”
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 5/7] btrfs-progs: volumes: Allow find_free_dev_extent() to return maximum hole size

[Qu Wenruo]

On 2018年02月02日 19:41, Nikolay Borisov wrote:
> 
> 
> On  2.02.2018 10:19, Qu Wenruo wrote:
>> Just as kernel find_free_dev_extent(), allow it to return maximum hole
>> size for us to build device list for later chunk allocator rework.
>>
>> Signed-off-by: Qu Wenruo 
>> ---
>>  volumes.c | 6 +++---
>>  1 file changed, 3 insertions(+), 3 deletions(-)
>>
>> diff --git a/volumes.c b/volumes.c
>> index b47ff1f392b5..f4009ffa7c9e 100644
>> --- a/volumes.c
>> +++ b/volumes.c
>> @@ -516,10 +516,10 @@ out:
>>  }
>>  
>>  static int find_free_dev_extent(struct btrfs_device *device, u64 num_bytes,
>> -u64 *start)
>> +u64 *start, u64 *len)
>>  {
>>  /* FIXME use last free of some kind */
>> -return find_free_dev_extent_start(device, num_bytes, 0, start, NULL);
>> +return find_free_dev_extent_start(device, num_bytes, 0, start, len);
> 
> Why do we need the free_dev_extent() wrapper over free_dev_extent_start
> at all?

Because kernel part still needs find_free_dev_extent(), in extent-tree.c.

I'd prefer to keep the function across kernel and btrfs-progs.

Thanks,
Qu

>>  }
>>  
>>  static int btrfs_alloc_dev_extent(struct btrfs_trans_handle *trans,
>> @@ -543,7 +543,7 @@ static int btrfs_alloc_dev_extent(struct 
>> btrfs_trans_handle *trans,
>>   * is responsible to make sure it's free.
>>   */
>>  if (!convert) {
>> -ret = find_free_dev_extent(device, num_bytes, start);
>> +ret = find_free_dev_extent(device, num_bytes, start, NULL);
>>  if (ret)
>>  goto err;
>>  }
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 



signature.asc
Description: OpenPGP digital signature

[PATCH 5/7] btrfs-progs: volumes: Allow find_free_dev_extent() to return maximum hole size

[Qu Wenruo]

Just as kernel find_free_dev_extent(), allow it to return maximum hole
size for us to build device list for later chunk allocator rework.

Signed-off-by: Qu Wenruo 
---
 volumes.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/volumes.c b/volumes.c
index b47ff1f392b5..f4009ffa7c9e 100644
--- a/volumes.c
+++ b/volumes.c
@@ -516,10 +516,10 @@ out:
 }
 
 static int find_free_dev_extent(struct btrfs_device *device, u64 num_bytes,
-   u64 *start)
+   u64 *start, u64 *len)
 {
/* FIXME use last free of some kind */
-   return find_free_dev_extent_start(device, num_bytes, 0, start, NULL);
+   return find_free_dev_extent_start(device, num_bytes, 0, start, len);
 }
 
 static int btrfs_alloc_dev_extent(struct btrfs_trans_handle *trans,
@@ -543,7 +543,7 @@ static int btrfs_alloc_dev_extent(struct btrfs_trans_handle 
*trans,
 * is responsible to make sure it's free.
 */
if (!convert) {
-   ret = find_free_dev_extent(device, num_bytes, start);
+   ret = find_free_dev_extent(device, num_bytes, start, NULL);
if (ret)
goto err;
}
-- 
2.16.1

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 7/7] btrfs-progs: volumes: Unify free dev extent search behavior between kernel and btrfs-progs.

[Qu Wenruo]

As preparation to create libbtrfs which shares code between kernel and
btrfs, this patch mainly unifies the search for free device extents.

The main modifications are:

1) Search for free device extent
   Use the kernel method, by sorting the devices by its max hole
   capability, and use that sorted result to determine stripe size
   and chunk size.

   The previous method, which uses available bytes of each device to
   search, can't handle scattered small holes in each device.

2) Chunk/stripe minimal size
   Remove the minimal size requirement.
   Now the real minimal stripe size is 64K (BTRFS_STRIPE_LEN), the same
   as kernel one.

   While up limit is still kept as is, and minimal device size is still
   kept for a while.
   But since we no longer have strange minimal stripe size limit,
   existing minimal device size calculation won't cause any problem.

3) How to insert device extents
   Although not the same as kernel, here we follow kernel behavior to
   delay dev extents insert.

   There is plan to follow kernel __btrfs_alloc_chunk() to make it only
   handle chunk mapping allocation, while do nothing with tree
   operation.

4) Usage of btrfs_raid_array[]
   Which makes a lot of old if-else branches disappear.

There are still a lot of work to do (both kernel and btrfs-progs) before
we could starting extracting code into libbtrfs, but this should make
libbtrfs inside our reach.

Signed-off-by: Qu Wenruo 
---
 kerncompat.h |   5 +
 volumes.c| 611 ++-
 volumes.h|   7 +
 3 files changed, 281 insertions(+), 342 deletions(-)

diff --git a/kerncompat.h b/kerncompat.h
index fa96715fb70c..658d28ed0792 100644
--- a/kerncompat.h
+++ b/kerncompat.h
@@ -285,6 +285,7 @@ static inline int IS_ERR_OR_NULL(const void *ptr)
  */
 #define kmalloc(x, y) malloc(x)
 #define kzalloc(x, y) calloc(1, x)
+#define kcalloc(x, y) calloc(x, y)
 #define kstrdup(x, y) strdup(x)
 #define kfree(x) free(x)
 #define vmalloc(x) malloc(x)
@@ -394,4 +395,8 @@ struct __una_u64 { __le64 x; } __attribute__((__packed__));
 #define noinline
 #endif
 
+static inline u64 div_u64(u64 dividend, u32 divisor)
+{
+   return dividend / ((u64) divisor);
+}
 #endif
diff --git a/volumes.c b/volumes.c
index f4009ffa7c9e..d14f46f8496e 100644
--- a/volumes.c
+++ b/volumes.c
@@ -29,6 +29,7 @@
 #include "volumes.h"
 #include "utils.h"
 #include "kernel-lib/raid56.h"
+#include "kernel-lib/sort.h"
 
 const struct btrfs_raid_attr btrfs_raid_array[BTRFS_NR_RAID_TYPES] = {
[BTRFS_RAID_RAID10] = {
@@ -522,55 +523,55 @@ static int find_free_dev_extent(struct btrfs_device 
*device, u64 num_bytes,
return find_free_dev_extent_start(device, num_bytes, 0, start, len);
 }
 
-static int btrfs_alloc_dev_extent(struct btrfs_trans_handle *trans,
- struct btrfs_device *device,
- u64 chunk_offset, u64 num_bytes, u64 *start,
- int convert)
+static int btrfs_insert_dev_extents(struct btrfs_trans_handle *trans,
+   struct btrfs_fs_info *fs_info,
+   struct map_lookup *map, u64 stripe_size)
 {
-   int ret;
-   struct btrfs_path *path;
-   struct btrfs_root *root = device->dev_root;
+   int ret = 0;
+   struct btrfs_path path;
+   struct btrfs_root *root = fs_info->dev_root;
struct btrfs_dev_extent *extent;
struct extent_buffer *leaf;
struct btrfs_key key;
+   int i;
 
-   path = btrfs_alloc_path();
-   if (!path)
-   return -ENOMEM;
+   btrfs_init_path();
 
-   /*
-* For convert case, just skip search free dev_extent, as caller
-* is responsible to make sure it's free.
-*/
-   if (!convert) {
-   ret = find_free_dev_extent(device, num_bytes, start, NULL);
-   if (ret)
-   goto err;
-   }
+   for (i = 0; i < map->num_stripes; i++) {
+   struct btrfs_device *device = map->stripes[i].dev;
 
-   key.objectid = device->devid;
-   key.offset = *start;
-   key.type = BTRFS_DEV_EXTENT_KEY;
-   ret = btrfs_insert_empty_item(trans, root, path, ,
- sizeof(*extent));
-   BUG_ON(ret);
+   key.objectid = device->devid;
+   key.offset = map->stripes[i].physical;
+   key.type = BTRFS_DEV_EXTENT_KEY;
 
-   leaf = path->nodes[0];
-   extent = btrfs_item_ptr(leaf, path->slots[0],
-   struct btrfs_dev_extent);
-   btrfs_set_dev_extent_chunk_tree(leaf, extent, 
BTRFS_CHUNK_TREE_OBJECTID);
-   btrfs_set_dev_extent_chunk_objectid(leaf, extent,
-   BTRFS_FIRST_CHUNK_TREE_OBJECTID);
-   btrfs_set_dev_extent_chunk_offset(leaf, extent, chunk_offset);
-
-   write_extent_buffer(leaf, 

[PATCH 1/7] btrfs-progs: Refactor parameter of BTRFS_MAX_DEVS() from root to fs_info

[Qu Wenruo]

Signed-off-by: Qu Wenruo 
---
 volumes.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/volumes.c b/volumes.c
index edad367b593c..677d085de96c 100644
--- a/volumes.c
+++ b/volumes.c
@@ -826,7 +826,7 @@ error:
return ret;
 }
 
-#define BTRFS_MAX_DEVS(r) ((BTRFS_LEAF_DATA_SIZE(r->fs_info)   \
+#define BTRFS_MAX_DEVS(info) ((BTRFS_LEAF_DATA_SIZE(info)  \
- sizeof(struct btrfs_item) \
- sizeof(struct btrfs_chunk))   \
/ sizeof(struct btrfs_stripe) + 1)
@@ -882,12 +882,12 @@ int btrfs_alloc_chunk(struct btrfs_trans_handle *trans,
calc_size = SZ_1G;
max_chunk_size = 10 * calc_size;
min_stripe_size = SZ_64M;
-   max_stripes = BTRFS_MAX_DEVS(chunk_root);
+   max_stripes = BTRFS_MAX_DEVS(info);
} else if (type & BTRFS_BLOCK_GROUP_METADATA) {
calc_size = SZ_1G;
max_chunk_size = 4 * calc_size;
min_stripe_size = SZ_32M;
-   max_stripes = BTRFS_MAX_DEVS(chunk_root);
+   max_stripes = BTRFS_MAX_DEVS(info);
}
}
if (type & BTRFS_BLOCK_GROUP_RAID1) {
-- 
2.16.1

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 2/7] btrfs-progs: Merge btrfs_alloc_data_chunk into btrfs_alloc_chunk

[Qu Wenruo]

We used to have two chunk allocators, btrfs_alloc_chunk() and
btrfs_alloc_data_chunk(), the former is the more generic one, while the
later is only used in mkfs and convert, to allocate SINGLE data chunk.

Although btrfs_alloc_data_chunk() has some special hacks to cooperate
with convert, it's quite simple to integrity it into the generic chunk
allocator.

So merge them into one btrfs_alloc_chunk(), with extra @convert
parameter and necessary comment, to make code less duplicated and less
thing to maintain.

Signed-off-by: Qu Wenruo 
---
 convert/main.c |   6 +-
 extent-tree.c  |   2 +-
 mkfs/main.c|   8 +--
 volumes.c  | 219 ++---
 volumes.h  |   5 +-
 5 files changed, 94 insertions(+), 146 deletions(-)

diff --git a/convert/main.c b/convert/main.c
index b3ea31d7af43..b2444bb2ff21 100644
--- a/convert/main.c
+++ b/convert/main.c
@@ -910,9 +910,9 @@ static int make_convert_data_block_groups(struct 
btrfs_trans_handle *trans,
 
len = min(max_chunk_size,
  cache->start + cache->size - cur);
-   ret = btrfs_alloc_data_chunk(trans, fs_info,
-   _backup, len,
-   BTRFS_BLOCK_GROUP_DATA, 1);
+   ret = btrfs_alloc_chunk(trans, fs_info,
+   _backup, ,
+   BTRFS_BLOCK_GROUP_DATA, true);
if (ret < 0)
break;
ret = btrfs_make_block_group(trans, fs_info, 0,
diff --git a/extent-tree.c b/extent-tree.c
index e2ae74a7fe66..b085ab0352b3 100644
--- a/extent-tree.c
+++ b/extent-tree.c
@@ -1906,7 +1906,7 @@ static int do_chunk_alloc(struct btrfs_trans_handle 
*trans,
return 0;
 
ret = btrfs_alloc_chunk(trans, fs_info, , _bytes,
-   space_info->flags);
+   space_info->flags, false);
if (ret == -ENOSPC) {
space_info->full = 1;
return 0;
diff --git a/mkfs/main.c b/mkfs/main.c
index ea65c6d897b2..358395ca0250 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -82,7 +82,7 @@ static int create_metadata_block_groups(struct btrfs_root 
*root, int mixed,
ret = btrfs_alloc_chunk(trans, fs_info,
_start, _size,
BTRFS_BLOCK_GROUP_METADATA |
-   BTRFS_BLOCK_GROUP_DATA);
+   BTRFS_BLOCK_GROUP_DATA, false);
if (ret == -ENOSPC) {
error("no space to allocate data/metadata chunk");
goto err;
@@ -99,7 +99,7 @@ static int create_metadata_block_groups(struct btrfs_root 
*root, int mixed,
} else {
ret = btrfs_alloc_chunk(trans, fs_info,
_start, _size,
-   BTRFS_BLOCK_GROUP_METADATA);
+   BTRFS_BLOCK_GROUP_METADATA, false);
if (ret == -ENOSPC) {
error("no space to allocate metadata chunk");
goto err;
@@ -133,7 +133,7 @@ static int create_data_block_groups(struct 
btrfs_trans_handle *trans,
if (!mixed) {
ret = btrfs_alloc_chunk(trans, fs_info,
_start, _size,
-   BTRFS_BLOCK_GROUP_DATA);
+   BTRFS_BLOCK_GROUP_DATA, false);
if (ret == -ENOSPC) {
error("no space to allocate data chunk");
goto err;
@@ -241,7 +241,7 @@ static int create_one_raid_group(struct btrfs_trans_handle 
*trans,
int ret;
 
ret = btrfs_alloc_chunk(trans, fs_info,
-   _start, _size, type);
+   _start, _size, type, false);
if (ret == -ENOSPC) {
error("not enough free space to allocate chunk");
exit(1);
diff --git a/volumes.c b/volumes.c
index 677d085de96c..9ee4650351c3 100644
--- a/volumes.c
+++ b/volumes.c
@@ -836,9 +836,23 @@ error:
- 2 * sizeof(struct btrfs_chunk))   \
/ sizeof(struct btrfs_stripe) + 1)
 
+/*
+ * Alloc a chunk, will insert dev extents, chunk item.
+ * NOTE: This function will not insert block group item nor mark newly
+ * allocated chunk available for later allocation.
+ * Block group item and free space update is handled by 
btrfs_make_block_group()
+ *
+ * @start: return value of allocated chunk start bytenr.
+ * @num_bytes: return value of allocated chunk size
+ * @type:  chunk type (including both profile and type)
+ * @convert:   if the chunk is 

[PATCH 0/7] Chunk allocator unification

[Qu Wenruo]

This patchset can be fetched from github:
https://github.com/adam900710/btrfs-progs/tree/libbtrfs_prepare

This patchset unified a large part of chunk allocator (free device
extent search) between kernel and btrfs-progs.

Although there are still differences in the following areas, the goal of
extract common code into libbtrfs is in the reach.

1) Member differences
   Btrfs-progs lacks quite some members of kernel, so something like
   fs_devices->total_rw_bytes still makes difference.

2) @convert parameter for mkfs/convert
   It should be able to extract them into a simpler function.

3) delayed bg creations vs direct bg creation
   Kernel delays block group creation while btrfs-progs directly
   allocates block group.
   This makes kernel __btrfs_alloc_chunk() more or less chunk mapping
   allocation only, while btrfs-progs handles extra chunk, dev extent
   and block group item creation.

   This needs extra function refactor before we really extra code into
   libbtrfs.

Despite the preparation for later libbtrfs, this rework still has some
benefit:

1) No minimal stripe/chunk size
   Unlike the intermediate minimal stripe/chunk size in btrfs-progs,
   kernel doesn't really implement a minimal stripe/chunk size.

   So now we don't have such limit in btrfs-progs side now.
   Although the minimal device size calculation doesn't make any harm so
   it's still kept as is.

2) Slightly less code for chunk allocator
   Just about -60 lines in the last patch.


Qu Wenruo (7):
  btrfs-progs: Refactor parameter of BTRFS_MAX_DEVS() from root to
fs_info
  btrfs-progs: Merge btrfs_alloc_data_chunk into btrfs_alloc_chunk
  btrfs-progs: Make btrfs_alloc_chunk to handle block group creation
  btrfs-progs: Introduce btrfs_raid_array and related infrastructures
  btrfs-progs: volumes: Allow find_free_dev_extent() to return maximum
hole size
  btrfs-progs: kernel-lib: Port kernel sort() to btrfs-progs
  btrfs-progs: volumes: Unify free dev extent search behavior between
kernel and btrfs-progs.

 Makefile  |   3 +-
 convert/main.c|  10 +-
 ctree.h   |  12 +-
 extent-tree.c |  12 +-
 kerncompat.h  |   5 +
 kernel-lib/sort.c | 104 +++
 kernel-lib/sort.h |  16 ++
 mkfs/main.c   |  27 +-
 volumes.c | 838 +-
 volumes.h |  42 ++-
 10 files changed, 578 insertions(+), 491 deletions(-)
 create mode 100644 kernel-lib/sort.c
 create mode 100644 kernel-lib/sort.h

-- 
2.16.1

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 4/7] btrfs-progs: Introduce btrfs_raid_array and related infrastructures

[Qu Wenruo]

As part of the effort to unify code and behavior between btrfs-progs and
kernel, copy the btrfs_raid_array from kernel to btrfs-progs.

So later we can use the btrfs_raid_array[] to get needed raid info other
than manually do if-else branches.

Signed-off-by: Qu Wenruo 
---
 ctree.h   | 12 +++-
 volumes.c | 66 +++
 volumes.h | 30 +
 3 files changed, 107 insertions(+), 1 deletion(-)

diff --git a/ctree.h b/ctree.h
index 17cdac76c58c..c76849d8deb7 100644
--- a/ctree.h
+++ b/ctree.h
@@ -958,7 +958,17 @@ struct btrfs_csum_item {
 #define BTRFS_BLOCK_GROUP_RAID5(1ULL << 7)
 #define BTRFS_BLOCK_GROUP_RAID6(1ULL << 8)
 #define BTRFS_BLOCK_GROUP_RESERVED BTRFS_AVAIL_ALLOC_BIT_SINGLE
-#define BTRFS_NR_RAID_TYPES 7
+
+enum btrfs_raid_types {
+   BTRFS_RAID_RAID10,
+   BTRFS_RAID_RAID1,
+   BTRFS_RAID_DUP,
+   BTRFS_RAID_RAID0,
+   BTRFS_RAID_SINGLE,
+   BTRFS_RAID_RAID5,
+   BTRFS_RAID_RAID6,
+   BTRFS_NR_RAID_TYPES
+};
 
 #define BTRFS_BLOCK_GROUP_TYPE_MASK(BTRFS_BLOCK_GROUP_DATA |\
 BTRFS_BLOCK_GROUP_SYSTEM |  \
diff --git a/volumes.c b/volumes.c
index a9dc8c939dc5..b47ff1f392b5 100644
--- a/volumes.c
+++ b/volumes.c
@@ -30,6 +30,72 @@
 #include "utils.h"
 #include "kernel-lib/raid56.h"
 
+const struct btrfs_raid_attr btrfs_raid_array[BTRFS_NR_RAID_TYPES] = {
+   [BTRFS_RAID_RAID10] = {
+   .sub_stripes= 2,
+   .dev_stripes= 1,
+   .devs_max   = 0,/* 0 == as many as possible */
+   .devs_min   = 4,
+   .tolerated_failures = 1,
+   .devs_increment = 2,
+   .ncopies= 2,
+   },
+   [BTRFS_RAID_RAID1] = {
+   .sub_stripes= 1,
+   .dev_stripes= 1,
+   .devs_max   = 2,
+   .devs_min   = 2,
+   .tolerated_failures = 1,
+   .devs_increment = 2,
+   .ncopies= 2,
+   },
+   [BTRFS_RAID_DUP] = {
+   .sub_stripes= 1,
+   .dev_stripes= 2,
+   .devs_max   = 1,
+   .devs_min   = 1,
+   .tolerated_failures = 0,
+   .devs_increment = 1,
+   .ncopies= 2,
+   },
+   [BTRFS_RAID_RAID0] = {
+   .sub_stripes= 1,
+   .dev_stripes= 1,
+   .devs_max   = 0,
+   .devs_min   = 2,
+   .tolerated_failures = 0,
+   .devs_increment = 1,
+   .ncopies= 1,
+   },
+   [BTRFS_RAID_SINGLE] = {
+   .sub_stripes= 1,
+   .dev_stripes= 1,
+   .devs_max   = 1,
+   .devs_min   = 1,
+   .tolerated_failures = 0,
+   .devs_increment = 1,
+   .ncopies= 1,
+   },
+   [BTRFS_RAID_RAID5] = {
+   .sub_stripes= 1,
+   .dev_stripes= 1,
+   .devs_max   = 0,
+   .devs_min   = 2,
+   .tolerated_failures = 1,
+   .devs_increment = 1,
+   .ncopies= 2,
+   },
+   [BTRFS_RAID_RAID6] = {
+   .sub_stripes= 1,
+   .dev_stripes= 1,
+   .devs_max   = 0,
+   .devs_min   = 3,
+   .tolerated_failures = 2,
+   .devs_increment = 1,
+   .ncopies= 3,
+   },
+};
+
 struct stripe {
struct btrfs_device *dev;
u64 physical;
diff --git a/volumes.h b/volumes.h
index 7bbdf615d31a..612a0a7586f4 100644
--- a/volumes.h
+++ b/volumes.h
@@ -108,6 +108,36 @@ struct map_lookup {
struct btrfs_bio_stripe stripes[];
 };
 
+struct btrfs_raid_attr {
+   int sub_stripes;/* sub_stripes info for map */
+   int dev_stripes;/* stripes per dev */
+   int devs_max;   /* max devs to use */
+   int devs_min;   /* min devs needed */
+   int tolerated_failures; /* max tolerated fail devs */
+   int devs_increment; /* ndevs has to be a multiple of this */
+   int ncopies;/* how many copies to data has */
+};
+
+extern const struct btrfs_raid_attr btrfs_raid_array[BTRFS_NR_RAID_TYPES];
+
+static inline enum btrfs_raid_types btrfs_bg_flags_to_raid_index(u64 flags)
+{
+   if (flags & BTRFS_BLOCK_GROUP_RAID10)
+   return BTRFS_RAID_RAID10;
+   else if (flags & BTRFS_BLOCK_GROUP_RAID1)
+   return BTRFS_RAID_RAID1;
+   else if (flags & BTRFS_BLOCK_GROUP_DUP)
+   return BTRFS_RAID_DUP;
+   else if (flags & BTRFS_BLOCK_GROUP_RAID0)
+   return BTRFS_RAID_RAID0;
+   else if (flags & BTRFS_BLOCK_GROUP_RAID5)
+   return 

[RFC PATCH v2 0/4] Add support for export testsuits

[Gu Jinxiang]

Achieved:
1. export testsuit by:
 $ make EXPORT=/where/you/want/to/generate/tests.tar.gz package
relative path and absolute path both be ok.
Besides tests itself, fssum and btrfs-corrupt-block will also be
included in tests.tar.gz, since misc and fsck tests depend on
btrfs-corrupt-block, and misc tests depend on fssum. 

2. after decompress tests.tar.gz, run test by:
 $ TEST=`MASK` ./mkfs-tests.sh
and, without MASK also be ok.
replenish:
The directory Structure after decompress tests.tar.gz is:
 $ tar -xzvf ./tests.tar.gz
 $ ls
   tests fssum btrfs-corrupt-block

Gu Jinxiang (4):
  btrfs-progs: Add package command for packaging tests
  btrfs-progs: Add EXEC represent path of executable file
  btrfs-progs: Modify rootdir files when use mkfs.btrfs
  btrfs-progs: Add readme for export testsuits

 Makefile   |  4 ++
 tests/README.md| 20 
 tests/cli-tests.sh |  5 ++
 tests/cli-tests/001-btrfs/test.sh  | 20 
 .../cli-tests/002-balance-full-no-filters/test.sh  | 10 ++--
 tests/cli-tests/003-fi-resize-args/test.sh | 30 +--
 .../cli-tests/004-send-parent-multi-subvol/test.sh | 12 ++---
 tests/cli-tests/005-qgroup-show/test.sh| 14 ++---
 tests/cli-tests/006-qgroup-show-sync/test.sh   | 14 ++---
 tests/cli-tests/007-check-force/test.sh| 14 ++---
 .../008-subvolume-get-set-default/test.sh  | 20 
 tests/common   | 16 +++---
 tests/common.convert   |  8 +--
 tests/convert-tests.sh |  5 ++
 .../004-ext2-backup-superblock-ranges/test.sh  | 10 ++--
 .../convert-tests/005-delete-all-rollback/test.sh  |  2 +-
 .../007-unsupported-block-sizes/test.sh|  2 +-
 .../011-reiserfs-delete-all-rollback/test.sh   |  2 +-
 .../015-no-rollback-after-balance/test.sh  |  6 +--
 tests/export-tests.sh  | 28 ++
 tests/fsck-tests.sh|  5 ++
 tests/fsck-tests/012-leaf-corruption/test.sh   |  2 +-
 tests/fsck-tests/013-extent-tree-rebuild/test.sh   | 12 ++---
 tests/fsck-tests/018-leaf-crossing-stripes/test.sh |  2 +-
 .../fsck-tests/019-non-skinny-false-alert/test.sh  |  2 +-
 tests/fsck-tests/020-extent-ref-cases/test.sh  |  2 +-
 .../021-partially-dropped-snapshot-case/test.sh|  2 +-
 tests/fsck-tests/022-qgroup-rescan-halfway/test.sh |  2 +-
 tests/fsck-tests/023-qgroup-stack-overflow/test.sh |  2 +-
 tests/fsck-tests/024-clear-space-cache/test.sh | 10 ++--
 tests/fsck-tests/025-file-extents/test.sh  | 14 ++---
 tests/fsck-tests/026-bad-dir-item-name/test.sh |  2 +-
 tests/fsck-tests/027-tree-reloc-tree/test.sh   |  2 +-
 .../028-unaligned-super-dev-sizes/test.sh  |  6 +--
 tests/fuzz-tests.sh|  5 ++
 .../fuzz-tests/001-simple-check-unmounted/test.sh  |  2 +-
 tests/fuzz-tests/002-simple-image/test.sh  |  2 +-
 tests/fuzz-tests/003-multi-check-unmounted/test.sh | 12 ++---
 tests/fuzz-tests/004-simple-dump-tree/test.sh  |  2 +-
 tests/fuzz-tests/005-simple-dump-super/test.sh |  4 +-
 tests/fuzz-tests/006-simple-tree-stats/test.sh |  2 +-
 tests/fuzz-tests/007-simple-super-recover/test.sh  |  2 +-
 tests/fuzz-tests/008-simple-chunk-recover/test.sh  |  2 +-
 tests/fuzz-tests/009-simple-zero-log/test.sh   |  2 +-
 tests/misc-tests.sh|  5 ++
 tests/misc-tests/001-btrfstune-features/test.sh| 10 ++--
 tests/misc-tests/002-uuid-rewrite/test.sh  | 24 -
 tests/misc-tests/003-zero-log/test.sh  | 18 +++
 tests/misc-tests/004-shrink-fs/test.sh | 14 ++---
 .../005-convert-progress-thread-crash/test.sh  |  2 +-
 .../misc-tests/006-image-on-missing-device/test.sh | 10 ++--
 tests/misc-tests/007-subvolume-sync/test.sh| 18 +++
 tests/misc-tests/008-leaf-crossing-stripes/test.sh |  4 +-
 .../009-subvolume-sync-must-wait/test.sh   | 22 
 .../010-convert-delete-ext2-subvol/test.sh | 10 ++--
 tests/misc-tests/011-delete-missing-device/test.sh | 18 +++
 tests/misc-tests/012-find-root-no-result/test.sh   |  4 +-
 tests/misc-tests/013-subvolume-sync-crash/test.sh  | 20 
 tests/misc-tests/014-filesystem-label/test.sh  | 22 
 tests/misc-tests/015-dump-super-garbage/test.sh| 18 +++
 tests/misc-tests/016-send-clone-src/test.sh| 12 ++---
 .../017-recv-stream-malformatted/test.sh   |  8 +--
 tests/misc-tests/018-recv-end-of-stream/test.sh| 60 +++---
 .../019-receive-clones-on-mounted-subvol/test.sh   | 38 +++---
 .../020-fix-superblock-corruption/test.sh  |  8 +--
 tests/misc-tests/021-image-multi-devices/test.sh   |  6 +--
 .../022-filesystem-du-on-empty-subvol/test.sh  | 12 ++---
 

[RFC PATCH v2 2/4] btrfs-progs: Add EXEC represent path of executable file

[Gu Jinxiang]

Use EXEC instead of TOP to represent the path of excutable file.
EXEC is set to TOP by default, but when there is no excutable file
in TOP, use the path where btrfs is install as EXEC.

Signed-off-by: Gu Jinxiang 
---
 tests/cli-tests.sh |  5 ++
 tests/cli-tests/001-btrfs/test.sh  | 20 
 .../cli-tests/002-balance-full-no-filters/test.sh  | 10 ++--
 tests/cli-tests/003-fi-resize-args/test.sh | 30 +--
 .../cli-tests/004-send-parent-multi-subvol/test.sh | 12 ++---
 tests/cli-tests/005-qgroup-show/test.sh| 14 ++---
 tests/cli-tests/006-qgroup-show-sync/test.sh   | 14 ++---
 tests/cli-tests/007-check-force/test.sh| 14 ++---
 .../008-subvolume-get-set-default/test.sh  | 20 
 tests/common   | 16 +++---
 tests/common.convert   |  8 +--
 tests/convert-tests.sh |  5 ++
 .../004-ext2-backup-superblock-ranges/test.sh  | 10 ++--
 .../convert-tests/005-delete-all-rollback/test.sh  |  2 +-
 .../007-unsupported-block-sizes/test.sh|  2 +-
 .../011-reiserfs-delete-all-rollback/test.sh   |  2 +-
 .../015-no-rollback-after-balance/test.sh  |  6 +--
 tests/fsck-tests.sh|  5 ++
 tests/fsck-tests/012-leaf-corruption/test.sh   |  2 +-
 tests/fsck-tests/013-extent-tree-rebuild/test.sh   | 12 ++---
 tests/fsck-tests/018-leaf-crossing-stripes/test.sh |  2 +-
 .../fsck-tests/019-non-skinny-false-alert/test.sh  |  2 +-
 tests/fsck-tests/020-extent-ref-cases/test.sh  |  2 +-
 .../021-partially-dropped-snapshot-case/test.sh|  2 +-
 tests/fsck-tests/022-qgroup-rescan-halfway/test.sh |  2 +-
 tests/fsck-tests/023-qgroup-stack-overflow/test.sh |  2 +-
 tests/fsck-tests/024-clear-space-cache/test.sh | 10 ++--
 tests/fsck-tests/025-file-extents/test.sh  | 14 ++---
 tests/fsck-tests/026-bad-dir-item-name/test.sh |  2 +-
 tests/fsck-tests/027-tree-reloc-tree/test.sh   |  2 +-
 .../028-unaligned-super-dev-sizes/test.sh  |  6 +--
 tests/fuzz-tests.sh|  5 ++
 .../fuzz-tests/001-simple-check-unmounted/test.sh  |  2 +-
 tests/fuzz-tests/002-simple-image/test.sh  |  2 +-
 tests/fuzz-tests/003-multi-check-unmounted/test.sh | 12 ++---
 tests/fuzz-tests/004-simple-dump-tree/test.sh  |  2 +-
 tests/fuzz-tests/005-simple-dump-super/test.sh |  4 +-
 tests/fuzz-tests/006-simple-tree-stats/test.sh |  2 +-
 tests/fuzz-tests/007-simple-super-recover/test.sh  |  2 +-
 tests/fuzz-tests/008-simple-chunk-recover/test.sh  |  2 +-
 tests/fuzz-tests/009-simple-zero-log/test.sh   |  2 +-
 tests/misc-tests.sh|  5 ++
 tests/misc-tests/001-btrfstune-features/test.sh| 10 ++--
 tests/misc-tests/002-uuid-rewrite/test.sh  | 20 
 tests/misc-tests/003-zero-log/test.sh  | 16 +++---
 tests/misc-tests/004-shrink-fs/test.sh | 14 ++---
 .../005-convert-progress-thread-crash/test.sh  |  2 +-
 .../misc-tests/006-image-on-missing-device/test.sh | 10 ++--
 tests/misc-tests/007-subvolume-sync/test.sh| 18 +++
 tests/misc-tests/008-leaf-crossing-stripes/test.sh |  4 +-
 .../009-subvolume-sync-must-wait/test.sh   | 22 
 .../010-convert-delete-ext2-subvol/test.sh | 10 ++--
 tests/misc-tests/011-delete-missing-device/test.sh | 18 +++
 tests/misc-tests/012-find-root-no-result/test.sh   |  4 +-
 tests/misc-tests/013-subvolume-sync-crash/test.sh  | 20 
 tests/misc-tests/014-filesystem-label/test.sh  | 22 
 tests/misc-tests/015-dump-super-garbage/test.sh| 18 +++
 tests/misc-tests/016-send-clone-src/test.sh| 12 ++---
 .../017-recv-stream-malformatted/test.sh   |  8 +--
 tests/misc-tests/018-recv-end-of-stream/test.sh| 60 +++---
 .../019-receive-clones-on-mounted-subvol/test.sh   | 38 +++---
 .../020-fix-superblock-corruption/test.sh  |  8 +--
 tests/misc-tests/021-image-multi-devices/test.sh   |  6 +--
 .../022-filesystem-du-on-empty-subvol/test.sh  | 12 ++---
 .../023-device-usage-with-missing-device/test.sh   | 10 ++--
 .../misc-tests/024-inspect-internal-rootid/test.sh | 24 -
 tests/misc-tests/025-zstd-compression/test.sh  |  6 +--
 .../026-image-non-printable-chars/test.sh  |  8 +--
 .../027-subvol-list-deleted-toplevel/test.sh   |  6 +--
 tests/mkfs-tests.sh|  5 ++
 tests/mkfs-tests/001-basic-profiles/test.sh| 12 ++---
 .../002-no-force-mixed-on-small-volume/test.sh |  2 +-
 .../003-mixed-with-wrong-nodesize/test.sh  |  2 +-
 tests/mkfs-tests/004-rootdir-keeps-size/test.sh|  4 +-
 .../005-long-device-name-for-ssd/test.sh   |  4 +-
 tests/mkfs-tests/006-partitioned-loopdev/test.sh   |  4 +-
 

[RFC PATCH v2 3/4] btrfs-progs: Modify rootdir files when use mkfs.btrfs

[Gu Jinxiang]

Since there is no $TOP/Documentation file after package the testsuite.
So use a common file that is usually exsits.

Signed-off-by: Gu Jinxiang 
---
 tests/misc-tests/002-uuid-rewrite/test.sh   | 4 ++--
 tests/misc-tests/003-zero-log/test.sh   | 2 +-
 tests/mkfs-tests/004-rootdir-keeps-size/test.sh | 2 +-
 3 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/tests/misc-tests/002-uuid-rewrite/test.sh 
b/tests/misc-tests/002-uuid-rewrite/test.sh
index 158bda90..27341322 100755
--- a/tests/misc-tests/002-uuid-rewrite/test.sh
+++ b/tests/misc-tests/002-uuid-rewrite/test.sh
@@ -25,7 +25,7 @@ test_uuid_random()
 
run_check $SUDO_HELPER $EXEC/mkfs.btrfs -f \
--uuid $origuuid \
-   --rootdir $TOP/Documentation \
+   --rootdir /lib/modules/`uname -r`/ \
$TEST_DEV
run_check $EXEC/btrfs inspect-internal dump-super "$TEST_DEV"
currentfsid=$(run_check_stdout $EXEC/btrfstune -f -u $TEST_DEV | \
@@ -47,7 +47,7 @@ test_uuid_user()
 
run_check $SUDO_HELPER $EXEC/mkfs.btrfs -f \
--uuid $origuuid \
-   --rootdir $TOP/Documentation \
+   --rootdir /lib/modules/`uname -r`/ \
$TEST_DEV
run_check $EXEC/btrfs inspect-internal dump-super "$TEST_DEV"
run_check $EXEC/btrfstune -f -U $newuuid \
diff --git a/tests/misc-tests/003-zero-log/test.sh 
b/tests/misc-tests/003-zero-log/test.sh
index 0f3d3cd6..db1b1c67 100755
--- a/tests/misc-tests/003-zero-log/test.sh
+++ b/tests/misc-tests/003-zero-log/test.sh
@@ -27,7 +27,7 @@ test_zero_log()
 {
# FIXME: we need an image with existing log_root
run_check $SUDO_HELPER $EXEC/mkfs.btrfs -f \
-   --rootdir $TOP/Documentation \
+   --rootdir /lib/modules/`uname -r`/ \
$TEST_DEV
run_check $EXEC/btrfs inspect-internal dump-super $TEST_DEV
if [ "$1" = 'standalone' ]; then
diff --git a/tests/mkfs-tests/004-rootdir-keeps-size/test.sh 
b/tests/mkfs-tests/004-rootdir-keeps-size/test.sh
index c8d920a5..3c53a1a7 100755
--- a/tests/mkfs-tests/004-rootdir-keeps-size/test.sh
+++ b/tests/mkfs-tests/004-rootdir-keeps-size/test.sh
@@ -16,7 +16,7 @@ test_mkfs_with_size() {
run_check truncate -s$size $TEST_DEV
imgsize=$(run_check_stdout stat --format=%s $TEST_DEV)
run_check $SUDO_HELPER $EXEC/mkfs.btrfs -f \
-   --rootdir $EXEC/Documentation \
+   --rootdir /lib/modules/`uname -r`/ \
$TEST_DEV
tmp=$(run_check_stdout stat --format=%s $TEST_DEV)
if ! [ "$imgsize" = "$tmp" ]; then
-- 
2.14.3



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[RFC PATCH v2 1/4] btrfs-progs: Add package command for packaging tests

[Gu Jinxiang]

Export the testsuite files to a separate tar.
Since fsck tests depend on btrfs-corrupt-block, and misc
tests depends on both btrfs-corrupt-block and fssum,
so set it as prerequisites for package commad.

Because,
althougth fssum can be generated by source that are all in
tests directory, and has no rely on the btrfs's structure.
But btrfs-corrupt-block deeply relys on btrfs's structure.
For consistency, at the present stage, generete the two
when create test tar.

Signed-off-by: Gu Jinxiang 
---
 Makefile  |  4 
 tests/export-tests.sh | 28 
 2 files changed, 32 insertions(+)
 create mode 100755 tests/export-tests.sh

diff --git a/Makefile b/Makefile
index 6369e8f4..23decc4d 100644
--- a/Makefile
+++ b/Makefile
@@ -333,6 +333,10 @@ test-inst: all
 
 test: test-fsck test-mkfs test-convert test-misc test-fuzz test-cli
 
+package: btrfs-corrupt-block fssum
+   @echo "Export tests as a package"
+   $(Q)bash tests/export-tests.sh
+
 #
 # NOTE: For static compiles, you need to have all the required libs
 #  static equivalent available
diff --git a/tests/export-tests.sh b/tests/export-tests.sh
new file mode 100755
index ..d7f6ca17
--- /dev/null
+++ b/tests/export-tests.sh
@@ -0,0 +1,28 @@
+#!/bin/bash
+# export the testsuite files to a separate tar
+
+SCRIPT_DIR=$(dirname $(readlink -f "$0"))
+TESTDIR=$(basename $SCRIPT_DIR)
+FSSUM=fssum
+CORRUPT=btrfs-corrupt-block
+DESTNAME="tests.tar.gz"
+DESTDIR="."
+
+test -n "$EXPORT" && DESTDIR=$(realpath "$EXPORT")
+if [ ! -d $DESTDIR ]; then
+   echo "dest directory is not exsit."
+   exit 1
+fi
+
+DEST=$DESTDIR/$DESTNAME
+
+if [ -f $DEST ];then
+   echo "remove exsit package: " $DEST
+   rm $DEST
+fi
+
+echo "begin create tar:  " $DEST
+tar --exclude-vcs-ignores -zScf $DEST $TESTDIR $FSSUM $CORRUPT
+if [ $? -eq 0 ]; then
+   echo "create tar successfully."
+fi
-- 
2.14.3



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[RFC PATCH v2 4/4] btrfs-progs: Add readme for export testsuits

[Gu Jinxiang]

Add the readme of command for export testsuits.
And add the excute method of exported testsuits.

Signed-off-by: Gu Jinxiang 
---
 tests/README.md | 20 
 1 file changed, 20 insertions(+)

diff --git a/tests/README.md b/tests/README.md
index 04d2ce2a..9f4c01ac 100644
--- a/tests/README.md
+++ b/tests/README.md
@@ -48,6 +48,26 @@ $ TEST=001\* ./fsck-tests.sh
 will run the first test in fsck-tests subdirectory.
 
 
+## Package testsuit
+
+The tests can be export as a tests.tar.gz into path given. Use:
+
+```shell
+$ make EXPORT=PATH package
+```
+
+where `PATH` is a directory where tests.tar.gz be genereated.
+Here comes handy:
+
+```shell
+$ make EXPORT=/home/user/ package
+$ make EXPORT=../../dir/ package
+```
+
+And, after decompress tests.tar.gz, test can be run selectively
+from `tests/` directory introduced above.
+
+
 ## Test structure
 
 *tests/fsck-tests/:*
-- 
2.14.3



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 01/10] btrfs-progs: copy btrfs_del_orphan_item from kernel

[Lu Fengqi]

Add btrfs_del_orphan_item for the later subvolume undelete.

Signed-off-by: Lu Fengqi 
---
 ctree.h |  2 ++
 inode.c | 30 ++
 2 files changed, 32 insertions(+)

diff --git a/ctree.h b/ctree.h
index 17cdac76c58c..84bad186a349 100644
--- a/ctree.h
+++ b/ctree.h
@@ -2784,6 +2784,8 @@ int btrfs_unlink(struct btrfs_trans_handle *trans, struct 
btrfs_root *root,
 int btrfs_add_orphan_item(struct btrfs_trans_handle *trans,
  struct btrfs_root *root, struct btrfs_path *path,
  u64 ino);
+int btrfs_del_orphan_item(struct btrfs_trans_handle *trans,
+ struct btrfs_root *root, u64 offset);
 int btrfs_mkdir(struct btrfs_trans_handle *trans, struct btrfs_root *root,
char *name, int namelen, u64 parent_ino, u64 *ino, int mode);
 struct btrfs_root *btrfs_mksubvol(struct btrfs_root *root, const char *base,
diff --git a/inode.c b/inode.c
index 2398bca4a109..8d0812c7cf50 100644
--- a/inode.c
+++ b/inode.c
@@ -262,6 +262,36 @@ int btrfs_add_orphan_item(struct btrfs_trans_handle *trans,
return btrfs_insert_empty_item(trans, root, path, , 0);
 }
 
+int btrfs_del_orphan_item(struct btrfs_trans_handle *trans,
+ struct btrfs_root *root, u64 offset)
+{
+   struct btrfs_path *path;
+   struct btrfs_key key;
+   int ret = 0;
+
+   key.objectid = BTRFS_ORPHAN_OBJECTID;
+   key.type = BTRFS_ORPHAN_ITEM_KEY;
+   key.offset = offset;
+
+   path = btrfs_alloc_path();
+   if (!path)
+   return -ENOMEM;
+
+   ret = btrfs_search_slot(trans, root, , path, -1, 1);
+   if (ret < 0)
+   goto out;
+   if (ret) {
+   ret = -ENOENT;
+   goto out;
+   }
+
+   ret = btrfs_del_item(trans, root, path);
+
+out:
+   btrfs_free_path(path);
+   return ret;
+}
+
 /*
  * Unlink an inode, which will remove its backref and corresponding dir_index/
  * dir_item if any of them exists.
-- 
2.16.1



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 03/10] btrfs-progs: use btrfs_find_free_dir_index to find free inode index

[Lu Fengqi]

Since we have an existing function to find free inode index, we can
reuse it here.

Signed-off-by: Lu Fengqi 
---
 inode.c | 27 +++
 1 file changed, 7 insertions(+), 20 deletions(-)

diff --git a/inode.c b/inode.c
index 478036562652..86905365dfd8 100644
--- a/inode.c
+++ b/inode.c
@@ -628,7 +628,7 @@ int btrfs_link_subvol(struct btrfs_trans_handle *trans, 
struct btrfs_root *root,
struct btrfs_inode_item *inode_item;
struct extent_buffer *leaf;
struct btrfs_key key;
-   u64 index = 2;
+   u64 index;
char buf[BTRFS_NAME_LEN + 1]; /* for snprintf null */
int len;
int i;
@@ -638,28 +638,15 @@ int btrfs_link_subvol(struct btrfs_trans_handle *trans, 
struct btrfs_root *root,
if (len == 0 || len > BTRFS_NAME_LEN)
return -EINVAL;
 
-   /* find the free dir_index */
-   btrfs_init_path();
-   key.objectid = dirid;
-   key.type = BTRFS_DIR_INDEX_KEY;
-   key.offset = (u64)-1;
-
-   ret = btrfs_search_slot(NULL, root, , , 0, 0);
-   if (ret <= 0) {
-   error("search for DIR_INDEX dirid %llu failed: %d",
-   (unsigned long long)dirid, ret);
+   /* add the dir_item/dir_index */
+   ret = btrfs_find_free_dir_index(root, dirid, );
+   if (ret < 0) {
+   error("unable to find free dir index dirid %llu failed: %d",
+ (unsigned long long)dirid, ret);
goto fail;
}
 
-   if (path.slots[0] > 0) {
-   path.slots[0]--;
-   btrfs_item_key_to_cpu(path.nodes[0], , path.slots[0]);
-   if (key.objectid == dirid && key.type == BTRFS_DIR_INDEX_KEY)
-   index = key.offset + 1;
-   }
-   btrfs_release_path();
-
-   /* add the dir_item/dir_index */
+   btrfs_init_path();
key.objectid = dirid;
key.offset = 0;
key.type =  BTRFS_INODE_ITEM_KEY;
-- 
2.16.1



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 00/10] btrfs-progs: introduce btrfs rescue undelete-subvol subcommand

[Lu Fengqi]

This command will undelete all subvolumes those are still intact on the
device, and all the recovered subvolumes will link with the name "sub"
to the lost+found directory.

Lu Fengqi (10):
  btrfs-progs: copy btrfs_del_orphan_item from kernel
  btrfs-progs: extract btrfs_link_subvol from btrfs_mksubvol
  btrfs-progs: use btrfs_find_free_dir_index to find free inode index
  btrfs-progs: undelete-subvol: introduce is_subvol_intact
  btrfs-progs: undelete-subvol: introduce recover_dead_root
  btrfs-progs: undelete-subvol: introduce link_subvol_to_lostfound
  btrfs-progs: undelete-subvol: introduce btrfs_undelete_intact_subvols
  btrfs-progs: undelete-subvol: add undelete-subvol subcommand
  btrfs-progs: tests: add testcase for undelete-subvol
  btrfs-progs: undelete-subvol: update completion and documentation

 Documentation/btrfs-rescue.asciidoc|   6 +
 Makefile   |   3 +-
 btrfs-completion   |   2 +-
 cmds-rescue.c  |  47 
 convert/main.c |  57 +
 ctree.h|   8 +-
 inode.c|  99 +
 .../029-undelete-subvol/deleted_subvolume.img  | Bin 0 -> 4096 bytes
 .../029-undelete-subvol/drop_progress.raw.xz   | Bin 0 -> 23452 bytes
 tests/misc-tests/029-undelete-subvol/test.sh   |  34 +++
 undelete-subvol.c  | 244 +
 undelete-subvol.h  |  19 ++
 12 files changed, 472 insertions(+), 47 deletions(-)
 create mode 100644 tests/misc-tests/029-undelete-subvol/deleted_subvolume.img
 create mode 100644 tests/misc-tests/029-undelete-subvol/drop_progress.raw.xz
 create mode 100755 tests/misc-tests/029-undelete-subvol/test.sh
 create mode 100644 undelete-subvol.c
 create mode 100644 undelete-subvol.h

-- 
2.16.1



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 04/10] btrfs-progs: undelete-subvol: introduce is_subvol_intact

[Lu Fengqi]

The function is used to determine whether the subvolume is intact.

Signed-off-by: Lu Fengqi 
---
 Makefile  |  3 ++-
 undelete-subvol.c | 53 +
 undelete-subvol.h | 17 +
 3 files changed, 72 insertions(+), 1 deletion(-)
 create mode 100644 undelete-subvol.c
 create mode 100644 undelete-subvol.h

diff --git a/Makefile b/Makefile
index 327cdfa08eba..accbd4069ad9 100644
--- a/Makefile
+++ b/Makefile
@@ -106,7 +106,8 @@ objects = ctree.o disk-io.o kernel-lib/radix-tree.o 
extent-tree.o print-tree.o \
  qgroup.o free-space-cache.o kernel-lib/list_sort.o props.o \
  kernel-shared/ulist.o qgroup-verify.o backref.o string-table.o 
task-utils.o \
  inode.o file.o find-root.o free-space-tree.o help.o send-dump.o \
- fsfeatures.o kernel-lib/tables.o kernel-lib/raid56.o transaction.o
+ fsfeatures.o kernel-lib/tables.o kernel-lib/raid56.o transaction.o \
+ undelete-subvol.o
 cmds_objects = cmds-subvolume.o cmds-filesystem.o cmds-device.o cmds-scrub.o \
   cmds-inspect.o cmds-balance.o cmds-send.o cmds-receive.o \
   cmds-quota.o cmds-qgroup.o cmds-replace.o check/main.o \
diff --git a/undelete-subvol.c b/undelete-subvol.c
new file mode 100644
index ..00fcc4895778
--- /dev/null
+++ b/undelete-subvol.c
@@ -0,0 +1,53 @@
+/*
+ * Copyright (C) 2018 Fujitsu.  All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public
+ * License v2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ */
+
+#include "ctree.h"
+
+/*
+ * Determines whether the subvolume is intact, according to the drop_progress
+ * of the root item specified by subvol_id.
+ *
+ * Return true if the subvolume is intact, otherwise return false.
+ */
+static bool is_subvol_intact(struct btrfs_root *root, u64 subvol_id)
+{
+   struct btrfs_path path;
+   struct btrfs_key key;
+   struct extent_buffer *leaf;
+   struct btrfs_root_item root_item;
+   u64 offset;
+   int ret;
+
+   key.objectid = subvol_id;
+   key.type = BTRFS_ROOT_ITEM_KEY;
+   key.offset = 0;
+
+   btrfs_init_path();
+   ret = btrfs_search_slot(NULL, root, , , 0, 0);
+   if (ret) {
+   ret = false;
+   goto out;
+   }
+
+   leaf = path.nodes[0];
+   offset = btrfs_item_ptr_offset(leaf, path.slots[0]);
+
+   read_extent_buffer(leaf, _item, offset, sizeof(root_item));
+
+   /* the subvolume is intact if the objectid of drop_progress == 0 */
+   ret = btrfs_disk_key_objectid(_item.drop_progress) ? false : true;
+
+out:
+   btrfs_release_path();
+   return ret;
+}
diff --git a/undelete-subvol.h b/undelete-subvol.h
new file mode 100644
index ..7cfd100cce37
--- /dev/null
+++ b/undelete-subvol.h
@@ -0,0 +1,17 @@
+/*
+ * Copyright (C) 2018 Fujitsu.  All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public
+ * License v2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ */
+
+#ifndef __BTRFS_UNDELETE_SUBVOLUME_H__
+#define __BTRFS_UNDELETE_SUBVOLUME_H__
+
+#endif
-- 
2.16.1



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 05/10] btrfs-progs: undelete-subvol: introduce recover_dead_root

[Lu Fengqi]

The function will find the root_item specified by the subvol_id,
clear the BTRFS_ROOT_SUBVOL_DEAD flag and set root_refs to one.

Signed-off-by: Lu Fengqi 
---
 ctree.h   |  1 +
 undelete-subvol.c | 55 +++
 2 files changed, 56 insertions(+)

diff --git a/ctree.h b/ctree.h
index 18f7c79ddeec..3df062cc7c25 100644
--- a/ctree.h
+++ b/ctree.h
@@ -185,6 +185,7 @@ static int btrfs_csum_sizes[] = { 4 };
 #define BTRFS_FT_MAX   9
 
 #define BTRFS_ROOT_SUBVOL_RDONLY   (1ULL << 0)
+#define BTRFS_ROOT_SUBVOL_DEAD (1ULL << 48)
 
 /*
  * the key defines the order in the tree, and so it also defines (optimal)
diff --git a/undelete-subvol.c b/undelete-subvol.c
index 00fcc4895778..781057df2b84 100644
--- a/undelete-subvol.c
+++ b/undelete-subvol.c
@@ -12,6 +12,9 @@
  */
 
 #include "ctree.h"
+#include "transaction.h"
+#include "disk-io.h"
+#include "messages.h"
 
 /*
  * Determines whether the subvolume is intact, according to the drop_progress
@@ -51,3 +54,55 @@ out:
btrfs_release_path();
return ret;
 }
+
+/*
+ * Clear BTRFS_ROOT_SUBVOL_DEAD flag and set the root_refs to one.
+ *
+ * @root   the root of root tree.
+ * @subvol_id  specify the root_item which will be modified.
+ *
+ * Return 0 if no error occurred.
+ */
+static int recover_dead_root(struct btrfs_trans_handle *trans,
+struct btrfs_root *root, u64 subvol_id)
+{
+   struct btrfs_key key;
+   struct btrfs_path path;
+   struct extent_buffer *leaf;
+   struct btrfs_root_item root_item;
+   u64 root_flags;
+   u64 offset;
+   int ret;
+
+   key.objectid = subvol_id;
+   key.type = BTRFS_ROOT_ITEM_KEY;
+   key.offset = 0;
+
+   btrfs_init_path();
+   ret = btrfs_search_slot(trans, root, , , 0, 0);
+   if (ret) {
+   error("couldn't find ROOT_ITEM for %llu failed: %d",
+   subvol_id, ret);
+   goto out;
+   }
+
+   leaf = path.nodes[0];
+
+   offset = btrfs_item_ptr_offset(leaf, path.slots[0]);
+   read_extent_buffer(leaf, _item, offset, sizeof(root_item));
+
+   /* Clear BTRFS_ROOT_SUBVOL_DEAD */
+   root_flags = btrfs_root_flags(_item);
+   btrfs_set_root_flags(_item,
+root_flags & ~BTRFS_ROOT_SUBVOL_DEAD);
+
+   /* Increase the refs */
+   btrfs_set_root_refs(_item, 1);
+
+   write_extent_buffer(leaf, _item, offset, sizeof(root_item));
+   btrfs_mark_buffer_dirty(leaf);
+
+out:
+   btrfs_release_path();
+   return ret;
+}
-- 
2.16.1



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 07/10] btrfs-progs: undelete-subvol: introduce btrfs_undelete_intact_subvols

[Lu Fengqi]

The function will traverse the all orphan items on the tree root, and
recover the all intact subvolumes.

Signed-off-by: Lu Fengqi 
---
 undelete-subvol.c | 60 +++
 undelete-subvol.h |  2 ++
 2 files changed, 62 insertions(+)

diff --git a/undelete-subvol.c b/undelete-subvol.c
index 9243e35545c5..b0852510a5c2 100644
--- a/undelete-subvol.c
+++ b/undelete-subvol.c
@@ -15,6 +15,7 @@
 #include "transaction.h"
 #include "disk-io.h"
 #include "messages.h"
+#include "undelete-subvol.h"
 
 /*
  * Determines whether the subvolume is intact, according to the drop_progress
@@ -182,3 +183,62 @@ static int link_subvol_to_lostfound(struct btrfs_root 
*root, u64 subvol_id)
 out:
return ret;
 }
+
+/*
+ * Traverse all orphan items on the root tree, restore them to the lost+found
+ * directory if the corresponding subvolumes are still intact left on the disk.
+ *
+ * @root   the root of the root tree.
+ *
+ * Return 0 if no error occurred.
+ */
+int btrfs_undelete_intact_subvols(struct btrfs_root *root)
+{
+   struct btrfs_key key;
+   struct btrfs_path path;
+   u64 found_count = 0;
+   u64 recovered_count = 0;
+   int ret;
+
+   key.objectid = BTRFS_ORPHAN_OBJECTID;
+   key.type = BTRFS_ORPHAN_ITEM_KEY;
+   key.offset = (u64)-1;
+
+   btrfs_init_path();
+   while (1) {
+   ret = btrfs_search_slot(NULL, root, , , 0, 0);
+   if (ret < 0) {
+   error("search ORPHAN_ITEM for %llu failed.\n",
+ key.offset);
+   break;
+   }
+
+   path.slots[0]--;
+   btrfs_item_key_to_cpu(path.nodes[0], , path.slots[0]);
+
+   /* No more BTRFS_ORPHAN_ITEM, so we don't need to continue. */
+   if (key.type != BTRFS_ORPHAN_ITEM_KEY) {
+   ret = 0;
+   break;
+   }
+
+   if (is_subvol_intact(root, key.offset)) {
+   /* Here we can confirm there is an intact subvolume. */
+   found_count++;
+   ret = link_subvol_to_lostfound(root, key.offset);
+   if (ret == 0) {
+   recovered_count++;
+   printf(
+   "Recovered subvolume %llu to lost+found successfully.\n",
+   key.offset);
+   }
+   }
+
+   btrfs_release_path();
+   }
+
+   printf("Found %llu subvols left intact\n", found_count);
+   printf("Recovered %llu subvols\n", found_count);
+   btrfs_release_path();
+   return ret;
+}
diff --git a/undelete-subvol.h b/undelete-subvol.h
index 7cfd100cce37..5ceb034beaf3 100644
--- a/undelete-subvol.h
+++ b/undelete-subvol.h
@@ -14,4 +14,6 @@
 #ifndef __BTRFS_UNDELETE_SUBVOLUME_H__
 #define __BTRFS_UNDELETE_SUBVOLUME_H__
 
+int btrfs_undelete_intact_subvols(struct btrfs_root *root);
+
 #endif
-- 
2.16.1



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 3/7] btrfs-progs: Make btrfs_alloc_chunk to handle block group creation

[Qu Wenruo]

Before this patch, chunk allocation is split into 2 parts:

1) Chunk allocation
   Handled by btrfs_alloc_chunk(), which will insert chunk and device
   extent items.

2) Block group allocation
   Handled by btrfs_make_block_group(), which will insert block group
   item and update space info.

However for chunk allocation, we don't really need to split these
operations as all btrfs_alloc_chunk() has btrfs_make_block_group()
followed.

So it's reasonable to merge btrfs_make_block_group() call into
btrfs_alloc_chunk() to save several lines, and provides the basis for
later btrfs_alloc_chunk() rework.

Signed-off-by: Qu Wenruo 
---
 convert/main.c |  4 
 extent-tree.c  | 10 ++
 mkfs/main.c| 19 ---
 volumes.c  | 10 ++
 4 files changed, 8 insertions(+), 35 deletions(-)

diff --git a/convert/main.c b/convert/main.c
index b2444bb2ff21..240d3aa46db9 100644
--- a/convert/main.c
+++ b/convert/main.c
@@ -915,10 +915,6 @@ static int make_convert_data_block_groups(struct 
btrfs_trans_handle *trans,
BTRFS_BLOCK_GROUP_DATA, true);
if (ret < 0)
break;
-   ret = btrfs_make_block_group(trans, fs_info, 0,
-   BTRFS_BLOCK_GROUP_DATA, cur, len);
-   if (ret < 0)
-   break;
cur += len;
}
}
diff --git a/extent-tree.c b/extent-tree.c
index b085ab0352b3..bccd83d1bae6 100644
--- a/extent-tree.c
+++ b/extent-tree.c
@@ -1909,15 +1909,9 @@ static int do_chunk_alloc(struct btrfs_trans_handle 
*trans,
space_info->flags, false);
if (ret == -ENOSPC) {
space_info->full = 1;
-   return 0;
+   return ret;
}
-
-   BUG_ON(ret);
-
-   ret = btrfs_make_block_group(trans, fs_info, 0, space_info->flags,
-start, num_bytes);
-   BUG_ON(ret);
-   return 0;
+   return ret;
 }
 
 static int update_block_group(struct btrfs_root *root,
diff --git a/mkfs/main.c b/mkfs/main.c
index 358395ca0250..49159ea533b9 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -87,12 +87,6 @@ static int create_metadata_block_groups(struct btrfs_root 
*root, int mixed,
error("no space to allocate data/metadata chunk");
goto err;
}
-   if (ret)
-   return ret;
-   ret = btrfs_make_block_group(trans, fs_info, 0,
-BTRFS_BLOCK_GROUP_METADATA |
-BTRFS_BLOCK_GROUP_DATA,
-chunk_start, chunk_size);
if (ret)
return ret;
allocation->mixed += chunk_size;
@@ -106,12 +100,7 @@ static int create_metadata_block_groups(struct btrfs_root 
*root, int mixed,
}
if (ret)
return ret;
-   ret = btrfs_make_block_group(trans, fs_info, 0,
-BTRFS_BLOCK_GROUP_METADATA,
-chunk_start, chunk_size);
allocation->metadata += chunk_size;
-   if (ret)
-   return ret;
}
 
root->fs_info->system_allocs = 0;
@@ -140,12 +129,7 @@ static int create_data_block_groups(struct 
btrfs_trans_handle *trans,
}
if (ret)
return ret;
-   ret = btrfs_make_block_group(trans, fs_info, 0,
-BTRFS_BLOCK_GROUP_DATA,
-chunk_start, chunk_size);
allocation->data += chunk_size;
-   if (ret)
-   return ret;
}
 
 err:
@@ -249,9 +233,6 @@ static int create_one_raid_group(struct btrfs_trans_handle 
*trans,
if (ret)
return ret;
 
-   ret = btrfs_make_block_group(trans, fs_info, 0,
-type, chunk_start, chunk_size);
-
type &= BTRFS_BLOCK_GROUP_TYPE_MASK;
if (type == BTRFS_BLOCK_GROUP_DATA) {
allocation->data += chunk_size;
diff --git a/volumes.c b/volumes.c
index 9ee4650351c3..a9dc8c939dc5 100644
--- a/volumes.c
+++ b/volumes.c
@@ -837,10 +837,9 @@ error:
/ sizeof(struct btrfs_stripe) + 1)
 
 /*
- * Alloc a chunk, will insert dev extents, chunk item.
- * NOTE: This function will not insert block group item nor mark newly
- * allocated chunk available for later allocation.
- * Block group item and free space update is handled by 
btrfs_make_block_group()
+ * Alloc a chunk, will insert dev extents, chunk item, and insert new
+ * block group and update space info (so 

[PATCH 6/7] btrfs-progs: kernel-lib: Port kernel sort() to btrfs-progs

[Qu Wenruo]

Used by later btrfs_alloc_chunk() rework.

Signed-off-by: Qu Wenruo 
---
 Makefile  |   3 +-
 kernel-lib/sort.c | 104 ++
 kernel-lib/sort.h |  16 +
 3 files changed, 122 insertions(+), 1 deletion(-)
 create mode 100644 kernel-lib/sort.c
 create mode 100644 kernel-lib/sort.h

diff --git a/Makefile b/Makefile
index 327cdfa08eba..3db7bbe04662 100644
--- a/Makefile
+++ b/Makefile
@@ -106,7 +106,8 @@ objects = ctree.o disk-io.o kernel-lib/radix-tree.o 
extent-tree.o print-tree.o \
  qgroup.o free-space-cache.o kernel-lib/list_sort.o props.o \
  kernel-shared/ulist.o qgroup-verify.o backref.o string-table.o 
task-utils.o \
  inode.o file.o find-root.o free-space-tree.o help.o send-dump.o \
- fsfeatures.o kernel-lib/tables.o kernel-lib/raid56.o transaction.o
+ fsfeatures.o kernel-lib/tables.o kernel-lib/raid56.o transaction.o \
+ kernel-lib/sort.o
 cmds_objects = cmds-subvolume.o cmds-filesystem.o cmds-device.o cmds-scrub.o \
   cmds-inspect.o cmds-balance.o cmds-send.o cmds-receive.o \
   cmds-quota.o cmds-qgroup.o cmds-replace.o check/main.o \
diff --git a/kernel-lib/sort.c b/kernel-lib/sort.c
new file mode 100644
index ..70ae3dbe2852
--- /dev/null
+++ b/kernel-lib/sort.c
@@ -0,0 +1,104 @@
+/*
+ * taken from linux kernel lib/sort.c, removed kernel config code and adapted
+ * for btrfsprogs
+ */
+
+#include "sort.h"
+
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * A fast, small, non-recursive O(nlog n) sort for the Linux kernel
+ *
+ * Jan 23 2005  Matt Mackall 
+ */
+
+static int alignment_ok(const void *base, int align)
+{
+   return ((unsigned long)base & (align - 1)) == 0;
+}
+
+static void u32_swap(void *a, void *b, int size)
+{
+   u32 t = *(u32 *)a;
+   *(u32 *)a = *(u32 *)b;
+   *(u32 *)b = t;
+}
+
+static void u64_swap(void *a, void *b, int size)
+{
+   u64 t = *(u64 *)a;
+   *(u64 *)a = *(u64 *)b;
+   *(u64 *)b = t;
+}
+
+static void generic_swap(void *a, void *b, int size)
+{
+   char t;
+
+   do {
+   t = *(char *)a;
+   *(char *)a++ = *(char *)b;
+   *(char *)b++ = t;
+   } while (--size > 0);
+}
+
+/**
+ * sort - sort an array of elements
+ * @base: pointer to data to sort
+ * @num: number of elements
+ * @size: size of each element
+ * @cmp_func: pointer to comparison function
+ * @swap_func: pointer to swap function or NULL
+ *
+ * This function does a heapsort on the given array. You may provide a
+ * swap_func function optimized to your element type.
+ *
+ * Sorting time is O(n log n) both on average and worst-case. While
+ * qsort is about 20% faster on average, it suffers from exploitable
+ * O(n*n) worst-case behavior and extra memory requirements that make
+ * it less suitable for kernel use.
+ */
+
+void sort(void *base, size_t num, size_t size,
+ int (*cmp_func)(const void *, const void *),
+ void (*swap_func)(void *, void *, int size))
+{
+   /* pre-scale counters for performance */
+   int i = (num/2 - 1) * size, n = num * size, c, r;
+
+   if (!swap_func) {
+   if (size == 4 && alignment_ok(base, 4))
+   swap_func = u32_swap;
+   else if (size == 8 && alignment_ok(base, 8))
+   swap_func = u64_swap;
+   else
+   swap_func = generic_swap;
+   }
+
+   /* heapify */
+   for ( ; i >= 0; i -= size) {
+   for (r = i; r * 2 + size < n; r  = c) {
+   c = r * 2 + size;
+   if (c < n - size &&
+   cmp_func(base + c, base + c + size) < 0)
+   c += size;
+   if (cmp_func(base + r, base + c) >= 0)
+   break;
+   swap_func(base + r, base + c, size);
+   }
+   }
+
+   /* sort */
+   for (i = n - size; i > 0; i -= size) {
+   swap_func(base, base + i, size);
+   for (r = 0; r * 2 + size < i; r = c) {
+   c = r * 2 + size;
+   if (c < i - size &&
+   cmp_func(base + c, base + c + size) < 0)
+   c += size;
+   if (cmp_func(base + r, base + c) >= 0)
+   break;
+   swap_func(base + r, base + c, size);
+   }
+   }
+}
diff --git a/kernel-lib/sort.h b/kernel-lib/sort.h
new file mode 100644
index ..9355e01248f2
--- /dev/null
+++ b/kernel-lib/sort.h
@@ -0,0 +1,16 @@
+/*
+ * taken from linux kernel include/list_sort.h
+ * changed include to kerncompat.h
+ */
+
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_SORT_H
+#define _LINUX_SORT_H
+
+#include "kerncompat.h"
+
+void sort(void *base, 

[PATCH 02/10] btrfs-progs: extract btrfs_link_subvol from btrfs_mksubvol

[Lu Fengqi]

The original btrfs_mksubvol is too specific to specify the directory that
the subvolume will link to. Furthermore, in this transaction, we don't only
need to create root_ref/dir-item, but also update the refs or flags of
root_item. Extract a generic btrfs_link_subvol that allow the caller pass a
trans argument for later subvolume undelete.

No functional changes for the btrfs_mksubvol.

Signed-off-by: Lu Fengqi 
---
 convert/main.c | 57 +
 ctree.h|  5 +++--
 inode.c| 46 +-
 3 files changed, 81 insertions(+), 27 deletions(-)

diff --git a/convert/main.c b/convert/main.c
index b3ea31d7af43..e997bdb822c5 100644
--- a/convert/main.c
+++ b/convert/main.c
@@ -1002,6 +1002,63 @@ err:
return ret;
 }
 
+/*
+ * Link the subvolume specified by @root_objectid to the root_dir of @root.
+ *
+ * @root   the root of the file tree which the subvolume will
+ * be linked to.
+ * @subvol_namethe name of the subvolume which will be linked.
+ * @root_objectid  specify the subvolume which will be linked.
+ * @convertthe flag to determine whether to try to resolve
+ * the name conflict.
+ *
+ * Return the root of the subvolume if success, otherwise return NULL.
+ */
+static struct btrfs_root *btrfs_mksubvol(struct btrfs_root *root,
+const char *subvol_name,
+u64 root_objectid,
+bool convert)
+{
+   struct btrfs_trans_handle *trans;
+   struct btrfs_root *subvol_root = NULL;
+   struct btrfs_key key;
+   u64 dirid = btrfs_root_dirid(>root_item);
+   int ret;
+
+
+   trans = btrfs_start_transaction(root, 1);
+   if (IS_ERR(trans)) {
+   error("unable to start transaction");
+   goto fail;
+   }
+
+   ret = btrfs_link_subvol(trans, root, subvol_name, root_objectid, dirid,
+   true);
+   if (ret) {
+   error("unable to link subvolume %s", subvol_name);
+   goto fail;
+   }
+
+   ret = btrfs_commit_transaction(trans, root);
+   if (ret) {
+   error("transaction commit failed: %d", ret);
+   goto fail;
+   }
+
+   key.objectid = root_objectid;
+   key.offset = (u64)-1;
+   key.type = BTRFS_ROOT_ITEM_KEY;
+
+   subvol_root = btrfs_read_fs_root(root->fs_info, );
+   if (!subvol_root) {
+   error("unable to link subvolume %s", subvol_name);
+   goto fail;
+   }
+
+fail:
+   return subvol_root;
+}
+
 /*
  * Migrate super block to its default position and zero 0 ~ 16k
  */
diff --git a/ctree.h b/ctree.h
index 84bad186a349..18f7c79ddeec 100644
--- a/ctree.h
+++ b/ctree.h
@@ -2788,8 +2788,9 @@ int btrfs_del_orphan_item(struct btrfs_trans_handle 
*trans,
  struct btrfs_root *root, u64 offset);
 int btrfs_mkdir(struct btrfs_trans_handle *trans, struct btrfs_root *root,
char *name, int namelen, u64 parent_ino, u64 *ino, int mode);
-struct btrfs_root *btrfs_mksubvol(struct btrfs_root *root, const char *base,
- u64 root_objectid, bool convert);
+int btrfs_link_subvol(struct btrfs_trans_handle *trans, struct btrfs_root 
*root,
+ const char *base, u64 root_objectid, u64 dirid,
+ bool convert);
 
 /* file.c */
 int btrfs_get_extent(struct btrfs_trans_handle *trans,
diff --git a/inode.c b/inode.c
index 8d0812c7cf50..478036562652 100644
--- a/inode.c
+++ b/inode.c
@@ -606,19 +606,28 @@ out:
return ret;
 }
 
-struct btrfs_root *btrfs_mksubvol(struct btrfs_root *root,
- const char *base, u64 root_objectid,
- bool convert)
+/*
+ * Link the subvolume specified by @root_objectid to the directory specified by
+ * @dirid on the file tree specified by @root.
+ *
+ * @root   the root of the file tree where the directory on.
+ * @base   the name of the subvolume which will be linked.
+ * @root_objectid  specify the subvolume which will be linked.
+ * @dirid  specify the directory which the subvolume will be
+ * linked to.
+ * @convertthe flag to determine whether to try to resolve
+ * the name conflict.
+ */
+int btrfs_link_subvol(struct btrfs_trans_handle *trans, struct btrfs_root 
*root,
+ const char *base, u64 root_objectid, u64 dirid,
+ bool convert)
 {
-   struct btrfs_trans_handle *trans;
struct btrfs_fs_info *fs_info = root->fs_info;
struct btrfs_root *tree_root = fs_info->tree_root;
-   struct btrfs_root *new_root = NULL;
struct btrfs_path path;
struct 

[PATCH 09/10] btrfs-progs: tests: add testcase for undelete-subvol

[Lu Fengqi]

The testcase checks the functionality of "btrfs rescue undelete-subvol",
including recovering an intact subvolume, and handling correctly
incomplete subvolume.

Signed-off-by: Lu Fengqi 
---
 .../029-undelete-subvol/deleted_subvolume.img  | Bin 0 -> 4096 bytes
 .../029-undelete-subvol/drop_progress.raw.xz   | Bin 0 -> 23452 bytes
 tests/misc-tests/029-undelete-subvol/test.sh   |  34 +
 3 files changed, 34 insertions(+)
 create mode 100644 tests/misc-tests/029-undelete-subvol/deleted_subvolume.img
 create mode 100644 tests/misc-tests/029-undelete-subvol/drop_progress.raw.xz
 create mode 100755 tests/misc-tests/029-undelete-subvol/test.sh

diff --git a/tests/misc-tests/029-undelete-subvol/deleted_subvolume.img 
b/tests/misc-tests/029-undelete-subvol/deleted_subvolume.img
new file mode 100644
index 
..9168a8e33fbbb14f9ea78514721a52191b94d1a0
GIT binary patch
literal 4096
zcmeH|do+|=AIB%x8p^0djl0}JQb@$O%TY$;lEx)vlF+ymV;WO45{D?%IqnhB$S}Fi
zjN6EihKkC4NQhyKaT(V$2k-mO={@KF_pD{Fy}r-)`R-@^*504B_p@W+PlCQF!fF8j
zZGs!m9b0|F@NPJOvGJw?_&4=#{gw~i3;@(EFS(@+TiUf5-~Erz<=ODYja#|MmTnGi
zw`~I31pbc*g!Km3O1HeZD~KRV`{Pj~uzt3r2|UgTH}a
zWDkSP-%;<0K6oq%LMk0=5)g}2I{;`k(RwtnGuI_hB01WRSAongy!#b`_p*d$38#+;
zcYI;9gzxul-@Q0?=XZCRp1a_O(t&`~#!SFI<>VfKQp$%5D%q@qRO^IYiZQ11
z*LJ*O@akHeln2DGHANbna7)texSTKS8w%9X2EBg2OP_*;ZbHjccM=V#lO%}
zATc9#siZ*izH<@v{m4ksm!$X!TLEok)EiGD1DIiBhF{t3@g30ydV%CQ75thpDR_Sz
zkNpn7>SYJJu-@rDqHlw-Wb-Wdq8i)|MhOAQWLvnU+THRaIvD@>=47X$XKjKl#3c>c
z+gVL~u-1qB(5Zq5RlSf2!q|mGkUWzvayXPkcZ+}`@>>iAb(MsWDCeg9_g?z@
z+UzN}@~(J8&7SGm+x+QKwvX*meP^nxg%)4Y+tC!c%#Li1uw)!=uIZLYL%h({>6
zI?6W8VFdIKTctfwulDqceyAL)1|1Zhpij6oC^|RGxbla6`S|q7kxq3u8hQMPc;I2Y
zCWZ`c;*ciOjl@3Y^DqdF9BWsR_*&1bIYVLtMwmWM2N+tNJ-7j(O%B-L35XizYGSd1
zB~$y~k-c18T=w!oFaLH~OTx)ncn*m3#tpvpB({MHrvM%M=n2J!-i}LGsv{?
z<~5bwAY@lJ)i%#aoOp^!R$6CJL^-x?kT(roP@)*Q8XnwB2j3pd-#ynd#jp7WCx#4WSfc|{y
z#5H*01m+|Z94ps6)UYtvKCre(ZF$dTAxE?$+j2>nxqYpZ_Rkl)A)IQAwvH~^(SzZi
z0p9u3gpBilS%_h>xu4tFT+N#=O{4QmFL|iRrJJjnU7WSMuKUyHtOvks7;`zGw0BVH
zJQ}jPdQ!xy(s!Us?OIYAl7nvL$7V~c{(G8ho_TID+~?_UeBFyGcNvvctBnsrrf
zR>%@OeHtPSWh}(@*~6Mo%wR1Xc{kY<0UPWj-p+;|;pI=V<}0+sd36pSJ9HL1GOmQHYWmh`5IF7C{XLOE6~i$Hfktz9xZp$KS@J7FBer
zU6XsQh2^|0*VWg^M;!+bUUhGGt|)c?CClbV{)#jL~#x7@Hgi$!??u)-rbQ^;oY
zY!$-MsH;KQ%fzKtwafY8t$(=9mvBoO{a>gHD^*bc*g7T=1Gzb*{j2yz>ok&
z7M6gGH4Q*~983Vsv#8?bD~SiDobt+CdWEqjraBk8ESoH7vV6%oI3Ejb*$
zuNVEyMF_j+Q>(9d(j#nI_W+HW?IqPn%^pKydSPdh>>Y3s5VIg;-RjrS*7CMI7{u~HVW_{Y*$se=Z}$CDIdvvKrhnca(1M-26(9HuWE?Ohji
zuBpmHt(QTgQ-Yq{m73~S!#xX-w!hB|`k?xhx6>;m!B6zHz=2>h)6=4D_a-K3kJ6t;
zWMpX8X0U`fZB*msQ+^4vT8hol-6nS`h35;Ds!prmWNZ?5xhs3K9QX}xIY9op0XE<2
zd-c9BbluK;C2_kTap|WlT=|N;-RvRtrenZQU
zPiVGB&=zwO#labtRhCASg?G_h$wLNDK8)U6ui@n~0zl=OVap45dP71-m(j?er4a{gMzE>X8BhF^HR&+yN
z``L#iGzIqve?b}mDKJ3R?!Sj~hEkClMj*aQPHs-je`Hw)G=a^_UGlEFuf?-JGn
zrt3Oz*O1f$_uqL2)~`vD^hS#j3mRtrVnrHk6fM)Az8Q5M9FVc;SjPt^dWinLHEw
z(Yd%niJ8}8Uk@IFxs`OJKq)!4{LOe!xf}C
zCCb1~=m9Lg$9ZHABR*mWv80B!dG3fdJeMV!XUBJIiz)c*3_^%D@4@C+zdY>o!34x-
zf=~s)a%m<7<=N$mEk3g{K0aS)5Q~o%4(9R>QL7W2uspH*KopFWYyiKEB0WdA*o`T(uUGBdW_*E
zzTw9jeaVG?gtI%=*AAzOt@~-PogS@pCRs>(P<0-IA0-=um{T^q#LWS2+;ZNGGoXTU+*0WG?kjb!RY9Rd@zE!
zprWN%HWIB*2GqTt(OQjR79XdXGC|eojjqRj@Ntje7V=+!*iK3
ze=gkgvO#(Wb3wy1yH#b!yCt$!YIsJE^$1(}*0xh|G%dc{X8vwN4zHQ}bVmsRRrWRKCJQwJZ4r;|fIzaNqWP$xtDSB^E`4R#+kNfF#ZlImmLpgm-*(wek`j)@7COOkP^)r&;tzG%wF$uJ>jkzy^dj
z*q1CnCUcRGf<3#$dF~&$F~=`u6%?zfjrjSgZ#=WTN%JruO@f4|;_Itq@hv``
znjt?-*%=Zcor6q8k0n~uoI_Sjmizo#?Qx)NGtnCjRYcZ6yKf~=j+^wl)tqQ;;UoIe
z6dYQm`k}lBct=xCfMBv`>_%x+lk+Tb7FuRhD^!2Cr(cdZ=3Hg+ski3Y-op
zFEWTlcBk^eKNL|WnH>4v24}=}Oi<&5mq8M6PBQuKPF}pW3#$g67be*!O|OI5jBm@GHp1l$d3pM0hxL^?iZNjay1ELxtIk(9q%;7k{=WOq
zVcqtGQXM9N6owoF$=%Ja6A{~D@{W^FvI{VZ9!L(Uc`zt6i4mv(BPmaC9yTV{>?PC1dsJG
z4AQ!27bo-Ofq5a~mLqp4hPsA?n;ise0$mkdvDwg^$n=EO81vd*1PIXPHN*)(Tyl%q
zOt)_gg-@hrLba(Mc9JL}fC;bOb?$U|cn^Ofk)*8B

[PATCH 08/10] btrfs-progs: undelete-subvol: add undelete-subvol subcommand

[Lu Fengqi]

Add the undelete-subvol subcommand for btrfs rescue. This subcommand is
used to recover all deleted subvolume left intact on the device.

Signed-off-by: Lu Fengqi 
---
 cmds-rescue.c | 47 +++
 1 file changed, 47 insertions(+)

diff --git a/cmds-rescue.c b/cmds-rescue.c
index c40088ad374e..17bc1df33a23 100644
--- a/cmds-rescue.c
+++ b/cmds-rescue.c
@@ -25,6 +25,7 @@
 #include "disk-io.h"
 #include "commands.h"
 #include "utils.h"
+#include "undelete-subvol.h"
 #include "help.h"
 
 static const char * const rescue_cmd_group_usage[] = {
@@ -248,6 +249,50 @@ out:
return !!ret;
 }
 
+static const char * const cmd_rescue_undelete_subvol_usage[] = {
+   "btrfs rescue undelete-subvol ",
+   "Undelete all deleted subvolume that still left intact on the device",
+   "",
+   NULL
+};
+
+static int cmd_rescue_undelete_subvol(int argc, char **argv)
+{
+   struct btrfs_fs_info *fs_info;
+   char *devname;
+   int ret;
+
+   clean_args_no_options(argc, argv, cmd_rescue_undelete_subvol_usage);
+
+   if (check_argc_exact(argc, 2))
+   usage(cmd_rescue_undelete_subvol_usage);
+
+   devname = argv[optind];
+   ret = check_mounted(devname);
+   if (ret < 0) {
+   error("could not check mount status: %s", strerror(-ret));
+   goto out;
+   } else if (ret) {
+   error("%s is currently mounted", devname);
+   ret = -EBUSY;
+   goto out;
+   }
+
+   fs_info = open_ctree_fs_info(devname, 0, 0, 0, OPEN_CTREE_WRITES |
+OPEN_CTREE_PARTIAL);
+   if (!fs_info) {
+   error("could not open btrfs");
+   ret = -EIO;
+   goto out;
+   }
+
+   ret = btrfs_undelete_intact_subvols(fs_info->tree_root);
+
+   close_ctree(fs_info->tree_root);
+out:
+   return ret;
+}
+
 static const char rescue_cmd_group_info[] =
 "toolbox for specific rescue operations";
 
@@ -260,6 +305,8 @@ const struct cmd_group rescue_cmd_group = {
{ "zero-log", cmd_rescue_zero_log, cmd_rescue_zero_log_usage, 
NULL, 0},
{ "fix-device-size", cmd_rescue_fix_device_size,
cmd_rescue_fix_device_size_usage, NULL, 0},
+   { "undelete-subvol", cmd_rescue_undelete_subvol,
+   cmd_rescue_undelete_subvol_usage, NULL, 0},
NULL_CMD_STRUCT
}
 };
-- 
2.16.1



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 06/10] btrfs-progs: undelete-subvol: introduce link_subvol_to_lostfound

[Lu Fengqi]

The function will create lost+found directory, link the deleted
subvolume specified by the subvol_id to the directory, update the
information of root_item and cleanup the associated orphan item.

Signed-off-by: Lu Fengqi 
---
 undelete-subvol.c | 76 +++
 1 file changed, 76 insertions(+)

diff --git a/undelete-subvol.c b/undelete-subvol.c
index 781057df2b84..9243e35545c5 100644
--- a/undelete-subvol.c
+++ b/undelete-subvol.c
@@ -106,3 +106,79 @@ out:
btrfs_release_path();
return ret;
 }
+
+/*
+ * Recover a subvolume specified by subvol_id, and link it to the lost+found
+ * directory.
+ *
+ * @root: the root of the root tree.
+ * @subvol_id: specify the subvolume which will be linked, and also be the part
+ * of the subvolume name.
+ *
+ * Return 0 if no error occurred.
+ */
+static int link_subvol_to_lostfound(struct btrfs_root *root, u64 subvol_id)
+{
+   struct btrfs_trans_handle *trans;
+   struct btrfs_fs_info *fs_info = root->fs_info;
+   struct btrfs_root *fs_root = fs_info->fs_root;
+   char buf[BTRFS_NAME_LEN + 1] = {0}; /* 1 for snprintf null */
+   char *dir_name = "lost+found";
+   u64 lost_found_ino = 0;
+   u32 mode = 0700;
+   int ret;
+
+   /*
+* For link subvolume to lost+found,
+* 2 for parent(256)'s dir_index and dir_item
+* 2 for lost+found dir's inode_item and inode_ref
+* 2 for lost+found dir's dir_index and dir_item for the subvolume
+* 2 for the subvolume's root_ref and root_backref
+*/
+   trans = btrfs_start_transaction(fs_root, 8);
+   if (IS_ERR(trans)) {
+   error("unable to start transaction");
+   ret = PTR_ERR(trans);
+   goto out;
+   }
+
+   /* Create lost+found directory */
+   ret = btrfs_mkdir(trans, fs_root, dir_name, strlen(dir_name),
+ BTRFS_FIRST_FREE_OBJECTID, _found_ino,
+ mode);
+   if (ret < 0) {
+   error("failed to create '%s' dir: %d", dir_name, ret);
+   goto out;
+   }
+
+   /* Link the subvolume to lost+found directory */
+   snprintf(buf, BTRFS_NAME_LEN + 1, "sub%llu", subvol_id);
+   ret = btrfs_link_subvol(trans, fs_root, buf, subvol_id, lost_found_ino,
+   false);
+   if (ret) {
+   error("failed to link the subvol %llu: %d", subvol_id, ret);
+   goto out;
+   }
+
+   /* Clear root flags BTRFS_ROOT_SUBVOL_DEAD and increase root refs */
+   ret = recover_dead_root(trans, root, subvol_id);
+   if (ret)
+   goto out;
+
+   /* Delete the orphan item after undeletion is completed. */
+   ret = btrfs_del_orphan_item(trans, root, subvol_id);
+   if (ret) {
+   error("failed to delete the orphan_item for %llu: %d",
+   subvol_id, ret);
+   goto out;
+   }
+
+   ret = btrfs_commit_transaction(trans, fs_root);
+   if (ret) {
+   error("transaction commit failed: %d", ret);
+   goto out;
+   }
+
+out:
+   return ret;
+}
-- 
2.16.1



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 10/10] btrfs-progs: undelete-subvol: update completion and documentation

[Lu Fengqi]

Add undelete-subvol to btrfs-completion, and update btrfs-rescue
documentation to introduce undelete-subvol.

Signed-off-by: Lu Fengqi 
---
 Documentation/btrfs-rescue.asciidoc | 6 ++
 btrfs-completion| 2 +-
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/Documentation/btrfs-rescue.asciidoc 
b/Documentation/btrfs-rescue.asciidoc
index 743a23a64287..9c6fbd82ef2d 100644
--- a/Documentation/btrfs-rescue.asciidoc
+++ b/Documentation/btrfs-rescue.asciidoc
@@ -60,6 +60,12 @@ assume an answer of 'yes' to all questions.
 -v
 verbose mode.
 
+*undelete-subvol* ::
+Undelete all deleted subvolumes that still left intact on the device.
++
+This command will create the lost+found directory, and recover all the
+subvolumes to this directory with the name "sub".
+
 *zero-log* ::
 clear the filesystem log tree
 +
diff --git a/btrfs-completion b/btrfs-completion
index ae683f4ecf61..859595155a4b 100644
--- a/btrfs-completion
+++ b/btrfs-completion
@@ -35,7 +35,7 @@ _btrfs()
commands_balance='start pause cancel resume status'
commands_device='scan add delete remove ready stats usage'
commands_scrub='start cancel resume status'
-   commands_rescue='chunk-recover super-recover zero-log'
+   commands_rescue='chunk-recover super-recover undelete-subvol zero-log'
commands_inspect_internal='inode-resolve logical-resolve 
subvolid-resolve rootid min-dev-size dump-tree dump-super tree-stats'
commands_property='get set list'
commands_quota='enable disable rescan'
-- 
2.16.1



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 2/2] btrfs: add read_mirror_policy parameter devid

[Edmund Nadolski]

On 02/01/2018 01:12 AM, Anand Jain wrote:
> 
> 
> On 02/01/2018 01:26 PM, Edmund Nadolski wrote:
>> On 1/31/18 7:36 AM, Anand Jain wrote:
>>>
>>>
>>> On 01/31/2018 09:42 PM, Nikolay Borisov wrote:
>>>
>>>
>> So usually this should be functionality handled by the raid/san
>> controller I guess, > but given that btrfs is playing the role of a
>> controller here at what point are we drawing the line of not
>> implementing block-level functionality into the filesystem ?
>
>Don't worry this is not invading into the block layer. How
>can you even build this functionality in the block layer ?
>Block layer even won't know that disks are mirrored. RAID
>does or BTRFS in our case.
>

 By block layer I guess I meant the storage driver of a particular raid
 card. Because what is currently happening is re-implementing
 functionality that will generally sit in the driver. So my question was
 more generic and high-level - at what point do we draw the line of
 implementing feature that are generally implemented in hardware devices
 (be it their drivers or firmware).
>>>
>>>   Not all HW configs use RAID capable HBAs. A server connected to a SATA
>>>   JBOD using a SATA HBA without MD will relay on BTRFS to provide all
>>> the
>>>   features and capabilities that otherwise would have provided by such a
>>>   presumable HW config.
>>
>> That does sort of sound like means implementing some portion of the
>> HBA features/capabilities in the filesystem.
>>
>> To me it seems this this could be workable at the fs level, provided it
>> deals just with policies and remains hardware-neutral.
> 
>  Thanks. Ok.
> 
>> However most
>> of the use cases appear to involve some hardware-dependent knowledge
>> or assumptions. 
> 
>> What happens when someone sets this on a virtual disk,
>> or say a (persistent) memory-backed block device? 
> 
>  Do you have any policy in particular ?

No, this is your proposal ;^)

You've said cases #3 thru #6 are illustrative only. However they make
assumptions about the underlying storage, and/or introduce potential for
unexpected behaviors. Plus they could end up replicating functionality
from other layers as Nikolay pointed out. Seems unlikely these would be
practical to implement.

Case #2 seems concerning if it exposes internal,
implementation-dependent filesystem data into a de facto user-level
interface. (Do we ensure the devid is unique, and cannot get changed or
re-assigned internally to a different device, etc?)

Thanks,
Ed


>> Case #6 seems to
>> open up some potential for unexpected interactions (which may be hard
>> to reproduce, esp. in error/recovery scenarios).
> 
>  Yep. Even the #1 pid based (current default) which motivated
>  me to provide the devid based policy.
> 
>> Case #2 takes a devid, but I notice btrfs_device::devid says, "the
>> internal btrfs device id".  How does a user obtain that internal value
>> so it can be set as a mount option?
> 
>   btrfs fi show -m
> 
> Thanks, Anand
> 
>> Thanks,
>> Ed
>>
>>
>>> ::
> diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c
> index 39ba59832f38..478623e6e074 100644
> --- a/fs/btrfs/volumes.c
> +++ b/fs/btrfs/volumes.c
> @@ -5270,6 +5270,16 @@ static int find_live_mirror(struct
> btrfs_fs_info *fs_info,
>  num = map->num_stripes;
>switch(fs_info->read_mirror_policy) {
> +case BTRFS_READ_MIRROR_BY_DEV:
> +optimal = first;
> +if (test_bit(BTRFS_DEV_STATE_READ_MIRROR,
> + >stripes[optimal].dev->dev_state))
> +break;
> +if (test_bit(BTRFS_DEV_STATE_READ_MIRROR,
> + >stripes[++optimal].dev->dev_state))
> +break;
> +optimal = first;

 you set optimal 2 times, the second one seems redundant.
>>>
>>> No actually. When both the disks containing the stripe does not
>>> have the BTRFS_DEV_STATE_READ_MIRROR, then I would just want to
>>> use first found stripe.
>>
>> Yes, and the fact that you've already set optimal = first right after
>> BTRFS_READ_MIRROR_BY_DEV ensures that, no ? Why do you need to again
>> set
>> optimal right before the final break? What am I missing here?
>
> Ah. I think you are missing ++optimal in the 2nd if.

 You are right, but I'd prefer you index the stripes array with
 'optimal'
 and 'optimal + 1' and leave just a single assignment
>>>
>>>   Ok. Will improve that.
>>>
>>> Thanks, Anand
>>>
>>>
>
> Thanks, Anand
>
 -- 
 To unsubscribe from this list: send the line "unsubscribe
 linux-btrfs" in
 the body of a message to majord...@vger.kernel.org
 More majordomo info at  http://vger.kernel.org/majordomo-info.html

>>> -- 
>>> To unsubscribe from this list: send 

Re: Writeback errors in kernel log with Linux 4.15 (m=s=raid1, d=raid5, 5 disks)

[Nikolay Borisov]

On  2.02.2018 03:28, Janos Toth F. wrote:
> I started seeing these on my d=raid5 filesystem after upgrading to Linux 4.15.
> 
> Some files created since the upgrade seem to be corrupted.
> 
> The disks seem to be fine (according to btrfs device stats and
> smartmontools device logs).
> 
> The rest of the Btrfs filesystems (with m=s=d=single profiles) do not
> seem to have any issues but it could be linked to the specific load
> the d=raid5 filesystem handles (unlikely since it's nothing too
> fancy).
> 
> The mount options are (for all Btrfs filesystems on this machine):
> noatime,max_inline=0,commit=300,space_cache=v2,flushoncommit
> 
> dmesg is full of these:
> 
> [12848.106644] Code: 8d 74 24 08 e8 ec fd ff ff 48 89 ee 48 89 df e8
> d1 fe ff ff 48 8b 44 24 48 65 48 33 04 25 28 00 00 00 75 0b 48 83 c4
> 50 5b 5d c3 <0f> ff eb c0 e8 71 70 ef ff 90 53 48 c7 c3 60 62 83 9d 48
> 85 ff
> [12848.106664] ---[ end trace bb7f6cdb25fa37db ]---
> [12850.114566] WARNING: CPU: 0 PID: 7277 at fs/fs-writeback.c:2339
> __writeback_inodes_sb_nr+0xc6/0xd0
> [12850.114573] CPU: 0 PID: 7277 Comm: btrfs Tainted: GW
> 4.15.0-gentoo #2
> [12850.114575] Hardware name: Gigabyte Technology Co., Ltd. X150M-PRO
> ECC/X150M-PRO ECC-CF, BIOS F22b 07/04/2017
> [12850.114581] RIP: 0010:__writeback_inodes_sb_nr+0xc6/0xd0
> [12850.114584] RSP: 0018:a2e7c619b9c0 EFLAGS: 00010246
> [12850.114588] RAX:  RBX: 8963b99ca800 RCX: 
> 
> [12850.114591] RDX: 0002 RSI: 2fcf RDI: 
> 8963b982a800
> [12850.114593] RBP: a2e7c619b9c4 R08: fffe R09: 
> 0003
> [12850.114596] R10: f2ec00094c40 R11: 008f R12: 
> 89612accc400
> [12850.114598] R13: 8963b98235b8 R14: 8961166a5000 R15: 
> 8963b98235e0
> [12850.114602] FS:  7f9c6280c8c0() GS:8963bec0()
> knlGS:
> [12850.114605] CS:  0010 DS:  ES:  CR0: 80050033
> [12850.114607] CR2: 7ffd26332e78 CR3: 00024443a005 CR4: 
> 002606f0
> [12850.114609] Call Trace:
> [12850.114618]  btrfs_commit_transaction+0x776/0x860
> [12850.114626]  prepare_to_merge+0x226/0x240
> [12850.114632]  relocate_block_group+0x234/0x620
> [12850.114638]  btrfs_relocate_block_group+0x17e/0x230
> [12850.114643]  btrfs_relocate_chunk+0x2e/0xc0
> [12850.114647]  btrfs_balance+0xb42/0x1270
> [12850.114654]  ? terminate_walk+0x8c/0x100
> [12850.114660]  btrfs_ioctl_balance+0x330/0x390
> [12850.114665]  btrfs_ioctl+0x4a6/0x23da
> [12850.114674]  ? __alloc_pages_nodemask+0xa9/0x130
> [12850.114678]  ? do_vfs_ioctl+0x9f/0x620
> [12850.114684]  ? btrfs_ioctl_get_supported_features+0x20/0x20
> [12850.114687]  do_vfs_ioctl+0x9f/0x620
> [12850.114692]  SyS_ioctl+0x36/0x70
> [12850.114698]  ? get_vtime_delta+0x9/0x40
> [12850.114702]  do_syscall_64+0x72/0x360
> [12850.114707]  ? get_vtime_delta+0x9/0x40
> [12850.114711]  ? get_vtime_delta+0x9/0x40
> [12850.114716]  ? __vtime_account_system+0x10/0x50
> [12850.114722]  entry_SYSCALL64_slow_path+0x25/0x25
> [12850.114726] RIP: 0033:0x7f9c616063a7
> [12850.114728] RSP: 002b:7ffe0d33a3d8 EFLAGS: 0206 ORIG_RAX:
> 0010
> [12850.114733] RAX: ffda RBX: 0001 RCX: 
> 7f9c616063a7
> [12850.114735] RDX: 7ffe0d33a470 RSI: c4009420 RDI: 
> 0003
> [12850.114738] RBP: 0003 R08: 0003 R09: 
> 0001
> [12850.114740] R10: 0559 R11: 0206 R12: 
> 7ffe0d33a470
> [12850.114743] R13: 7ffe0d33a470 R14: 0001 R15: 
> 7ffe0d33caf6
> --


Yeah, that's a known issue and there is still no fix. Do you have
flushoncommit option enabled ? If so, remount without it and this
warning wil disappear.

> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC PATCH v2 0/4] Add support for export testsuits

[Nikolay Borisov]

On  2.02.2018 10:34, Gu Jinxiang wrote:
> Achieved:
> 1. export testsuit by:
>  $ make EXPORT=/where/you/want/to/generate/tests.tar.gz package
> relative path and absolute path both be ok.
> Besides tests itself, fssum and btrfs-corrupt-block will also be
> included in tests.tar.gz, since misc and fsck tests depend on
> btrfs-corrupt-block, and misc tests depend on fssum. 
> 
> 2. after decompress tests.tar.gz, run test by:
>  $ TEST=`MASK` ./mkfs-tests.sh
> and, without MASK also be ok.
> replenish:
> The directory Structure after decompress tests.tar.gz is:
>  $ tar -xzvf ./tests.tar.gz
>  $ ls
>tests fssum btrfs-corrupt-block

So *WHY* do we need this what problem are you solving?

> 
> Gu Jinxiang (4):
>   btrfs-progs: Add package command for packaging tests
>   btrfs-progs: Add EXEC represent path of executable file
>   btrfs-progs: Modify rootdir files when use mkfs.btrfs
>   btrfs-progs: Add readme for export testsuits
> 
>  Makefile   |  4 ++
>  tests/README.md| 20 
>  tests/cli-tests.sh |  5 ++
>  tests/cli-tests/001-btrfs/test.sh  | 20 
>  .../cli-tests/002-balance-full-no-filters/test.sh  | 10 ++--
>  tests/cli-tests/003-fi-resize-args/test.sh | 30 +--
>  .../cli-tests/004-send-parent-multi-subvol/test.sh | 12 ++---
>  tests/cli-tests/005-qgroup-show/test.sh| 14 ++---
>  tests/cli-tests/006-qgroup-show-sync/test.sh   | 14 ++---
>  tests/cli-tests/007-check-force/test.sh| 14 ++---
>  .../008-subvolume-get-set-default/test.sh  | 20 
>  tests/common   | 16 +++---
>  tests/common.convert   |  8 +--
>  tests/convert-tests.sh |  5 ++
>  .../004-ext2-backup-superblock-ranges/test.sh  | 10 ++--
>  .../convert-tests/005-delete-all-rollback/test.sh  |  2 +-
>  .../007-unsupported-block-sizes/test.sh|  2 +-
>  .../011-reiserfs-delete-all-rollback/test.sh   |  2 +-
>  .../015-no-rollback-after-balance/test.sh  |  6 +--
>  tests/export-tests.sh  | 28 ++
>  tests/fsck-tests.sh|  5 ++
>  tests/fsck-tests/012-leaf-corruption/test.sh   |  2 +-
>  tests/fsck-tests/013-extent-tree-rebuild/test.sh   | 12 ++---
>  tests/fsck-tests/018-leaf-crossing-stripes/test.sh |  2 +-
>  .../fsck-tests/019-non-skinny-false-alert/test.sh  |  2 +-
>  tests/fsck-tests/020-extent-ref-cases/test.sh  |  2 +-
>  .../021-partially-dropped-snapshot-case/test.sh|  2 +-
>  tests/fsck-tests/022-qgroup-rescan-halfway/test.sh |  2 +-
>  tests/fsck-tests/023-qgroup-stack-overflow/test.sh |  2 +-
>  tests/fsck-tests/024-clear-space-cache/test.sh | 10 ++--
>  tests/fsck-tests/025-file-extents/test.sh  | 14 ++---
>  tests/fsck-tests/026-bad-dir-item-name/test.sh |  2 +-
>  tests/fsck-tests/027-tree-reloc-tree/test.sh   |  2 +-
>  .../028-unaligned-super-dev-sizes/test.sh  |  6 +--
>  tests/fuzz-tests.sh|  5 ++
>  .../fuzz-tests/001-simple-check-unmounted/test.sh  |  2 +-
>  tests/fuzz-tests/002-simple-image/test.sh  |  2 +-
>  tests/fuzz-tests/003-multi-check-unmounted/test.sh | 12 ++---
>  tests/fuzz-tests/004-simple-dump-tree/test.sh  |  2 +-
>  tests/fuzz-tests/005-simple-dump-super/test.sh |  4 +-
>  tests/fuzz-tests/006-simple-tree-stats/test.sh |  2 +-
>  tests/fuzz-tests/007-simple-super-recover/test.sh  |  2 +-
>  tests/fuzz-tests/008-simple-chunk-recover/test.sh  |  2 +-
>  tests/fuzz-tests/009-simple-zero-log/test.sh   |  2 +-
>  tests/misc-tests.sh|  5 ++
>  tests/misc-tests/001-btrfstune-features/test.sh| 10 ++--
>  tests/misc-tests/002-uuid-rewrite/test.sh  | 24 -
>  tests/misc-tests/003-zero-log/test.sh  | 18 +++
>  tests/misc-tests/004-shrink-fs/test.sh | 14 ++---
>  .../005-convert-progress-thread-crash/test.sh  |  2 +-
>  .../misc-tests/006-image-on-missing-device/test.sh | 10 ++--
>  tests/misc-tests/007-subvolume-sync/test.sh| 18 +++
>  tests/misc-tests/008-leaf-crossing-stripes/test.sh |  4 +-
>  .../009-subvolume-sync-must-wait/test.sh   | 22 
>  .../010-convert-delete-ext2-subvol/test.sh | 10 ++--
>  tests/misc-tests/011-delete-missing-device/test.sh | 18 +++
>  tests/misc-tests/012-find-root-no-result/test.sh   |  4 +-
>  tests/misc-tests/013-subvolume-sync-crash/test.sh  | 20 
>  tests/misc-tests/014-filesystem-label/test.sh  | 22 
>  tests/misc-tests/015-dump-super-garbage/test.sh| 18 +++
>  tests/misc-tests/016-send-clone-src/test.sh| 12 ++---
>  .../017-recv-stream-malformatted/test.sh   |  8 +--
>  tests/misc-tests/018-recv-end-of-stream/test.sh| 60 
> 

Re: [PATCH 2/7] btrfs-progs: Merge btrfs_alloc_data_chunk into btrfs_alloc_chunk

[Su Yue]

On 02/02/2018 04:19 PM, Qu Wenruo wrote:

We used to have two chunk allocators, btrfs_alloc_chunk() and
btrfs_alloc_data_chunk(), the former is the more generic one, while the
later is only used in mkfs and convert, to allocate SINGLE data chunk.

Although btrfs_alloc_data_chunk() has some special hacks to cooperate
with convert, it's quite simple to integrity it into the generic chunk
allocator.

So merge them into one btrfs_alloc_chunk(), with extra @convert
parameter and necessary comment, to make code less duplicated and less
thing to maintain.

Signed-off-by: Qu Wenruo 
---
  convert/main.c |   6 +-
  extent-tree.c  |   2 +-
  mkfs/main.c|   8 +--
  volumes.c  | 219 ++---
  volumes.h  |   5 +-
  5 files changed, 94 insertions(+), 146 deletions(-)

diff --git a/convert/main.c b/convert/main.c
index b3ea31d7af43..b2444bb2ff21 100644
--- a/convert/main.c
+++ b/convert/main.c
@@ -910,9 +910,9 @@ static int make_convert_data_block_groups(struct 
btrfs_trans_handle *trans,
  
  			len = min(max_chunk_size,

  cache->start + cache->size - cur);
-   ret = btrfs_alloc_data_chunk(trans, fs_info,
-   _backup, len,
-   BTRFS_BLOCK_GROUP_DATA, 1);
+   ret = btrfs_alloc_chunk(trans, fs_info,
+   _backup, ,
+   BTRFS_BLOCK_GROUP_DATA, true);
if (ret < 0)
break;
ret = btrfs_make_block_group(trans, fs_info, 0,
diff --git a/extent-tree.c b/extent-tree.c
index e2ae74a7fe66..b085ab0352b3 100644
--- a/extent-tree.c
+++ b/extent-tree.c
@@ -1906,7 +1906,7 @@ static int do_chunk_alloc(struct btrfs_trans_handle 
*trans,
return 0;
  
  	ret = btrfs_alloc_chunk(trans, fs_info, , _bytes,

-   space_info->flags);
+   space_info->flags, false);
if (ret == -ENOSPC) {
space_info->full = 1;
return 0;
diff --git a/mkfs/main.c b/mkfs/main.c
index ea65c6d897b2..358395ca0250 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -82,7 +82,7 @@ static int create_metadata_block_groups(struct btrfs_root 
*root, int mixed,
ret = btrfs_alloc_chunk(trans, fs_info,
_start, _size,
BTRFS_BLOCK_GROUP_METADATA |
-   BTRFS_BLOCK_GROUP_DATA);
+   BTRFS_BLOCK_GROUP_DATA, false);
if (ret == -ENOSPC) {
error("no space to allocate data/metadata chunk");
goto err;
@@ -99,7 +99,7 @@ static int create_metadata_block_groups(struct btrfs_root 
*root, int mixed,
} else {
ret = btrfs_alloc_chunk(trans, fs_info,
_start, _size,
-   BTRFS_BLOCK_GROUP_METADATA);
+   BTRFS_BLOCK_GROUP_METADATA, false);
if (ret == -ENOSPC) {
error("no space to allocate metadata chunk");
goto err;
@@ -133,7 +133,7 @@ static int create_data_block_groups(struct 
btrfs_trans_handle *trans,
if (!mixed) {
ret = btrfs_alloc_chunk(trans, fs_info,
_start, _size,
-   BTRFS_BLOCK_GROUP_DATA);
+   BTRFS_BLOCK_GROUP_DATA, false);
if (ret == -ENOSPC) {
error("no space to allocate data chunk");
goto err;
@@ -241,7 +241,7 @@ static int create_one_raid_group(struct btrfs_trans_handle 
*trans,
int ret;
  
  	ret = btrfs_alloc_chunk(trans, fs_info,

-   _start, _size, type);
+   _start, _size, type, false);
if (ret == -ENOSPC) {
error("not enough free space to allocate chunk");
exit(1);
diff --git a/volumes.c b/volumes.c
index 677d085de96c..9ee4650351c3 100644
--- a/volumes.c
+++ b/volumes.c
@@ -836,9 +836,23 @@ error:
- 2 * sizeof(struct btrfs_chunk))   \
/ sizeof(struct btrfs_stripe) + 1)
  
+/*

+ * Alloc a chunk, will insert dev extents, chunk item.
+ * NOTE: This function will not insert block group item nor mark newly
+ * allocated chunk available for later allocation.
+ * Block group item and free space update is handled by 
btrfs_make_block_group()
+ *
+ * @start: return value of allocated chunk start bytenr.
+ * @num_bytes: return value of allocated chunk size
+ * @type:  chunk type (including both profile and type)
+ * 

Re: [PATCH 2/7] btrfs-progs: Merge btrfs_alloc_data_chunk into btrfs_alloc_chunk

[Qu Wenruo]

On 2018年02月02日 17:20, Su Yue wrote:
> 
> 
> On 02/02/2018 04:19 PM, Qu Wenruo wrote:
>> We used to have two chunk allocators, btrfs_alloc_chunk() and
>> btrfs_alloc_data_chunk(), the former is the more generic one, while the
>> later is only used in mkfs and convert, to allocate SINGLE data chunk.
>>
>> Although btrfs_alloc_data_chunk() has some special hacks to cooperate
>> with convert, it's quite simple to integrity it into the generic chunk
>> allocator.
>>
>> So merge them into one btrfs_alloc_chunk(), with extra @convert
>> parameter and necessary comment, to make code less duplicated and less
>> thing to maintain.
>>
>> Signed-off-by: Qu Wenruo 
>> ---
>>   convert/main.c |   6 +-
>>   extent-tree.c  |   2 +-
>>   mkfs/main.c    |   8 +--
>>   volumes.c  | 219
>> ++---
[snip]
>> diff --git a/volumes.c b/volumes.c
>> index 677d085de96c..9ee4650351c3 100644
>> --- a/volumes.c
>> +++ b/volumes.c
>> @@ -836,9 +836,23 @@ error:
>>   - 2 * sizeof(struct btrfs_chunk))    \
>>   / sizeof(struct btrfs_stripe) + 1)
>>   +/*
>> + * Alloc a chunk, will insert dev extents, chunk item.
>> + * NOTE: This function will not insert block group item nor mark newly
>> + * allocated chunk available for later allocation.
>> + * Block group item and free space update is handled by
>> btrfs_make_block_group()
>> + *
>> + * @start:    return value of allocated chunk start bytenr.
>> + * @num_bytes:    return value of allocated chunk size
>> + * @type:    chunk type (including both profile and type)
>> + * @convert:    if the chunk is allocated for convert case.
>> + * If @convert is true, chunk allocator will skip device extent
>> + * search, but use *start and *num_bytes as chunk
>> start/num_bytes
>> + * and devive offset, to build a 1:1 chunk mapping for convert.
>> + */
>>   int btrfs_alloc_chunk(struct btrfs_trans_handle *trans,
>>     struct btrfs_fs_info *info, u64 *start,
>> -  u64 *num_bytes, u64 type)
>> +  u64 *num_bytes, u64 type, bool convert)
>>   {
>>   u64 dev_offset;
>>   struct btrfs_root *extent_root = info->extent_root;
>> @@ -868,10 +882,38 @@ int btrfs_alloc_chunk(struct btrfs_trans_handle
>> *trans,
>>   struct btrfs_key key;
>>   u64 offset;
>>   -    if (list_empty(dev_list)) {
>> +    if (list_empty(dev_list))
>>   return -ENOSPC;
>> +
>> +    if (convert) {
>> +    /* For convert, profile must be SINGLE */
>> +    if (type & BTRFS_BLOCK_GROUP_PROFILE_MASK) {
> Maybe BTRFS_RAID_SINGLE?

Here we're using type and raid index is not introduced yet, so what
we're doing is correct.

Thanks,
Qu

> 
> Thanks,
> Su
> 
>> +    error("convert only suports SINGLE profile");
>> +    return -EINVAL;
>> +    }
>> +    if (!IS_ALIGNED(*start, info->sectorsize)) {
>> +    error("chunk start not aligned, start=%llu sectorsize=%u",
>> +    *start, info->sectorsize);
>> +    return -EINVAL;
>> +    }
>> +    if (!IS_ALIGNED(*num_bytes, info->sectorsize)) {
>> +    error("chunk size not aligned, size=%llu sectorsize=%u",
>> +    *num_bytes, info->sectorsize);
>> +    return -EINVAL;
>> +    }
>> +    calc_size = *num_bytes;
>> +    offset = *start;
>> +    /*
>> + * For convert, we use 1:1 chunk mapping specified by @start and
>> + * @num_bytes, so there is no need to go through dev_extent
>> + * searching.
>> + */
>> +    goto alloc_chunk;
>>   }
>>   +    /*
>> + * Chunk size calculation part.
>> + */
>>   if (type & BTRFS_BLOCK_GROUP_PROFILE_MASK) {
>>   if (type & BTRFS_BLOCK_GROUP_SYSTEM) {
>>   calc_size = SZ_8M;
>> @@ -942,6 +984,9 @@ int btrfs_alloc_chunk(struct btrfs_trans_handle
>> *trans,
>>   percent_max =
>> div_factor(btrfs_super_total_bytes(info->super_copy), 1);
>>   max_chunk_size = min(percent_max, max_chunk_size);
>>   +    /*
>> + * Reserve space from each device.
>> + */
>>   again:
>>   if (chunk_bytes_by_type(type, calc_size, num_stripes,
>> sub_stripes) >
>>   max_chunk_size) {
>> @@ -972,7 +1017,8 @@ again:
>>   return ret;
>>   cur = cur->next;
>>   if (avail >= min_free) {
>> -    list_move_tail(>dev_list, _devs);
>> +    list_move_tail(>dev_list,
>> +   _devs);
>>   index++;
>>   if (type & BTRFS_BLOCK_GROUP_DUP)
>>   index++;
>> @@ -999,9 +1045,16 @@ again:
>>   }
>>   return -ENOSPC;
>>   }
>> -    ret = find_next_chunk(info, );
>> -    if (ret)
>> -    return ret;
>> +
>> +    /*
>> + * Fill chunk mapping and chunk stripes
>> + */
>> +alloc_chunk:
>> +    if (!convert) {
>> +    ret = find_next_chunk(info, );
>> +    if (ret)
>> +    return 

Re: [PATCH 2/7] btrfs-progs: Merge btrfs_alloc_data_chunk into btrfs_alloc_chunk

[Su Yue]

On 02/02/2018 05:41 PM, Qu Wenruo wrote:



On 2018年02月02日 17:20, Su Yue wrote:



On 02/02/2018 04:19 PM, Qu Wenruo wrote:

We used to have two chunk allocators, btrfs_alloc_chunk() and
btrfs_alloc_data_chunk(), the former is the more generic one, while the
later is only used in mkfs and convert, to allocate SINGLE data chunk.

Although btrfs_alloc_data_chunk() has some special hacks to cooperate
with convert, it's quite simple to integrity it into the generic chunk
allocator.

So merge them into one btrfs_alloc_chunk(), with extra @convert
parameter and necessary comment, to make code less duplicated and less
thing to maintain.

Signed-off-by: Qu Wenruo 
---
   convert/main.c |   6 +-
   extent-tree.c  |   2 +-
   mkfs/main.c    |   8 +--
   volumes.c  | 219
++---

[snip]

diff --git a/volumes.c b/volumes.c
index 677d085de96c..9ee4650351c3 100644
--- a/volumes.c
+++ b/volumes.c
@@ -836,9 +836,23 @@ error:
   - 2 * sizeof(struct btrfs_chunk))    \
   / sizeof(struct btrfs_stripe) + 1)
   +/*
+ * Alloc a chunk, will insert dev extents, chunk item.
+ * NOTE: This function will not insert block group item nor mark newly
+ * allocated chunk available for later allocation.
+ * Block group item and free space update is handled by
btrfs_make_block_group()
+ *
+ * @start:    return value of allocated chunk start bytenr.
+ * @num_bytes:    return value of allocated chunk size
+ * @type:    chunk type (including both profile and type)
+ * @convert:    if the chunk is allocated for convert case.
+ * If @convert is true, chunk allocator will skip device extent
+ * search, but use *start and *num_bytes as chunk
start/num_bytes
+ * and devive offset, to build a 1:1 chunk mapping for convert.
+ */
   int btrfs_alloc_chunk(struct btrfs_trans_handle *trans,
     struct btrfs_fs_info *info, u64 *start,
-  u64 *num_bytes, u64 type)
+  u64 *num_bytes, u64 type, bool convert)
   {
   u64 dev_offset;
   struct btrfs_root *extent_root = info->extent_root;
@@ -868,10 +882,38 @@ int btrfs_alloc_chunk(struct btrfs_trans_handle
*trans,
   struct btrfs_key key;
   u64 offset;
   -    if (list_empty(dev_list)) {
+    if (list_empty(dev_list))
   return -ENOSPC;
+
+    if (convert) {
+    /* For convert, profile must be SINGLE */
+    if (type & BTRFS_BLOCK_GROUP_PROFILE_MASK) {

Maybe BTRFS_RAID_SINGLE?


Here we're using type and raid index is not introduced yet, so what
we're doing is correct.


OK.

Reviewed-by: Su Yue 


Thanks,
Qu



Thanks,
Su


+    error("convert only suports SINGLE profile");
+    return -EINVAL;
+    }
+    if (!IS_ALIGNED(*start, info->sectorsize)) {
+    error("chunk start not aligned, start=%llu sectorsize=%u",
+    *start, info->sectorsize);
+    return -EINVAL;
+    }
+    if (!IS_ALIGNED(*num_bytes, info->sectorsize)) {
+    error("chunk size not aligned, size=%llu sectorsize=%u",
+    *num_bytes, info->sectorsize);
+    return -EINVAL;
+    }
+    calc_size = *num_bytes;
+    offset = *start;
+    /*
+ * For convert, we use 1:1 chunk mapping specified by @start and
+ * @num_bytes, so there is no need to go through dev_extent
+ * searching.
+ */
+    goto alloc_chunk;
   }
   +    /*
+ * Chunk size calculation part.
+ */
   if (type & BTRFS_BLOCK_GROUP_PROFILE_MASK) {
   if (type & BTRFS_BLOCK_GROUP_SYSTEM) {
   calc_size = SZ_8M;
@@ -942,6 +984,9 @@ int btrfs_alloc_chunk(struct btrfs_trans_handle
*trans,
   percent_max =
div_factor(btrfs_super_total_bytes(info->super_copy), 1);
   max_chunk_size = min(percent_max, max_chunk_size);
   +    /*
+ * Reserve space from each device.
+ */
   again:
   if (chunk_bytes_by_type(type, calc_size, num_stripes,
sub_stripes) >
   max_chunk_size) {
@@ -972,7 +1017,8 @@ again:
   return ret;
   cur = cur->next;
   if (avail >= min_free) {
-    list_move_tail(>dev_list, _devs);
+    list_move_tail(>dev_list,
+   _devs);
   index++;
   if (type & BTRFS_BLOCK_GROUP_DUP)
   index++;
@@ -999,9 +1045,16 @@ again:
   }
   return -ENOSPC;
   }
-    ret = find_next_chunk(info, );
-    if (ret)
-    return ret;
+
+    /*
+ * Fill chunk mapping and chunk stripes
+ */
+alloc_chunk:
+    if (!convert) {
+    ret = find_next_chunk(info, );
+    if (ret)
+    return ret;
+    }
   key.objectid = BTRFS_FIRST_CHUNK_TREE_OBJECTID;
   key.type = BTRFS_CHUNK_ITEM_KEY;
   key.offset = offset;
@@ -1022,17 +1075,31 @@ again:
   index = 0;
   while(index < num_stripes) {
   struct btrfs_stripe 

Re: [RFC PATCH v2 0/4] Add support for export testsuits

[Qu Wenruo]

On 2018年02月02日 18:41, Nikolay Borisov wrote:
> 
> 
> On  2.02.2018 10:34, Gu Jinxiang wrote:
>> Achieved:
>> 1. export testsuit by:
>>  $ make EXPORT=/where/you/want/to/generate/tests.tar.gz package
>> relative path and absolute path both be ok.
>> Besides tests itself, fssum and btrfs-corrupt-block will also be
>> included in tests.tar.gz, since misc and fsck tests depend on
>> btrfs-corrupt-block, and misc tests depend on fssum. 
>>
>> 2. after decompress tests.tar.gz, run test by:
>>  $ TEST=`MASK` ./mkfs-tests.sh
>> and, without MASK also be ok.
>> replenish:
>> The directory Structure after decompress tests.tar.gz is:
>>  $ tar -xzvf ./tests.tar.gz
>>  $ ls
>>tests fssum btrfs-corrupt-block
> 
> So *WHY* do we need this what problem are you solving?

I used to think the same, until Gu pointed out it's in the project idea,
and here is the reason:

https://github.com/kdave/btrfs-progs/issues/77

Thanks,
Qu

> 
>>
>> Gu Jinxiang (4):
>>   btrfs-progs: Add package command for packaging tests
>>   btrfs-progs: Add EXEC represent path of executable file
>>   btrfs-progs: Modify rootdir files when use mkfs.btrfs
>>   btrfs-progs: Add readme for export testsuits
>>
>>  Makefile   |  4 ++
>>  tests/README.md| 20 
>>  tests/cli-tests.sh |  5 ++
>>  tests/cli-tests/001-btrfs/test.sh  | 20 
>>  .../cli-tests/002-balance-full-no-filters/test.sh  | 10 ++--
>>  tests/cli-tests/003-fi-resize-args/test.sh | 30 +--
>>  .../cli-tests/004-send-parent-multi-subvol/test.sh | 12 ++---
>>  tests/cli-tests/005-qgroup-show/test.sh| 14 ++---
>>  tests/cli-tests/006-qgroup-show-sync/test.sh   | 14 ++---
>>  tests/cli-tests/007-check-force/test.sh| 14 ++---
>>  .../008-subvolume-get-set-default/test.sh  | 20 
>>  tests/common   | 16 +++---
>>  tests/common.convert   |  8 +--
>>  tests/convert-tests.sh |  5 ++
>>  .../004-ext2-backup-superblock-ranges/test.sh  | 10 ++--
>>  .../convert-tests/005-delete-all-rollback/test.sh  |  2 +-
>>  .../007-unsupported-block-sizes/test.sh|  2 +-
>>  .../011-reiserfs-delete-all-rollback/test.sh   |  2 +-
>>  .../015-no-rollback-after-balance/test.sh  |  6 +--
>>  tests/export-tests.sh  | 28 ++
>>  tests/fsck-tests.sh|  5 ++
>>  tests/fsck-tests/012-leaf-corruption/test.sh   |  2 +-
>>  tests/fsck-tests/013-extent-tree-rebuild/test.sh   | 12 ++---
>>  tests/fsck-tests/018-leaf-crossing-stripes/test.sh |  2 +-
>>  .../fsck-tests/019-non-skinny-false-alert/test.sh  |  2 +-
>>  tests/fsck-tests/020-extent-ref-cases/test.sh  |  2 +-
>>  .../021-partially-dropped-snapshot-case/test.sh|  2 +-
>>  tests/fsck-tests/022-qgroup-rescan-halfway/test.sh |  2 +-
>>  tests/fsck-tests/023-qgroup-stack-overflow/test.sh |  2 +-
>>  tests/fsck-tests/024-clear-space-cache/test.sh | 10 ++--
>>  tests/fsck-tests/025-file-extents/test.sh  | 14 ++---
>>  tests/fsck-tests/026-bad-dir-item-name/test.sh |  2 +-
>>  tests/fsck-tests/027-tree-reloc-tree/test.sh   |  2 +-
>>  .../028-unaligned-super-dev-sizes/test.sh  |  6 +--
>>  tests/fuzz-tests.sh|  5 ++
>>  .../fuzz-tests/001-simple-check-unmounted/test.sh  |  2 +-
>>  tests/fuzz-tests/002-simple-image/test.sh  |  2 +-
>>  tests/fuzz-tests/003-multi-check-unmounted/test.sh | 12 ++---
>>  tests/fuzz-tests/004-simple-dump-tree/test.sh  |  2 +-
>>  tests/fuzz-tests/005-simple-dump-super/test.sh |  4 +-
>>  tests/fuzz-tests/006-simple-tree-stats/test.sh |  2 +-
>>  tests/fuzz-tests/007-simple-super-recover/test.sh  |  2 +-
>>  tests/fuzz-tests/008-simple-chunk-recover/test.sh  |  2 +-
>>  tests/fuzz-tests/009-simple-zero-log/test.sh   |  2 +-
>>  tests/misc-tests.sh|  5 ++
>>  tests/misc-tests/001-btrfstune-features/test.sh| 10 ++--
>>  tests/misc-tests/002-uuid-rewrite/test.sh  | 24 -
>>  tests/misc-tests/003-zero-log/test.sh  | 18 +++
>>  tests/misc-tests/004-shrink-fs/test.sh | 14 ++---
>>  .../005-convert-progress-thread-crash/test.sh  |  2 +-
>>  .../misc-tests/006-image-on-missing-device/test.sh | 10 ++--
>>  tests/misc-tests/007-subvolume-sync/test.sh| 18 +++
>>  tests/misc-tests/008-leaf-crossing-stripes/test.sh |  4 +-
>>  .../009-subvolume-sync-must-wait/test.sh   | 22 
>>  .../010-convert-delete-ext2-subvol/test.sh | 10 ++--
>>  tests/misc-tests/011-delete-missing-device/test.sh | 18 +++
>>  tests/misc-tests/012-find-root-no-result/test.sh   |  4 +-
>>  tests/misc-tests/013-subvolume-sync-crash/test.sh  | 20 
>>