Re: [OE-core] [PATCH] wic: add --reserved-size wks option

[Maciej Borzęcki]

On Tue, Oct 18, 2016 at 4:24 PM, Ed Bartosh  wrote:
> On Tue, Oct 18, 2016 at 03:59:23PM +0200, Maciej Borzęcki wrote:
>> >> >> >> > What's the advantage of creating unusable gap over creating 
>> >> >> >> > partition of
>> >> >> >> > the same size that can be used?
>> >> >> >>
>> >> >> >> Just convenience.
>> >> >> >>
>> >> >> > What's the convenience of having extra space on partition that can't 
>> >> >> > be
>> >> >> > used for data over having it formatted and used?
>> >> >> >
>> >> >> >> > Even if that space is not needed it doesn't harm to have it, does 
>> >> >> >> > it?
>> >> >> >>
>> >> >> >> I have not seen any negative side effects.
>> >> >> >>
>> >> >> > I do. If user needs that reserved space it's impossible to get it 
>> >> >> > without
>> >> >> > reformatting partition. The free space exists, but can't be used.
>> >> >>
>> >> >> That's not the point and is not aligned with use case I'm trying to 
>> >> >> solve.
>> >> >>
>> >> >> My case is rather simple, I'm creating an image for SD card that is
>> >> >> deployed in the field. In that particular case, there's a primary and
>> >> >> a secondary (aka. active and inactive) rootfs partitions that are
>> >> >> switched whenever a system update comes in. The update is a file
>> >> >> system image that is copied over to the inactive partition, followed
>> >> >> by a system reboot.
>> >> >>
>> >> >> What I need is the ability to set a certain size of a partition (say
>> >> >> 100MB), regardless of current rootfs size (which may be, say 70MB).
>> >> >> The remaining unused space sets an upper limit on how much the rootfs
>> >> >> may grow in the future (in this example case, it's 30MB). RIght now
>> >> >> the best I can do is to describe a partition like this: `part /
>> >> >> --source rootfs --size 100MB --overhead-factor 1`, hoping that if
>> >> >> rootfs grows beyond 100MB I will somehow still be able to catch that
>> >> >> and that the future images remain size compatible.
>> >> >>
>> >> >> The resulting filesystem inside the partition is larger than what
>> >> >> IMAGE_CMD (ex. IMAGE_CMD_ext4) would give me, because of explicit
>> >> >> --size in kickstart. I would prefer to have something comparable in
>> >> >> size just to avoid later surprises, what implies using defaults.
>> >> >> However, using defaults, means that I cannot control the layout
>> >> >> because it will likely change each time rootfs size gets changed.
>> >> >> There is no `--fixed-size` or other option to enforce specific size.
>> >> >>
>> >> >> Summing up, a simple use case that cannot be currently solved using 
>> >> >> wic.
>> >> >>
>> >> >> BTW. actually we're missing an ability to enforce --size (i.e.
>> >> >> --fixed-size?) and a method passing an explicit partition offset
>> >> >> inside the disk image (something useful for `--source rawcopy
>> >> >> --no-table` partitions, currently solved with `--align`).
>> >> >>
>> >> > I undertood the problem and I agree that wic doesn't provide a solution.
>> >> >
>> >> > However, instead of making unformatted space I'd propose to format it,
>> >> > i.e. to have --max-size option that would confict with --size and
>> >> > specify upper size limit for the partition. All partition will be
>> >> > formatted and available for data. This is identical to --fixed-size 
>> >> > option
>> >> > you've described. This approach would solve the problem you're
>> >> > addressing and it would also make additional space usable.
>> >> >
>> >> > I'd also suggest to rename --size to --min-size and make --size 
>> >> > deprecated.
>> >> >
>> >> > Does this make sense to you?
>> >>
>> >> No strong opinions here, just that deprecating --size might current
>> >> users uneasy.
>> >>
>> > By deprecting --size I didn't mean removing it completely. We can just
>> > print a warning suggesting usage of other options.
>> >
>> >> Perhaps --max-size could be a boolean switch? We could just name it
>> >> --fixed-size (bool, defaults to False), with semantics that if
>> >> --fixed-size is provided, the partition will have size --size,
>> >> occurrence of --overhead-factor or --extra-space will raise an error.
>> >>
>> > That would work too, but it looks a bit confusing to me to have 2 different
>> > types of size-related options.
>>
>> Ok, but now we would have --min-size (previously known as --size) and
>> --size (or --max-size?). That's still 2 size related options plus a
>> deprecation warning.
>>
>
> I agree, it doesn't look good. Moreover --max-size doesn't make it
> clear that partition will be this size. It rather suggests that partition
> can be this size or less.
>
> So, we have --size which is actually minimum partition size, we have
> couple of options to extend partition size (--overhead-factor and
> --extra-space), but we don't have ability to set upper limit for partition
> size.
>
> OK, let's agree on using --fixed-size(int)
> Using --fixed-size with any of --size or --overhead-factor or --extra-space
> options should 

Re: [OE-core] [PATCH] wic: add --reserved-size wks option

[Ed Bartosh]

On Tue, Oct 18, 2016 at 03:59:23PM +0200, Maciej Borzęcki wrote:
> >> >> >> > What's the advantage of creating unusable gap over creating 
> >> >> >> > partition of
> >> >> >> > the same size that can be used?
> >> >> >>
> >> >> >> Just convenience.
> >> >> >>
> >> >> > What's the convenience of having extra space on partition that can't 
> >> >> > be
> >> >> > used for data over having it formatted and used?
> >> >> >
> >> >> >> > Even if that space is not needed it doesn't harm to have it, does 
> >> >> >> > it?
> >> >> >>
> >> >> >> I have not seen any negative side effects.
> >> >> >>
> >> >> > I do. If user needs that reserved space it's impossible to get it 
> >> >> > without
> >> >> > reformatting partition. The free space exists, but can't be used.
> >> >>
> >> >> That's not the point and is not aligned with use case I'm trying to 
> >> >> solve.
> >> >>
> >> >> My case is rather simple, I'm creating an image for SD card that is
> >> >> deployed in the field. In that particular case, there's a primary and
> >> >> a secondary (aka. active and inactive) rootfs partitions that are
> >> >> switched whenever a system update comes in. The update is a file
> >> >> system image that is copied over to the inactive partition, followed
> >> >> by a system reboot.
> >> >>
> >> >> What I need is the ability to set a certain size of a partition (say
> >> >> 100MB), regardless of current rootfs size (which may be, say 70MB).
> >> >> The remaining unused space sets an upper limit on how much the rootfs
> >> >> may grow in the future (in this example case, it's 30MB). RIght now
> >> >> the best I can do is to describe a partition like this: `part /
> >> >> --source rootfs --size 100MB --overhead-factor 1`, hoping that if
> >> >> rootfs grows beyond 100MB I will somehow still be able to catch that
> >> >> and that the future images remain size compatible.
> >> >>
> >> >> The resulting filesystem inside the partition is larger than what
> >> >> IMAGE_CMD (ex. IMAGE_CMD_ext4) would give me, because of explicit
> >> >> --size in kickstart. I would prefer to have something comparable in
> >> >> size just to avoid later surprises, what implies using defaults.
> >> >> However, using defaults, means that I cannot control the layout
> >> >> because it will likely change each time rootfs size gets changed.
> >> >> There is no `--fixed-size` or other option to enforce specific size.
> >> >>
> >> >> Summing up, a simple use case that cannot be currently solved using wic.
> >> >>
> >> >> BTW. actually we're missing an ability to enforce --size (i.e.
> >> >> --fixed-size?) and a method passing an explicit partition offset
> >> >> inside the disk image (something useful for `--source rawcopy
> >> >> --no-table` partitions, currently solved with `--align`).
> >> >>
> >> > I undertood the problem and I agree that wic doesn't provide a solution.
> >> >
> >> > However, instead of making unformatted space I'd propose to format it,
> >> > i.e. to have --max-size option that would confict with --size and
> >> > specify upper size limit for the partition. All partition will be
> >> > formatted and available for data. This is identical to --fixed-size 
> >> > option
> >> > you've described. This approach would solve the problem you're
> >> > addressing and it would also make additional space usable.
> >> >
> >> > I'd also suggest to rename --size to --min-size and make --size 
> >> > deprecated.
> >> >
> >> > Does this make sense to you?
> >>
> >> No strong opinions here, just that deprecating --size might current
> >> users uneasy.
> >>
> > By deprecting --size I didn't mean removing it completely. We can just
> > print a warning suggesting usage of other options.
> >
> >> Perhaps --max-size could be a boolean switch? We could just name it
> >> --fixed-size (bool, defaults to False), with semantics that if
> >> --fixed-size is provided, the partition will have size --size,
> >> occurrence of --overhead-factor or --extra-space will raise an error.
> >>
> > That would work too, but it looks a bit confusing to me to have 2 different
> > types of size-related options.
> 
> Ok, but now we would have --min-size (previously known as --size) and
> --size (or --max-size?). That's still 2 size related options plus a
> deprecation warning.
> 

I agree, it doesn't look good. Moreover --max-size doesn't make it
clear that partition will be this size. It rather suggests that partition
can be this size or less.

So, we have --size which is actually minimum partition size, we have
couple of options to extend partition size (--overhead-factor and
--extra-space), but we don't have ability to set upper limit for partition
size.

OK, let's agree on using --fixed-size(int)
Using --fixed-size with any of --size or --overhead-factor or --extra-space
options should raise ks parser error. If rootfs size is bigger than
--fixed-size wic should raise an error too.

Any other opinions?

--
Regards,
Ed
-- 
___

Re: [OE-core] [PATCH] wic: add --reserved-size wks option

[Maciej Borzęcki]

On Tue, Oct 18, 2016 at 1:05 PM, Ed Bartosh  wrote:
> On Tue, Oct 18, 2016 at 01:07:48PM +0200, Maciej Borzęcki wrote:
>> On Tue, Oct 18, 2016 at 12:17 PM, Ed Bartosh  
>> wrote:
>> > On Tue, Oct 18, 2016 at 12:24:55PM +0200, Maciej Borzęcki wrote:
>> >> On Tue, Oct 18, 2016 at 11:27 AM, Ed Bartosh  
>> >> wrote:
>> >> > On Tue, Oct 18, 2016 at 11:24:14AM +0200, Maciej Borzęcki wrote:
>> >> >> On Tue, Oct 18, 2016 at 10:38 AM, Ed Bartosh 
>> >> >>  wrote:
>> >> >> > On Tue, Oct 18, 2016 at 10:37:22AM +0200, Maciej Borzęcki wrote:
>> >> >> >> On Tue, Oct 18, 2016 at 9:31 AM, Ed Bartosh 
>> >> >> >>  wrote:
>> >> >> >> > On Mon, Oct 17, 2016 at 04:46:00PM +0200, Maciej Borzęcki wrote:
>> >> >> >> >> On Mon, Oct 17, 2016 at 3:22 PM, Ed Bartosh 
>> >> >> >> >>  wrote:
>> >> >> >> >> > Hi Maciej,
>> >> >> >> >> >
>> >> >> >> >> > There is already --size and --extra-space options.
>> >> >> >> >> > Can we get the same or similar result by just using them? Do 
>> >> >> >> >> > we really
>> >> >> >> >> > need new option for similar purpose?
>> >> >> >> >>
>> >> >> >> >> --reserved-size serves a different purpose, it establishes an 
>> >> >> >> >> upper
>> >> >> >> >> bound on the size of a partition during layout. Unlike
>> >> >> >> >> --size/--extra-space does not depend on the size of the 
>> >> >> >> >> filesystem
>> >> >> >> >> image.
>> >> >> >> >>
>> >> >> >> >> For instance, assume I'm creating an image for SD card/eMMC with 
>> >> >> >> >> a
>> >> >> >> >> fixed partition layout (something simple: boot partition, 
>> >> >> >> >> primary &
>> >> >> >> >> secondary rootfs partitions, some data partition). Because future
>> >> >> >> >> system updates are delivered as filesystem image, I want to make 
>> >> >> >> >> sure
>> >> >> >> >> that there is exactly xxx MBs for my current and future rootfs 
>> >> >> >> >> images
>> >> >> >> >> (regardless of current image size). Neither --size nor 
>> >> >> >> >> --extra-space
>> >> >> >> >> can do that. I could use, say `--size 200 --overhead-factor 1`, 
>> >> >> >> >> but
>> >> >> >> >> this will needlessly create a 200MB rootfs image and if I happen 
>> >> >> >> >> to
>> >> >> >> >> cross the 200MB boundary I will not get an error.
>> >> >> >> >>
>> >> >> >> >> I had a private patch that added --fixed-size to enforce --size, 
>> >> >> >> >> but
>> >> >> >> >> that would still end up creating filesystem image to fill the 
>> >> >> >> >> whole
>> >> >> >> >> space.
>> >> >> >> > I didn't get the difference between enforcing partition size and 
>> >> >> >> > below
>> >> >> >> > implementation. Can you elaborate a bit?
>> >> >> >>
>> >> >> >> `--fixed-size` was something that I had added to my fork back in 
>> >> >> >> 2014,
>> >> >> >> even before `--overhead-factor` came in. The problem is that 
>> >> >> >> depending
>> >> >> >> on a project you might want to have more control over how partitions
>> >> >> >> are laid out, or even need to have a fixed layout. Adding
>> >> >> >> `--fixed-size` would had a similar effect to what `--overhead-factor
>> >> >> >> 1` does right now. Combined with `--size` would ensure that rootfs 
>> >> >> >> is
>> >> >> >> say, 200MB large. The downside was that wic would actually create a
>> >> >> >> 200MB rootfs, something that is not really necessary. In fact, I 
>> >> >> >> only
>> >> >> >> wanted to have 200MB gap so that I have some spare space for future
>> >> >> >> updates (where update is just a rootfs image you dd to the 
>> >> >> >> partition).
>> >> >> >>
>> >> >> > Thanks for the explanations. Now I got it - reserved size is not a 
>> >> >> > part
>> >> >> > of partition, it's a gap between partitions.
>> >> >>
>> >> >> I might have not been clear enough when explaining. It's not a gap,
>> >> >> it's just a container of size --reserved-size listed in MBR/GPT.
>> >> >> There's probably a filesystem inside but not necessarily.
>> >> >> Graphically it looks as like this:
>> >> >>
>> >> >>  --reserved-size
>> >> >>   |--|
>> >> >>   v  v
>> >> >> +-+--+-+
>> >> >> |. stuff .|xx|. stuff .|
>> >> >> +-+--+-+
>> >> >>   ^ ^^
>> >> >>   |-||
>> >> >>--size--extra-space
>> >> >>
>> >> >>
>> >> > Ah, I'm wrong again. It's a partition size limit, but it's not necessary
>> >> > formatted, right? It's only formatted if size == reserved_size.
>> >> >
>> >> >> >
>> >> >> > What's the advantage of creating unusable gap over creating 
>> >> >> > partition of
>> >> >> > the same size that can be used?
>> >> >>
>> 

Re: [OE-core] [PATCH] wic: add --reserved-size wks option

[Ed Bartosh]

On Tue, Oct 18, 2016 at 01:07:48PM +0200, Maciej Borzęcki wrote:
> On Tue, Oct 18, 2016 at 12:17 PM, Ed Bartosh  
> wrote:
> > On Tue, Oct 18, 2016 at 12:24:55PM +0200, Maciej Borzęcki wrote:
> >> On Tue, Oct 18, 2016 at 11:27 AM, Ed Bartosh  
> >> wrote:
> >> > On Tue, Oct 18, 2016 at 11:24:14AM +0200, Maciej Borzęcki wrote:
> >> >> On Tue, Oct 18, 2016 at 10:38 AM, Ed Bartosh 
> >> >>  wrote:
> >> >> > On Tue, Oct 18, 2016 at 10:37:22AM +0200, Maciej Borzęcki wrote:
> >> >> >> On Tue, Oct 18, 2016 at 9:31 AM, Ed Bartosh 
> >> >> >>  wrote:
> >> >> >> > On Mon, Oct 17, 2016 at 04:46:00PM +0200, Maciej Borzęcki wrote:
> >> >> >> >> On Mon, Oct 17, 2016 at 3:22 PM, Ed Bartosh 
> >> >> >> >>  wrote:
> >> >> >> >> > Hi Maciej,
> >> >> >> >> >
> >> >> >> >> > There is already --size and --extra-space options.
> >> >> >> >> > Can we get the same or similar result by just using them? Do we 
> >> >> >> >> > really
> >> >> >> >> > need new option for similar purpose?
> >> >> >> >>
> >> >> >> >> --reserved-size serves a different purpose, it establishes an 
> >> >> >> >> upper
> >> >> >> >> bound on the size of a partition during layout. Unlike
> >> >> >> >> --size/--extra-space does not depend on the size of the filesystem
> >> >> >> >> image.
> >> >> >> >>
> >> >> >> >> For instance, assume I'm creating an image for SD card/eMMC with a
> >> >> >> >> fixed partition layout (something simple: boot partition, primary 
> >> >> >> >> &
> >> >> >> >> secondary rootfs partitions, some data partition). Because future
> >> >> >> >> system updates are delivered as filesystem image, I want to make 
> >> >> >> >> sure
> >> >> >> >> that there is exactly xxx MBs for my current and future rootfs 
> >> >> >> >> images
> >> >> >> >> (regardless of current image size). Neither --size nor 
> >> >> >> >> --extra-space
> >> >> >> >> can do that. I could use, say `--size 200 --overhead-factor 1`, 
> >> >> >> >> but
> >> >> >> >> this will needlessly create a 200MB rootfs image and if I happen 
> >> >> >> >> to
> >> >> >> >> cross the 200MB boundary I will not get an error.
> >> >> >> >>
> >> >> >> >> I had a private patch that added --fixed-size to enforce --size, 
> >> >> >> >> but
> >> >> >> >> that would still end up creating filesystem image to fill the 
> >> >> >> >> whole
> >> >> >> >> space.
> >> >> >> > I didn't get the difference between enforcing partition size and 
> >> >> >> > below
> >> >> >> > implementation. Can you elaborate a bit?
> >> >> >>
> >> >> >> `--fixed-size` was something that I had added to my fork back in 
> >> >> >> 2014,
> >> >> >> even before `--overhead-factor` came in. The problem is that 
> >> >> >> depending
> >> >> >> on a project you might want to have more control over how partitions
> >> >> >> are laid out, or even need to have a fixed layout. Adding
> >> >> >> `--fixed-size` would had a similar effect to what `--overhead-factor
> >> >> >> 1` does right now. Combined with `--size` would ensure that rootfs is
> >> >> >> say, 200MB large. The downside was that wic would actually create a
> >> >> >> 200MB rootfs, something that is not really necessary. In fact, I only
> >> >> >> wanted to have 200MB gap so that I have some spare space for future
> >> >> >> updates (where update is just a rootfs image you dd to the 
> >> >> >> partition).
> >> >> >>
> >> >> > Thanks for the explanations. Now I got it - reserved size is not a 
> >> >> > part
> >> >> > of partition, it's a gap between partitions.
> >> >>
> >> >> I might have not been clear enough when explaining. It's not a gap,
> >> >> it's just a container of size --reserved-size listed in MBR/GPT.
> >> >> There's probably a filesystem inside but not necessarily.
> >> >> Graphically it looks as like this:
> >> >>
> >> >>  --reserved-size
> >> >>   |--|
> >> >>   v  v
> >> >> +-+--+-+
> >> >> |. stuff .|xx|. stuff .|
> >> >> +-+--+-+
> >> >>   ^ ^^
> >> >>   |-||
> >> >>--size--extra-space
> >> >>
> >> >>
> >> > Ah, I'm wrong again. It's a partition size limit, but it's not necessary
> >> > formatted, right? It's only formatted if size == reserved_size.
> >> >
> >> >> >
> >> >> > What's the advantage of creating unusable gap over creating partition 
> >> >> > of
> >> >> > the same size that can be used?
> >> >>
> >> >> Just convenience.
> >> >>
> >> > What's the convenience of having extra space on partition that can't be
> >> > used for data over having it formatted and used?
> >> >
> >> >> > Even if that space is not needed it doesn't harm 

Re: [OE-core] [PATCH] wic: add --reserved-size wks option

[Maciej Borzęcki]

On Tue, Oct 18, 2016 at 12:17 PM, Ed Bartosh  wrote:
> On Tue, Oct 18, 2016 at 12:24:55PM +0200, Maciej Borzęcki wrote:
>> On Tue, Oct 18, 2016 at 11:27 AM, Ed Bartosh  
>> wrote:
>> > On Tue, Oct 18, 2016 at 11:24:14AM +0200, Maciej Borzęcki wrote:
>> >> On Tue, Oct 18, 2016 at 10:38 AM, Ed Bartosh  
>> >> wrote:
>> >> > On Tue, Oct 18, 2016 at 10:37:22AM +0200, Maciej Borzęcki wrote:
>> >> >> On Tue, Oct 18, 2016 at 9:31 AM, Ed Bartosh 
>> >> >>  wrote:
>> >> >> > On Mon, Oct 17, 2016 at 04:46:00PM +0200, Maciej Borzęcki wrote:
>> >> >> >> On Mon, Oct 17, 2016 at 3:22 PM, Ed Bartosh 
>> >> >> >>  wrote:
>> >> >> >> > Hi Maciej,
>> >> >> >> >
>> >> >> >> > There is already --size and --extra-space options.
>> >> >> >> > Can we get the same or similar result by just using them? Do we 
>> >> >> >> > really
>> >> >> >> > need new option for similar purpose?
>> >> >> >>
>> >> >> >> --reserved-size serves a different purpose, it establishes an upper
>> >> >> >> bound on the size of a partition during layout. Unlike
>> >> >> >> --size/--extra-space does not depend on the size of the filesystem
>> >> >> >> image.
>> >> >> >>
>> >> >> >> For instance, assume I'm creating an image for SD card/eMMC with a
>> >> >> >> fixed partition layout (something simple: boot partition, primary &
>> >> >> >> secondary rootfs partitions, some data partition). Because future
>> >> >> >> system updates are delivered as filesystem image, I want to make 
>> >> >> >> sure
>> >> >> >> that there is exactly xxx MBs for my current and future rootfs 
>> >> >> >> images
>> >> >> >> (regardless of current image size). Neither --size nor --extra-space
>> >> >> >> can do that. I could use, say `--size 200 --overhead-factor 1`, but
>> >> >> >> this will needlessly create a 200MB rootfs image and if I happen to
>> >> >> >> cross the 200MB boundary I will not get an error.
>> >> >> >>
>> >> >> >> I had a private patch that added --fixed-size to enforce --size, but
>> >> >> >> that would still end up creating filesystem image to fill the whole
>> >> >> >> space.
>> >> >> > I didn't get the difference between enforcing partition size and 
>> >> >> > below
>> >> >> > implementation. Can you elaborate a bit?
>> >> >>
>> >> >> `--fixed-size` was something that I had added to my fork back in 2014,
>> >> >> even before `--overhead-factor` came in. The problem is that depending
>> >> >> on a project you might want to have more control over how partitions
>> >> >> are laid out, or even need to have a fixed layout. Adding
>> >> >> `--fixed-size` would had a similar effect to what `--overhead-factor
>> >> >> 1` does right now. Combined with `--size` would ensure that rootfs is
>> >> >> say, 200MB large. The downside was that wic would actually create a
>> >> >> 200MB rootfs, something that is not really necessary. In fact, I only
>> >> >> wanted to have 200MB gap so that I have some spare space for future
>> >> >> updates (where update is just a rootfs image you dd to the partition).
>> >> >>
>> >> > Thanks for the explanations. Now I got it - reserved size is not a part
>> >> > of partition, it's a gap between partitions.
>> >>
>> >> I might have not been clear enough when explaining. It's not a gap,
>> >> it's just a container of size --reserved-size listed in MBR/GPT.
>> >> There's probably a filesystem inside but not necessarily.
>> >> Graphically it looks as like this:
>> >>
>> >>  --reserved-size
>> >>   |--|
>> >>   v  v
>> >> +-+--+-+
>> >> |. stuff .|xx|. stuff .|
>> >> +-+--+-+
>> >>   ^ ^^
>> >>   |-||
>> >>--size--extra-space
>> >>
>> >>
>> > Ah, I'm wrong again. It's a partition size limit, but it's not necessary
>> > formatted, right? It's only formatted if size == reserved_size.
>> >
>> >> >
>> >> > What's the advantage of creating unusable gap over creating partition of
>> >> > the same size that can be used?
>> >>
>> >> Just convenience.
>> >>
>> > What's the convenience of having extra space on partition that can't be
>> > used for data over having it formatted and used?
>> >
>> >> > Even if that space is not needed it doesn't harm to have it, does it?
>> >>
>> >> I have not seen any negative side effects.
>> >>
>> > I do. If user needs that reserved space it's impossible to get it without
>> > reformatting partition. The free space exists, but can't be used.
>>
>> That's not the point and is not aligned with use case I'm trying to solve.
>>
>> My case is rather simple, I'm creating an image for SD card that is
>> deployed in the 

Re: [OE-core] [PATCH] wic: add --reserved-size wks option

[Ed Bartosh]

On Tue, Oct 18, 2016 at 12:24:55PM +0200, Maciej Borzęcki wrote:
> On Tue, Oct 18, 2016 at 11:27 AM, Ed Bartosh  
> wrote:
> > On Tue, Oct 18, 2016 at 11:24:14AM +0200, Maciej Borzęcki wrote:
> >> On Tue, Oct 18, 2016 at 10:38 AM, Ed Bartosh  
> >> wrote:
> >> > On Tue, Oct 18, 2016 at 10:37:22AM +0200, Maciej Borzęcki wrote:
> >> >> On Tue, Oct 18, 2016 at 9:31 AM, Ed Bartosh 
> >> >>  wrote:
> >> >> > On Mon, Oct 17, 2016 at 04:46:00PM +0200, Maciej Borzęcki wrote:
> >> >> >> On Mon, Oct 17, 2016 at 3:22 PM, Ed Bartosh 
> >> >> >>  wrote:
> >> >> >> > Hi Maciej,
> >> >> >> >
> >> >> >> > There is already --size and --extra-space options.
> >> >> >> > Can we get the same or similar result by just using them? Do we 
> >> >> >> > really
> >> >> >> > need new option for similar purpose?
> >> >> >>
> >> >> >> --reserved-size serves a different purpose, it establishes an upper
> >> >> >> bound on the size of a partition during layout. Unlike
> >> >> >> --size/--extra-space does not depend on the size of the filesystem
> >> >> >> image.
> >> >> >>
> >> >> >> For instance, assume I'm creating an image for SD card/eMMC with a
> >> >> >> fixed partition layout (something simple: boot partition, primary &
> >> >> >> secondary rootfs partitions, some data partition). Because future
> >> >> >> system updates are delivered as filesystem image, I want to make sure
> >> >> >> that there is exactly xxx MBs for my current and future rootfs images
> >> >> >> (regardless of current image size). Neither --size nor --extra-space
> >> >> >> can do that. I could use, say `--size 200 --overhead-factor 1`, but
> >> >> >> this will needlessly create a 200MB rootfs image and if I happen to
> >> >> >> cross the 200MB boundary I will not get an error.
> >> >> >>
> >> >> >> I had a private patch that added --fixed-size to enforce --size, but
> >> >> >> that would still end up creating filesystem image to fill the whole
> >> >> >> space.
> >> >> > I didn't get the difference between enforcing partition size and below
> >> >> > implementation. Can you elaborate a bit?
> >> >>
> >> >> `--fixed-size` was something that I had added to my fork back in 2014,
> >> >> even before `--overhead-factor` came in. The problem is that depending
> >> >> on a project you might want to have more control over how partitions
> >> >> are laid out, or even need to have a fixed layout. Adding
> >> >> `--fixed-size` would had a similar effect to what `--overhead-factor
> >> >> 1` does right now. Combined with `--size` would ensure that rootfs is
> >> >> say, 200MB large. The downside was that wic would actually create a
> >> >> 200MB rootfs, something that is not really necessary. In fact, I only
> >> >> wanted to have 200MB gap so that I have some spare space for future
> >> >> updates (where update is just a rootfs image you dd to the partition).
> >> >>
> >> > Thanks for the explanations. Now I got it - reserved size is not a part
> >> > of partition, it's a gap between partitions.
> >>
> >> I might have not been clear enough when explaining. It's not a gap,
> >> it's just a container of size --reserved-size listed in MBR/GPT.
> >> There's probably a filesystem inside but not necessarily.
> >> Graphically it looks as like this:
> >>
> >>  --reserved-size
> >>   |--|
> >>   v  v
> >> +-+--+-+
> >> |. stuff .|xx|. stuff .|
> >> +-+--+-+
> >>   ^ ^^
> >>   |-||
> >>--size--extra-space
> >>
> >>
> > Ah, I'm wrong again. It's a partition size limit, but it's not necessary
> > formatted, right? It's only formatted if size == reserved_size.
> >
> >> >
> >> > What's the advantage of creating unusable gap over creating partition of
> >> > the same size that can be used?
> >>
> >> Just convenience.
> >>
> > What's the convenience of having extra space on partition that can't be
> > used for data over having it formatted and used?
> >
> >> > Even if that space is not needed it doesn't harm to have it, does it?
> >>
> >> I have not seen any negative side effects.
> >>
> > I do. If user needs that reserved space it's impossible to get it without
> > reformatting partition. The free space exists, but can't be used.
> 
> That's not the point and is not aligned with use case I'm trying to solve.
> 
> My case is rather simple, I'm creating an image for SD card that is
> deployed in the field. In that particular case, there's a primary and
> a secondary (aka. active and inactive) rootfs partitions that are
> switched whenever a system update comes in. The update is a file
> system image that is 

Re: [OE-core] [PATCH] wic: add --reserved-size wks option

[Maciej Borzęcki]

On Tue, Oct 18, 2016 at 11:27 AM, Ed Bartosh  wrote:
> On Tue, Oct 18, 2016 at 11:24:14AM +0200, Maciej Borzęcki wrote:
>> On Tue, Oct 18, 2016 at 10:38 AM, Ed Bartosh  
>> wrote:
>> > On Tue, Oct 18, 2016 at 10:37:22AM +0200, Maciej Borzęcki wrote:
>> >> On Tue, Oct 18, 2016 at 9:31 AM, Ed Bartosh  
>> >> wrote:
>> >> > On Mon, Oct 17, 2016 at 04:46:00PM +0200, Maciej Borzęcki wrote:
>> >> >> On Mon, Oct 17, 2016 at 3:22 PM, Ed Bartosh 
>> >> >>  wrote:
>> >> >> > Hi Maciej,
>> >> >> >
>> >> >> > There is already --size and --extra-space options.
>> >> >> > Can we get the same or similar result by just using them? Do we 
>> >> >> > really
>> >> >> > need new option for similar purpose?
>> >> >>
>> >> >> --reserved-size serves a different purpose, it establishes an upper
>> >> >> bound on the size of a partition during layout. Unlike
>> >> >> --size/--extra-space does not depend on the size of the filesystem
>> >> >> image.
>> >> >>
>> >> >> For instance, assume I'm creating an image for SD card/eMMC with a
>> >> >> fixed partition layout (something simple: boot partition, primary &
>> >> >> secondary rootfs partitions, some data partition). Because future
>> >> >> system updates are delivered as filesystem image, I want to make sure
>> >> >> that there is exactly xxx MBs for my current and future rootfs images
>> >> >> (regardless of current image size). Neither --size nor --extra-space
>> >> >> can do that. I could use, say `--size 200 --overhead-factor 1`, but
>> >> >> this will needlessly create a 200MB rootfs image and if I happen to
>> >> >> cross the 200MB boundary I will not get an error.
>> >> >>
>> >> >> I had a private patch that added --fixed-size to enforce --size, but
>> >> >> that would still end up creating filesystem image to fill the whole
>> >> >> space.
>> >> > I didn't get the difference between enforcing partition size and below
>> >> > implementation. Can you elaborate a bit?
>> >>
>> >> `--fixed-size` was something that I had added to my fork back in 2014,
>> >> even before `--overhead-factor` came in. The problem is that depending
>> >> on a project you might want to have more control over how partitions
>> >> are laid out, or even need to have a fixed layout. Adding
>> >> `--fixed-size` would had a similar effect to what `--overhead-factor
>> >> 1` does right now. Combined with `--size` would ensure that rootfs is
>> >> say, 200MB large. The downside was that wic would actually create a
>> >> 200MB rootfs, something that is not really necessary. In fact, I only
>> >> wanted to have 200MB gap so that I have some spare space for future
>> >> updates (where update is just a rootfs image you dd to the partition).
>> >>
>> > Thanks for the explanations. Now I got it - reserved size is not a part
>> > of partition, it's a gap between partitions.
>>
>> I might have not been clear enough when explaining. It's not a gap,
>> it's just a container of size --reserved-size listed in MBR/GPT.
>> There's probably a filesystem inside but not necessarily.
>> Graphically it looks as like this:
>>
>>  --reserved-size
>>   |--|
>>   v  v
>> +-+--+-+
>> |. stuff .|xx|. stuff .|
>> +-+--+-+
>>   ^ ^^
>>   |-||
>>--size--extra-space
>>
>>
> Ah, I'm wrong again. It's a partition size limit, but it's not necessary
> formatted, right? It's only formatted if size == reserved_size.
>
>> >
>> > What's the advantage of creating unusable gap over creating partition of
>> > the same size that can be used?
>>
>> Just convenience.
>>
> What's the convenience of having extra space on partition that can't be
> used for data over having it formatted and used?
>
>> > Even if that space is not needed it doesn't harm to have it, does it?
>>
>> I have not seen any negative side effects.
>>
> I do. If user needs that reserved space it's impossible to get it without
> reformatting partition. The free space exists, but can't be used.

That's not the point and is not aligned with use case I'm trying to solve.

My case is rather simple, I'm creating an image for SD card that is
deployed in the field. In that particular case, there's a primary and
a secondary (aka. active and inactive) rootfs partitions that are
switched whenever a system update comes in. The update is a file
system image that is copied over to the inactive partition, followed
by a system reboot.

What I need is the ability to set a certain size of a partition (say
100MB), regardless of current rootfs size (which may be, say 70MB).
The remaining unused space sets an upper limit on 

Re: [OE-core] [PATCH] wic: add --reserved-size wks option

[Ed Bartosh]

On Tue, Oct 18, 2016 at 11:24:14AM +0200, Maciej Borzęcki wrote:
> On Tue, Oct 18, 2016 at 10:38 AM, Ed Bartosh  
> wrote:
> > On Tue, Oct 18, 2016 at 10:37:22AM +0200, Maciej Borzęcki wrote:
> >> On Tue, Oct 18, 2016 at 9:31 AM, Ed Bartosh  
> >> wrote:
> >> > On Mon, Oct 17, 2016 at 04:46:00PM +0200, Maciej Borzęcki wrote:
> >> >> On Mon, Oct 17, 2016 at 3:22 PM, Ed Bartosh 
> >> >>  wrote:
> >> >> > Hi Maciej,
> >> >> >
> >> >> > There is already --size and --extra-space options.
> >> >> > Can we get the same or similar result by just using them? Do we really
> >> >> > need new option for similar purpose?
> >> >>
> >> >> --reserved-size serves a different purpose, it establishes an upper
> >> >> bound on the size of a partition during layout. Unlike
> >> >> --size/--extra-space does not depend on the size of the filesystem
> >> >> image.
> >> >>
> >> >> For instance, assume I'm creating an image for SD card/eMMC with a
> >> >> fixed partition layout (something simple: boot partition, primary &
> >> >> secondary rootfs partitions, some data partition). Because future
> >> >> system updates are delivered as filesystem image, I want to make sure
> >> >> that there is exactly xxx MBs for my current and future rootfs images
> >> >> (regardless of current image size). Neither --size nor --extra-space
> >> >> can do that. I could use, say `--size 200 --overhead-factor 1`, but
> >> >> this will needlessly create a 200MB rootfs image and if I happen to
> >> >> cross the 200MB boundary I will not get an error.
> >> >>
> >> >> I had a private patch that added --fixed-size to enforce --size, but
> >> >> that would still end up creating filesystem image to fill the whole
> >> >> space.
> >> > I didn't get the difference between enforcing partition size and below
> >> > implementation. Can you elaborate a bit?
> >>
> >> `--fixed-size` was something that I had added to my fork back in 2014,
> >> even before `--overhead-factor` came in. The problem is that depending
> >> on a project you might want to have more control over how partitions
> >> are laid out, or even need to have a fixed layout. Adding
> >> `--fixed-size` would had a similar effect to what `--overhead-factor
> >> 1` does right now. Combined with `--size` would ensure that rootfs is
> >> say, 200MB large. The downside was that wic would actually create a
> >> 200MB rootfs, something that is not really necessary. In fact, I only
> >> wanted to have 200MB gap so that I have some spare space for future
> >> updates (where update is just a rootfs image you dd to the partition).
> >>
> > Thanks for the explanations. Now I got it - reserved size is not a part
> > of partition, it's a gap between partitions.
> 
> I might have not been clear enough when explaining. It's not a gap,
> it's just a container of size --reserved-size listed in MBR/GPT.
> There's probably a filesystem inside but not necessarily.
> Graphically it looks as like this:
> 
>  --reserved-size
>   |--|
>   v  v
> +-+--+-+
> |. stuff .|xx|. stuff .|
> +-+--+-+
>   ^ ^^
>   |-||
>--size--extra-space
> 
> 
Ah, I'm wrong again. It's a partition size limit, but it's not necessary
formatted, right? It's only formatted if size == reserved_size.

> >
> > What's the advantage of creating unusable gap over creating partition of
> > the same size that can be used?
> 
> Just convenience.
>
What's the convenience of having extra space on partition that can't be
used for data over having it formatted and used?

> > Even if that space is not needed it doesn't harm to have it, does it?
> 
> I have not seen any negative side effects.
>
I do. If user needs that reserved space it's impossible to get it without
reformatting partition. The free space exists, but can't be used.

--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] wic: add --reserved-size wks option

[Maciej Borzęcki]

On Tue, Oct 18, 2016 at 10:38 AM, Ed Bartosh  wrote:
> On Tue, Oct 18, 2016 at 10:37:22AM +0200, Maciej Borzęcki wrote:
>> On Tue, Oct 18, 2016 at 9:31 AM, Ed Bartosh  
>> wrote:
>> > On Mon, Oct 17, 2016 at 04:46:00PM +0200, Maciej Borzęcki wrote:
>> >> On Mon, Oct 17, 2016 at 3:22 PM, Ed Bartosh  
>> >> wrote:
>> >> > Hi Maciej,
>> >> >
>> >> > There is already --size and --extra-space options.
>> >> > Can we get the same or similar result by just using them? Do we really
>> >> > need new option for similar purpose?
>> >>
>> >> --reserved-size serves a different purpose, it establishes an upper
>> >> bound on the size of a partition during layout. Unlike
>> >> --size/--extra-space does not depend on the size of the filesystem
>> >> image.
>> >>
>> >> For instance, assume I'm creating an image for SD card/eMMC with a
>> >> fixed partition layout (something simple: boot partition, primary &
>> >> secondary rootfs partitions, some data partition). Because future
>> >> system updates are delivered as filesystem image, I want to make sure
>> >> that there is exactly xxx MBs for my current and future rootfs images
>> >> (regardless of current image size). Neither --size nor --extra-space
>> >> can do that. I could use, say `--size 200 --overhead-factor 1`, but
>> >> this will needlessly create a 200MB rootfs image and if I happen to
>> >> cross the 200MB boundary I will not get an error.
>> >>
>> >> I had a private patch that added --fixed-size to enforce --size, but
>> >> that would still end up creating filesystem image to fill the whole
>> >> space.
>> > I didn't get the difference between enforcing partition size and below
>> > implementation. Can you elaborate a bit?
>>
>> `--fixed-size` was something that I had added to my fork back in 2014,
>> even before `--overhead-factor` came in. The problem is that depending
>> on a project you might want to have more control over how partitions
>> are laid out, or even need to have a fixed layout. Adding
>> `--fixed-size` would had a similar effect to what `--overhead-factor
>> 1` does right now. Combined with `--size` would ensure that rootfs is
>> say, 200MB large. The downside was that wic would actually create a
>> 200MB rootfs, something that is not really necessary. In fact, I only
>> wanted to have 200MB gap so that I have some spare space for future
>> updates (where update is just a rootfs image you dd to the partition).
>>
> Thanks for the explanations. Now I got it - reserved size is not a part
> of partition, it's a gap between partitions.

I might have not been clear enough when explaining. It's not a gap,
it's just a container of size --reserved-size listed in MBR/GPT.
There's probably a filesystem inside but not necessarily.
Graphically it looks as like this:

 --reserved-size
  |--|
  v  v
+-+--+-+
|. stuff .|xx|. stuff .|
+-+--+-+
  ^ ^^
  |-||
   --size--extra-space


>
> What's the advantage of creating unusable gap over creating partition of
> the same size that can be used?

Just convenience.

> Even if that space is not needed it doesn't harm to have it, does it?

I have not seen any negative side effects.

>
>> >> >>
>> >> >> + --reserved-size: This option specifies that there should be
>> >> >> +  at least that many bytes reserved for
>> > According to the code if partition size > reserved size wic will throw
>> > an error. This somehow conflicts with 'at least that many bytes' part of 
>> > the
>> > description in my opinion.
>> >
>> > As far as I uderstood your code this parameter specifies hard upper
>> > limit for partition size, right?
>>
>> Thanks, the wording is a bit off. I should rephrase it to 'that many
>> bytes' or just state that it's an upper limit.
>>
> Upper limit is a wrong term here. I'm sorry for confusing you.
> I proposed it because I misunderstood that it's not a partition size but a 
> reserved
> space after partition. Anyway, I think above description should be
> modified to explain the option in a clear way. Using ascii picture of
> the partition layout would probably help to show what this option means.
>
>> >> >>  msger.debug("Assigned %s to %s%d, sectors range %d-%d 
>> >> >> size %d "
>> >> >> -"sectors (%d bytes)." \
>> >> >> +"sectors (%d bytes), reserved %d bytes." \
>> >> >>  % (part['mountpoint'], part['disk_name'], 
>> >> >> part['num'],
>> >> >> -   part['start'], part['start'] + 
>> >> >> part['size'] - 1,
>> 

Re: [OE-core] [PATCH] wic: add --reserved-size wks option

[Ed Bartosh]

On Tue, Oct 18, 2016 at 10:37:22AM +0200, Maciej Borzęcki wrote:
> On Tue, Oct 18, 2016 at 9:31 AM, Ed Bartosh  
> wrote:
> > On Mon, Oct 17, 2016 at 04:46:00PM +0200, Maciej Borzęcki wrote:
> >> On Mon, Oct 17, 2016 at 3:22 PM, Ed Bartosh  
> >> wrote:
> >> > Hi Maciej,
> >> >
> >> > There is already --size and --extra-space options.
> >> > Can we get the same or similar result by just using them? Do we really
> >> > need new option for similar purpose?
> >>
> >> --reserved-size serves a different purpose, it establishes an upper
> >> bound on the size of a partition during layout. Unlike
> >> --size/--extra-space does not depend on the size of the filesystem
> >> image.
> >>
> >> For instance, assume I'm creating an image for SD card/eMMC with a
> >> fixed partition layout (something simple: boot partition, primary &
> >> secondary rootfs partitions, some data partition). Because future
> >> system updates are delivered as filesystem image, I want to make sure
> >> that there is exactly xxx MBs for my current and future rootfs images
> >> (regardless of current image size). Neither --size nor --extra-space
> >> can do that. I could use, say `--size 200 --overhead-factor 1`, but
> >> this will needlessly create a 200MB rootfs image and if I happen to
> >> cross the 200MB boundary I will not get an error.
> >>
> >> I had a private patch that added --fixed-size to enforce --size, but
> >> that would still end up creating filesystem image to fill the whole
> >> space.
> > I didn't get the difference between enforcing partition size and below
> > implementation. Can you elaborate a bit?
> 
> `--fixed-size` was something that I had added to my fork back in 2014,
> even before `--overhead-factor` came in. The problem is that depending
> on a project you might want to have more control over how partitions
> are laid out, or even need to have a fixed layout. Adding
> `--fixed-size` would had a similar effect to what `--overhead-factor
> 1` does right now. Combined with `--size` would ensure that rootfs is
> say, 200MB large. The downside was that wic would actually create a
> 200MB rootfs, something that is not really necessary. In fact, I only
> wanted to have 200MB gap so that I have some spare space for future
> updates (where update is just a rootfs image you dd to the partition).
>
Thanks for the explanations. Now I got it - reserved size is not a part
of partition, it's a gap between partitions.

What's the advantage of creating unusable gap over creating partition of
the same size that can be used? Even if that space is not needed it
doesn't harm to have it, does it?

> >> >>
> >> >> + --reserved-size: This option specifies that there should be
> >> >> +  at least that many bytes reserved for
> > According to the code if partition size > reserved size wic will throw
> > an error. This somehow conflicts with 'at least that many bytes' part of the
> > description in my opinion.
> >
> > As far as I uderstood your code this parameter specifies hard upper
> > limit for partition size, right?
> 
> Thanks, the wording is a bit off. I should rephrase it to 'that many
> bytes' or just state that it's an upper limit.
>
Upper limit is a wrong term here. I'm sorry for confusing you.
I proposed it because I misunderstood that it's not a partition size but a 
reserved
space after partition. Anyway, I think above description should be
modified to explain the option in a clear way. Using ascii picture of
the partition layout would probably help to show what this option means.

> >> >>  msger.debug("Assigned %s to %s%d, sectors range %d-%d size 
> >> >> %d "
> >> >> -"sectors (%d bytes)." \
> >> >> +"sectors (%d bytes), reserved %d bytes." \
> >> >>  % (part['mountpoint'], part['disk_name'], 
> >> >> part['num'],
> >> >> -   part['start'], part['start'] + 
> >> >> part['size'] - 1,
> >> >> -   part['size'], part['size'] * 
> >> >> self.sector_size))
> >> >> +   part['start'], disk['offset'] - 1,
> >> >> +   part['size'], part['size'] * 
> >> >> self.sector_size,
> >> >> +   part['reserved_size'] * 
> >> >> self.sector_size))
> > This will produce incorrect output about partition size as in reality 
> > partition size will
> > be set to part['reserved_size'].
> 
> I tried to avoid confusing the reader here. Partition size is one
> thing and reservation is another. That is why reservation is logged
> explicitly. Logging just the size would raise questions about why the
> partition has grown.
> 
> Perhaps it makes sense to log reservation like this instead?
> 
What about using 'reserved %d bytes of space after partition' or
something like that?

>part['reserved_size'] * self.sector_size if part['reserved_size']

Re: [OE-core] [PATCH] wic: add --reserved-size wks option

[Maciej Borzęcki]

On Tue, Oct 18, 2016 at 9:31 AM, Ed Bartosh  wrote:
> On Mon, Oct 17, 2016 at 04:46:00PM +0200, Maciej Borzęcki wrote:
>> On Mon, Oct 17, 2016 at 3:22 PM, Ed Bartosh  
>> wrote:
>> > Hi Maciej,
>> >
>> > There is already --size and --extra-space options.
>> > Can we get the same or similar result by just using them? Do we really
>> > need new option for similar purpose?
>>
>> --reserved-size serves a different purpose, it establishes an upper
>> bound on the size of a partition during layout. Unlike
>> --size/--extra-space does not depend on the size of the filesystem
>> image.
>>
>> For instance, assume I'm creating an image for SD card/eMMC with a
>> fixed partition layout (something simple: boot partition, primary &
>> secondary rootfs partitions, some data partition). Because future
>> system updates are delivered as filesystem image, I want to make sure
>> that there is exactly xxx MBs for my current and future rootfs images
>> (regardless of current image size). Neither --size nor --extra-space
>> can do that. I could use, say `--size 200 --overhead-factor 1`, but
>> this will needlessly create a 200MB rootfs image and if I happen to
>> cross the 200MB boundary I will not get an error.
>>
>> I had a private patch that added --fixed-size to enforce --size, but
>> that would still end up creating filesystem image to fill the whole
>> space.
> I didn't get the difference between enforcing partition size and below
> implementation. Can you elaborate a bit?

`--fixed-size` was something that I had added to my fork back in 2014,
even before `--overhead-factor` came in. The problem is that depending
on a project you might want to have more control over how partitions
are laid out, or even need to have a fixed layout. Adding
`--fixed-size` would had a similar effect to what `--overhead-factor
1` does right now. Combined with `--size` would ensure that rootfs is
say, 200MB large. The downside was that wic would actually create a
200MB rootfs, something that is not really necessary. In fact, I only
wanted to have 200MB gap so that I have some spare space for future
updates (where update is just a rootfs image you dd to the partition).


>
>> Another workaround we used until now was to create a filesystem image
>> manually and use fsimage source plugin. To get a fixed layout with
>> rootfs image that may grow/shrink between builds, you still need to
>> fiddle with --align (for instance, set it to 215040 to have the
>> partition start at 210MB offset).
>>
>
> Thanks for the explanations. I'm almost convinced.
>
> See my comments below.
>
>> >
>> >
>> > On Mon, Oct 17, 2016 at 03:06:18PM +0200, Maciej Borzecki wrote:
>> >> Added new option --reserved-size to wks. The option can be used to
>> >> indicate how much space should be reserved for a partition. This is
>> >> useful if the disk will be a subject to full filesystem image updates
>> >> and puts an upper limit of the size of filesystem images.
>> >>
>> >> The actual filesystem image may be smaller than the reserved space, thus
>> >> leaving some room for growth. If it is larger, an error will be raised.
>> >>
>> >> Signed-off-by: Maciej Borzecki 
>> >> ---
>> >>  scripts/lib/wic/help.py| 11 ++
>> >>  scripts/lib/wic/imager/direct.py   |  3 ++-
>> >>  scripts/lib/wic/ksparser.py|  1 +
>> >>  scripts/lib/wic/partition.py   |  1 +
>> >>  scripts/lib/wic/utils/partitionedfs.py | 37 
>> >> +-
>> >>  5 files changed, 43 insertions(+), 10 deletions(-)
>> >>
>> >> diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
>> >> index 
>> >> e5347ec4b7c900c68fc64351a5293e75de0672b3..1a5c7020ba0cdc5ef2e477a2b14360e09098a896
>> >>  100644
>> >> --- a/scripts/lib/wic/help.py
>> >> +++ b/scripts/lib/wic/help.py
>> >> @@ -646,6 +646,17 @@ DESCRIPTION
>> >>   not specified, the size is in MB.
>> >>   You do not need this option if you use --source.
>> >>
>> >> + --reserved-size: This option specifies that there should be
>> >> +  at least that many bytes reserved for
> According to the code if partition size > reserved size wic will throw
> an error. This somehow conflicts with 'at least that many bytes' part of the
> description in my opinion.
>
> As far as I uderstood your code this parameter specifies hard upper
> limit for partition size, right?

Thanks, the wording is a bit off. I should rephrase it to 'that many
bytes' or just state that it's an upper limit.


>
>> >> +  the partition during layout. This is useful
>> >> +  when the target disk will be a subject
>> >> +  to full system image updates in the future.
>> >> +  Specifying --reserved-size ensures that
>> >> +  there is extra space in the partition allowing
>> 

Re: [OE-core] [PATCH] wic: add --reserved-size wks option

[Ed Bartosh]

On Mon, Oct 17, 2016 at 04:46:00PM +0200, Maciej Borzęcki wrote:
> On Mon, Oct 17, 2016 at 3:22 PM, Ed Bartosh  
> wrote:
> > Hi Maciej,
> >
> > There is already --size and --extra-space options.
> > Can we get the same or similar result by just using them? Do we really
> > need new option for similar purpose?
> 
> --reserved-size serves a different purpose, it establishes an upper
> bound on the size of a partition during layout. Unlike
> --size/--extra-space does not depend on the size of the filesystem
> image.
> 
> For instance, assume I'm creating an image for SD card/eMMC with a
> fixed partition layout (something simple: boot partition, primary &
> secondary rootfs partitions, some data partition). Because future
> system updates are delivered as filesystem image, I want to make sure
> that there is exactly xxx MBs for my current and future rootfs images
> (regardless of current image size). Neither --size nor --extra-space
> can do that. I could use, say `--size 200 --overhead-factor 1`, but
> this will needlessly create a 200MB rootfs image and if I happen to
> cross the 200MB boundary I will not get an error.
> 
> I had a private patch that added --fixed-size to enforce --size, but
> that would still end up creating filesystem image to fill the whole
> space.
I didn't get the difference between enforcing partition size and below
implementation. Can you elaborate a bit?

> Another workaround we used until now was to create a filesystem image
> manually and use fsimage source plugin. To get a fixed layout with
> rootfs image that may grow/shrink between builds, you still need to
> fiddle with --align (for instance, set it to 215040 to have the
> partition start at 210MB offset).
> 

Thanks for the explanations. I'm almost convinced.

See my comments below.

> >
> >
> > On Mon, Oct 17, 2016 at 03:06:18PM +0200, Maciej Borzecki wrote:
> >> Added new option --reserved-size to wks. The option can be used to
> >> indicate how much space should be reserved for a partition. This is
> >> useful if the disk will be a subject to full filesystem image updates
> >> and puts an upper limit of the size of filesystem images.
> >>
> >> The actual filesystem image may be smaller than the reserved space, thus
> >> leaving some room for growth. If it is larger, an error will be raised.
> >>
> >> Signed-off-by: Maciej Borzecki 
> >> ---
> >>  scripts/lib/wic/help.py| 11 ++
> >>  scripts/lib/wic/imager/direct.py   |  3 ++-
> >>  scripts/lib/wic/ksparser.py|  1 +
> >>  scripts/lib/wic/partition.py   |  1 +
> >>  scripts/lib/wic/utils/partitionedfs.py | 37 
> >> +-
> >>  5 files changed, 43 insertions(+), 10 deletions(-)
> >>
> >> diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
> >> index 
> >> e5347ec4b7c900c68fc64351a5293e75de0672b3..1a5c7020ba0cdc5ef2e477a2b14360e09098a896
> >>  100644
> >> --- a/scripts/lib/wic/help.py
> >> +++ b/scripts/lib/wic/help.py
> >> @@ -646,6 +646,17 @@ DESCRIPTION
> >>   not specified, the size is in MB.
> >>   You do not need this option if you use --source.
> >>
> >> + --reserved-size: This option specifies that there should be
> >> +  at least that many bytes reserved for
According to the code if partition size > reserved size wic will throw
an error. This somehow conflicts with 'at least that many bytes' part of the
description in my opinion.

As far as I uderstood your code this parameter specifies hard upper
limit for partition size, right?

> >> +  the partition during layout. This is useful
> >> +  when the target disk will be a subject
> >> +  to full system image updates in the future.
> >> +  Specifying --reserved-size ensures that
> >> +  there is extra space in the partition allowing
> >> +  for future growth of the file system stored
> >> +  inside. Value format is the same as for
> >> +  --size option.
> >> +
> >>   --source: This option is a wic-specific option that names the
> >> source of the data that will populate the
> >> partition.  The most common value for this option
> >> diff --git a/scripts/lib/wic/imager/direct.py 
> >> b/scripts/lib/wic/imager/direct.py
> >> index 
> >> edf5e5d2214f8e78b6c2a98d7f6cd45fcc0065c4..02e293b9d744b760fcdf17610505dafef3e164ad
> >>  100644
> >> --- a/scripts/lib/wic/imager/direct.py
> >> +++ b/scripts/lib/wic/imager/direct.py
> >> @@ -301,7 +301,8 @@ class DirectImageCreator(BaseImageCreator):
> >> no_table=part.no_table,
> >> part_type=part.part_type,
> >> uuid=part.uuid,

Re: [OE-core] [PATCH] wic: add --reserved-size wks option

[Maciej Borzęcki]

On Mon, Oct 17, 2016 at 3:22 PM, Ed Bartosh  wrote:
> Hi Maciej,
>
> There is already --size and --extra-space options.
> Can we get the same or similar result by just using them? Do we really
> need new option for similar purpose?

--reserved-size serves a different purpose, it establishes an upper
bound on the size of a partition during layout. Unlike
--size/--extra-space does not depend on the size of the filesystem
image.

For instance, assume I'm creating an image for SD card/eMMC with a
fixed partition layout (something simple: boot partition, primary &
secondary rootfs partitions, some data partition). Because future
system updates are delivered as filesystem image, I want to make sure
that there is exactly xxx MBs for my current and future rootfs images
(regardless of current image size). Neither --size nor --extra-space
can do that. I could use, say `--size 200 --overhead-factor 1`, but
this will needlessly create a 200MB rootfs image and if I happen to
cross the 200MB boundary I will not get an error.

I had a private patch that added --fixed-size to enforce --size, but
that would still end up creating filesystem image to fill the whole
space.
Another workaround we used until now was to create a filesystem image
manually and use fsimage source plugin. To get a fixed layout with
rootfs image that may grow/shrink between builds, you still need to
fiddle with --align (for instance, set it to 215040 to have the
partition start at 210MB offset).

>
>
> On Mon, Oct 17, 2016 at 03:06:18PM +0200, Maciej Borzecki wrote:
>> Added new option --reserved-size to wks. The option can be used to
>> indicate how much space should be reserved for a partition. This is
>> useful if the disk will be a subject to full filesystem image updates
>> and puts an upper limit of the size of filesystem images.
>>
>> The actual filesystem image may be smaller than the reserved space, thus
>> leaving some room for growth. If it is larger, an error will be raised.
>>
>> Signed-off-by: Maciej Borzecki 
>> ---
>>  scripts/lib/wic/help.py| 11 ++
>>  scripts/lib/wic/imager/direct.py   |  3 ++-
>>  scripts/lib/wic/ksparser.py|  1 +
>>  scripts/lib/wic/partition.py   |  1 +
>>  scripts/lib/wic/utils/partitionedfs.py | 37 
>> +-
>>  5 files changed, 43 insertions(+), 10 deletions(-)
>>
>> diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
>> index 
>> e5347ec4b7c900c68fc64351a5293e75de0672b3..1a5c7020ba0cdc5ef2e477a2b14360e09098a896
>>  100644
>> --- a/scripts/lib/wic/help.py
>> +++ b/scripts/lib/wic/help.py
>> @@ -646,6 +646,17 @@ DESCRIPTION
>>   not specified, the size is in MB.
>>   You do not need this option if you use --source.
>>
>> + --reserved-size: This option specifies that there should be
>> +  at least that many bytes reserved for
>> +  the partition during layout. This is useful
>> +  when the target disk will be a subject
>> +  to full system image updates in the future.
>> +  Specifying --reserved-size ensures that
>> +  there is extra space in the partition allowing
>> +  for future growth of the file system stored
>> +  inside. Value format is the same as for
>> +  --size option.
>> +
>>   --source: This option is a wic-specific option that names the
>> source of the data that will populate the
>> partition.  The most common value for this option
>> diff --git a/scripts/lib/wic/imager/direct.py 
>> b/scripts/lib/wic/imager/direct.py
>> index 
>> edf5e5d2214f8e78b6c2a98d7f6cd45fcc0065c4..02e293b9d744b760fcdf17610505dafef3e164ad
>>  100644
>> --- a/scripts/lib/wic/imager/direct.py
>> +++ b/scripts/lib/wic/imager/direct.py
>> @@ -301,7 +301,8 @@ class DirectImageCreator(BaseImageCreator):
>> no_table=part.no_table,
>> part_type=part.part_type,
>> uuid=part.uuid,
>> -   system_id=part.system_id)
>> +   system_id=part.system_id,
>> +   reserved_size=part.reserved_size)
>>
>>  if fstab_path:
>>  shutil.move(fstab_path + ".orig", fstab_path)
>> diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
>> index 
>> 0894e2b199a299fbbed272f2e1c95e9d692e3ab1..4118bffdf4337f2d2d393d7e096632cd7aa37402
>>  100644
>> --- a/scripts/lib/wic/ksparser.py
>> +++ b/scripts/lib/wic/ksparser.py
>> @@ -137,6 +137,7 @@ class KickStart():
>>  part.add_argument('--part-type')
>>  part.add_argument('--rootfs-dir')
>>  

Re: [OE-core] [PATCH] wic: add --reserved-size wks option

[Ed Bartosh]

Hi Maciej,

There is already --size and --extra-space options.
Can we get the same or similar result by just using them? Do we really
need new option for similar purpose?


On Mon, Oct 17, 2016 at 03:06:18PM +0200, Maciej Borzecki wrote:
> Added new option --reserved-size to wks. The option can be used to
> indicate how much space should be reserved for a partition. This is
> useful if the disk will be a subject to full filesystem image updates
> and puts an upper limit of the size of filesystem images.
> 
> The actual filesystem image may be smaller than the reserved space, thus
> leaving some room for growth. If it is larger, an error will be raised.
> 
> Signed-off-by: Maciej Borzecki 
> ---
>  scripts/lib/wic/help.py| 11 ++
>  scripts/lib/wic/imager/direct.py   |  3 ++-
>  scripts/lib/wic/ksparser.py|  1 +
>  scripts/lib/wic/partition.py   |  1 +
>  scripts/lib/wic/utils/partitionedfs.py | 37 
> +-
>  5 files changed, 43 insertions(+), 10 deletions(-)
> 
> diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
> index 
> e5347ec4b7c900c68fc64351a5293e75de0672b3..1a5c7020ba0cdc5ef2e477a2b14360e09098a896
>  100644
> --- a/scripts/lib/wic/help.py
> +++ b/scripts/lib/wic/help.py
> @@ -646,6 +646,17 @@ DESCRIPTION
>   not specified, the size is in MB.
>   You do not need this option if you use --source.
>  
> + --reserved-size: This option specifies that there should be
> +  at least that many bytes reserved for
> +  the partition during layout. This is useful
> +  when the target disk will be a subject
> +  to full system image updates in the future.
> +  Specifying --reserved-size ensures that
> +  there is extra space in the partition allowing
> +  for future growth of the file system stored
> +  inside. Value format is the same as for
> +  --size option.
> +
>   --source: This option is a wic-specific option that names the
> source of the data that will populate the
> partition.  The most common value for this option
> diff --git a/scripts/lib/wic/imager/direct.py 
> b/scripts/lib/wic/imager/direct.py
> index 
> edf5e5d2214f8e78b6c2a98d7f6cd45fcc0065c4..02e293b9d744b760fcdf17610505dafef3e164ad
>  100644
> --- a/scripts/lib/wic/imager/direct.py
> +++ b/scripts/lib/wic/imager/direct.py
> @@ -301,7 +301,8 @@ class DirectImageCreator(BaseImageCreator):
> no_table=part.no_table,
> part_type=part.part_type,
> uuid=part.uuid,
> -   system_id=part.system_id)
> +   system_id=part.system_id,
> +   reserved_size=part.reserved_size)
>  
>  if fstab_path:
>  shutil.move(fstab_path + ".orig", fstab_path)
> diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
> index 
> 0894e2b199a299fbbed272f2e1c95e9d692e3ab1..4118bffdf4337f2d2d393d7e096632cd7aa37402
>  100644
> --- a/scripts/lib/wic/ksparser.py
> +++ b/scripts/lib/wic/ksparser.py
> @@ -137,6 +137,7 @@ class KickStart():
>  part.add_argument('--part-type')
>  part.add_argument('--rootfs-dir')
>  part.add_argument('--size', type=sizetype, default=0)
> +part.add_argument('--reserved-size', type=sizetype, default=0)
>  part.add_argument('--source')
>  part.add_argument('--sourceparams')
>  part.add_argument('--system-id', type=systemidtype)
> diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
> index 
> 90f65a1e3976a5460cd1b265b238168cce22781f..162a3a289de891ccf81437876c1f7a6f3c797b3b
>  100644
> --- a/scripts/lib/wic/partition.py
> +++ b/scripts/lib/wic/partition.py
> @@ -54,6 +54,7 @@ class Partition():
>  self.part_type = args.part_type
>  self.rootfs_dir = args.rootfs_dir
>  self.size = args.size
> +self.reserved_size = args.reserved_size
>  self.source = args.source
>  self.sourceparams = args.sourceparams
>  self.system_id = args.system_id
> diff --git a/scripts/lib/wic/utils/partitionedfs.py 
> b/scripts/lib/wic/utils/partitionedfs.py
> index 
> cb03009fc7e3c97305079629ded7d2ff01eba4c4..5d3b1588231459dedf0142f807114736f0bb28ea
>  100644
> --- a/scripts/lib/wic/utils/partitionedfs.py
> +++ b/scripts/lib/wic/utils/partitionedfs.py
> @@ -91,7 +91,7 @@ class Image():
>  
>  def add_partition(self, size, disk_name, mountpoint, source_file=None, 
> fstype=None,
>label=None, fsopts=None, boot=False, align=None, 
> no_table=False,
> - 

[OE-core] [PATCH] wic: add --reserved-size wks option

[Maciej Borzecki]

Added new option --reserved-size to wks. The option can be used to
indicate how much space should be reserved for a partition. This is
useful if the disk will be a subject to full filesystem image updates
and puts an upper limit of the size of filesystem images.

The actual filesystem image may be smaller than the reserved space, thus
leaving some room for growth. If it is larger, an error will be raised.

Signed-off-by: Maciej Borzecki 
---
 scripts/lib/wic/help.py| 11 ++
 scripts/lib/wic/imager/direct.py   |  3 ++-
 scripts/lib/wic/ksparser.py|  1 +
 scripts/lib/wic/partition.py   |  1 +
 scripts/lib/wic/utils/partitionedfs.py | 37 +-
 5 files changed, 43 insertions(+), 10 deletions(-)

diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
index 
e5347ec4b7c900c68fc64351a5293e75de0672b3..1a5c7020ba0cdc5ef2e477a2b14360e09098a896
 100644
--- a/scripts/lib/wic/help.py
+++ b/scripts/lib/wic/help.py
@@ -646,6 +646,17 @@ DESCRIPTION
  not specified, the size is in MB.
  You do not need this option if you use --source.
 
+ --reserved-size: This option specifies that there should be
+  at least that many bytes reserved for
+  the partition during layout. This is useful
+  when the target disk will be a subject
+  to full system image updates in the future.
+  Specifying --reserved-size ensures that
+  there is extra space in the partition allowing
+  for future growth of the file system stored
+  inside. Value format is the same as for
+  --size option.
+
  --source: This option is a wic-specific option that names the
source of the data that will populate the
partition.  The most common value for this option
diff --git a/scripts/lib/wic/imager/direct.py b/scripts/lib/wic/imager/direct.py
index 
edf5e5d2214f8e78b6c2a98d7f6cd45fcc0065c4..02e293b9d744b760fcdf17610505dafef3e164ad
 100644
--- a/scripts/lib/wic/imager/direct.py
+++ b/scripts/lib/wic/imager/direct.py
@@ -301,7 +301,8 @@ class DirectImageCreator(BaseImageCreator):
no_table=part.no_table,
part_type=part.part_type,
uuid=part.uuid,
-   system_id=part.system_id)
+   system_id=part.system_id,
+   reserved_size=part.reserved_size)
 
 if fstab_path:
 shutil.move(fstab_path + ".orig", fstab_path)
diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
index 
0894e2b199a299fbbed272f2e1c95e9d692e3ab1..4118bffdf4337f2d2d393d7e096632cd7aa37402
 100644
--- a/scripts/lib/wic/ksparser.py
+++ b/scripts/lib/wic/ksparser.py
@@ -137,6 +137,7 @@ class KickStart():
 part.add_argument('--part-type')
 part.add_argument('--rootfs-dir')
 part.add_argument('--size', type=sizetype, default=0)
+part.add_argument('--reserved-size', type=sizetype, default=0)
 part.add_argument('--source')
 part.add_argument('--sourceparams')
 part.add_argument('--system-id', type=systemidtype)
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 
90f65a1e3976a5460cd1b265b238168cce22781f..162a3a289de891ccf81437876c1f7a6f3c797b3b
 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -54,6 +54,7 @@ class Partition():
 self.part_type = args.part_type
 self.rootfs_dir = args.rootfs_dir
 self.size = args.size
+self.reserved_size = args.reserved_size
 self.source = args.source
 self.sourceparams = args.sourceparams
 self.system_id = args.system_id
diff --git a/scripts/lib/wic/utils/partitionedfs.py 
b/scripts/lib/wic/utils/partitionedfs.py
index 
cb03009fc7e3c97305079629ded7d2ff01eba4c4..5d3b1588231459dedf0142f807114736f0bb28ea
 100644
--- a/scripts/lib/wic/utils/partitionedfs.py
+++ b/scripts/lib/wic/utils/partitionedfs.py
@@ -91,7 +91,7 @@ class Image():
 
 def add_partition(self, size, disk_name, mountpoint, source_file=None, 
fstype=None,
   label=None, fsopts=None, boot=False, align=None, 
no_table=False,
-  part_type=None, uuid=None, system_id=None):
+  part_type=None, uuid=None, system_id=None, 
reserved_size=0):
 """ Add the next partition. Prtitions have to be added in the
 first-to-last order. """
 
@@ -99,9 +99,11 @@ class Image():
 
 # Converting kB to sectors for parted
 size = size * 1024 // self.sector_size
+reserved_size = reserved_size * 1024 //