Re: Problem with ISO9660 and files stored on multiple extents

2017-02-17 Thread Andrei Borzenkov 
17.02.2017 21:18, Andrei Borzenkov пишет:
> 
> Not quite. We assume that fragment and block list are mutually
> exclusive, while squashfs can pack file tail that does not fill full
> block as fragment. So we actually need to do direct_read() 

And BTW it does not look like direct_read() copes with sparse files either.

___
Bug-grub mailing list
Bug-grub@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-grub

Re: Problem with ISO9660 and files stored on multiple extents

2017-02-17 Thread Andrei Borzenkov 
16.02.2017 20:42, Carlo Caione пишет:
> On Thu, Feb 16, 2017 at 5:52 PM, Andrei Borzenkov  wrote:
>> 16.02.2017 19:04, Vladimir 'phcoder' Serbinenko пишет:
>>> On Thu, Feb 16, 2017, 16:56 Carlo Caione  wrote:
> 
>>> Then it's possible that your patch is correct. I don't know why the format
>>> changes or what the 0 means in this field
>>>
>>
>> Looking at current squash tools git, 0 is valid fragment number. Any
>> chance we hit different file type here? Carlo, could you make your
>> squashfs image available?
> 
> Here the whole non-booting ISO
> https://d1anzknqnc1kmb.cloudfront.net/eos-amd64-amd64/eos3.1/sea/170214-155339/eos-eos3.1-amd64-amd64.170214-155339.sea.iso
> (`endless/endless.squash` is the squash image)
> `qemu-system-x86_64 -m 512 -machine q35,accel=kvm -hda $iso` drops you
> into a GRUB shell (same booting with EFI). For smaller images it boots
> fine.
> 
> We actually solved this problem creating the squashfs passing the
> option `-no-fragments` to `mksquashfs`. So I guess that really the
> squash4 module has some bug when dealing with big images with a number
> of fragments !=0.
> 

Not quite. We assume that fragment and block list are mutually
exclusive, while squashfs can pack file tail that does not fill full
block as fragment. So we actually need to do direct_read() followed by
the rest of grub_squash_read_data() if fragment is not invalid. Your
patch worked by accident (because there was only one fragment with
number 0) and actually made it truncate file.

I need to wrap my head around our code unless Vladimir beats me (I'm
sure he will :) )

For reference, here is what unsquashfs does

for(i = 0; i < inode->blocks; i++) {
... read blocks etc. This is our direct_read
}

if(inode->frag_bytes) {
...
s_ops.read_fragment(inode->fragment, , );
block->buffer = cache_get(fragment_cache, start, size);
block->offset = inode->offset;
block->size = inode->frag_bytes;
queue_put(to_writer, block);
}

inode->frag_bytes != 0 if fragment != 0x and file size is not
exact multiple of blocks.

i.frag_bytes = inode->fragment ==
SQUASHFS_INVALID_FRAG
?  0 : inode->file_size % sBlk.s.block_size;


___
Bug-grub mailing list
Bug-grub@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-grub

Re: Problem with ISO9660 and files stored on multiple extents

2017-02-16 Thread Carlo Caione 
On Thu, Feb 16, 2017 at 5:52 PM, Andrei Borzenkov  wrote:
> 16.02.2017 19:04, Vladimir 'phcoder' Serbinenko пишет:
>> On Thu, Feb 16, 2017, 16:56 Carlo Caione  wrote:

>> Then it's possible that your patch is correct. I don't know why the format
>> changes or what the 0 means in this field
>>
>
> Looking at current squash tools git, 0 is valid fragment number. Any
> chance we hit different file type here? Carlo, could you make your
> squashfs image available?

Here the whole non-booting ISO
https://d1anzknqnc1kmb.cloudfront.net/eos-amd64-amd64/eos3.1/sea/170214-155339/eos-eos3.1-amd64-amd64.170214-155339.sea.iso
(`endless/endless.squash` is the squash image)
`qemu-system-x86_64 -m 512 -machine q35,accel=kvm -hda $iso` drops you
into a GRUB shell (same booting with EFI). For smaller images it boots
fine.

We actually solved this problem creating the squashfs passing the
option `-no-fragments` to `mksquashfs`. So I guess that really the
squash4 module has some bug when dealing with big images with a number
of fragments !=0.

-- 
Carlo Caione  |  +39.340.80.30.096  |  Endless

___
Bug-grub mailing list
Bug-grub@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-grub

Re: Problem with ISO9660 and files stored on multiple extents

2017-02-16 Thread Vladimir 'phcoder' Serbinenko 
On Thu, Feb 16, 2017, 16:56 Carlo Caione  wrote:

> On Thu, Feb 16, 2017 at 4:54 PM, Vladimir 'phcoder' Serbinenko
>  wrote:
> >
> >
> > On Thu, Feb 16, 2017, 14:14 Carlo Caione  wrote:
> >>
> >> On Thu, Feb 16, 2017 at 2:09 PM, Vladimir 'phcoder' Serbinenko
> >>  wrote:
> >> >
> >> >
> >> > On Thu, Feb 16, 2017, 14:04 Carlo Caione  wrote:
> >> >>
> >> >> On Thu, Feb 16, 2017 at 12:37 PM, Carlo Caione 
> >> >> wrote:
> >> >> > On Thu, Feb 16, 2017 at 12:32 PM, Thomas Schmitt <
> scdbac...@gmx.net>
> >> >> > wrote:
> >> >> >> Hi,
> >> >> >>
> >> >> >> Carlo Caione wrote:
> >> >> >>> > I think that the problem
> >> >> >>> > here is that endless.squash has been stored in two extents in
> the
> >> >> >>> > ISO9660 and GRUB doesn't deal fine with that (also according to
> >> >> >>> > this
> >> >> >>> > comment
> >> >> >>> >
> >> >> >>> >
> >> >> >>> >
> http://git.savannah.gnu.org/cgit/grub.git/tree/grub-core/fs/iso9660.c#n960
> >> >> >>
> >> >> >> Andrei Borzenkov wrote:
> >> >> >>> This comment could be stale and misleading.
> >> >> >>
> >> >> >> The code of read_node() in iso9660.c looks ready for multi-extent
> >> >> >> (aka
> >> >> >> ISO
> >> >> >> level 3).
> >> >> >>
> >> >> >> I have a very densely compressed test ISO for large file support
> >> >> >>http://scdbackup.webframe.org/large.iso.bz2
> >> >> >> which bunzip2 inflates from 4.5 KiB to 4+ GiB.
> >> >> >
> >> >> > Yeah, I made some more tests and apparently the problem is not in
> the
> >> >> > ISO9660 module but in the SQUASH4 module.
> >> >> > To be perfectly honest I have already a patch that solves my
> problem
> >> >> > but I'm having a hard time to understand WHY it solves the issue
> >> >> > since
> >> >> > I'm still not really familiar with squashfs (I casually found it
> >> >> > during my debugging).
> >> >> > Of course probably it is only masking a problem somewhere else.
> Still
> >> >> > digging.
> >> >> >
> >> >> > diff --git a/grub-core/fs/squash4.c b/grub-core/fs/squash4.c
> >> >> > index b97b344..4fef813 100644
> >> >> > --- a/grub-core/fs/squash4.c
> >> >> > +++ b/grub-core/fs/squash4.c
> >> >> > @@ -895,7 +895,7 @@ grub_squash_read_data (struct grub_squash_data
> >> >> > *data,
> >> >> >break;
> >> >> >  }
> >> >> >
> >> >> > -  if (fragment == 0x)
> >> >> > +  if (fragment == 0x || fragment == 0)
> >> >> >  return direct_read (data, ino, off, buf, len);
> >> >> >
> >> >> >err = read_chunk (data, , sizeof (frag),
> >> >>
> >> >> Hey Vladimir,
> >> >> Any idea about this?
> >> >
> >> > How big is the file in question? What is the squash tools version?
> >>
> >> The squash image is 4564504576
> >
> > I mean the size of file you try to access
>
> In the squashfs image there is one single file of size 1147128
>
Then it's possible that your patch is correct. I don't know why the format
changes or what the 0 means in this field

>
> --
> Carlo Caione  |  +39.340.80.30.096  |  Endless
>
___
Bug-grub mailing list
Bug-grub@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-grub

Re: Problem with ISO9660 and files stored on multiple extents

2017-02-16 Thread Andrei Borzenkov 
16.02.2017 19:04, Vladimir 'phcoder' Serbinenko пишет:
> On Thu, Feb 16, 2017, 16:56 Carlo Caione  wrote:
> 
>> On Thu, Feb 16, 2017 at 4:54 PM, Vladimir 'phcoder' Serbinenko
>>  wrote:
>>>
>>>
>>> On Thu, Feb 16, 2017, 14:14 Carlo Caione  wrote:

 On Thu, Feb 16, 2017 at 2:09 PM, Vladimir 'phcoder' Serbinenko
  wrote:
>
>
> On Thu, Feb 16, 2017, 14:04 Carlo Caione  wrote:
>>
>> On Thu, Feb 16, 2017 at 12:37 PM, Carlo Caione 
>> wrote:
>>> On Thu, Feb 16, 2017 at 12:32 PM, Thomas Schmitt <
>> scdbac...@gmx.net>
>>> wrote:
 Hi,

 Carlo Caione wrote:
>> I think that the problem
>> here is that endless.squash has been stored in two extents in
>> the
>> ISO9660 and GRUB doesn't deal fine with that (also according to
>> this
>> comment
>>
>>
>>
>> http://git.savannah.gnu.org/cgit/grub.git/tree/grub-core/fs/iso9660.c#n960

 Andrei Borzenkov wrote:
> This comment could be stale and misleading.

 The code of read_node() in iso9660.c looks ready for multi-extent
 (aka
 ISO
 level 3).

 I have a very densely compressed test ISO for large file support
http://scdbackup.webframe.org/large.iso.bz2
 which bunzip2 inflates from 4.5 KiB to 4+ GiB.
>>>
>>> Yeah, I made some more tests and apparently the problem is not in
>> the
>>> ISO9660 module but in the SQUASH4 module.
>>> To be perfectly honest I have already a patch that solves my
>> problem
>>> but I'm having a hard time to understand WHY it solves the issue
>>> since
>>> I'm still not really familiar with squashfs (I casually found it
>>> during my debugging).
>>> Of course probably it is only masking a problem somewhere else.
>> Still
>>> digging.
>>>
>>> diff --git a/grub-core/fs/squash4.c b/grub-core/fs/squash4.c
>>> index b97b344..4fef813 100644
>>> --- a/grub-core/fs/squash4.c
>>> +++ b/grub-core/fs/squash4.c
>>> @@ -895,7 +895,7 @@ grub_squash_read_data (struct grub_squash_data
>>> *data,
>>>break;
>>>  }
>>>
>>> -  if (fragment == 0x)
>>> +  if (fragment == 0x || fragment == 0)
>>>  return direct_read (data, ino, off, buf, len);
>>>
>>>err = read_chunk (data, , sizeof (frag),
>>
>> Hey Vladimir,
>> Any idea about this?
>
> How big is the file in question? What is the squash tools version?

 The squash image is 4564504576
>>>
>>> I mean the size of file you try to access
>>
>> In the squashfs image there is one single file of size 1147128
>>
> Then it's possible that your patch is correct. I don't know why the format
> changes or what the 0 means in this field
> 

Looking at current squash tools git, 0 is valid fragment number. Any
chance we hit different file type here? Carlo, could you make your
squashfs image available?

>>
>> --
>> Carlo Caione  |  +39.340.80.30.096  |  Endless
>>
> 


___
Bug-grub mailing list
Bug-grub@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-grub

Re: Problem with ISO9660 and files stored on multiple extents

2017-02-16 Thread Vladimir 'phcoder' Serbinenko 
On Thu, Feb 16, 2017, 14:14 Carlo Caione  wrote:

> On Thu, Feb 16, 2017 at 2:09 PM, Vladimir 'phcoder' Serbinenko
>  wrote:
> >
> >
> > On Thu, Feb 16, 2017, 14:04 Carlo Caione  wrote:
> >>
> >> On Thu, Feb 16, 2017 at 12:37 PM, Carlo Caione 
> wrote:
> >> > On Thu, Feb 16, 2017 at 12:32 PM, Thomas Schmitt 
> >> > wrote:
> >> >> Hi,
> >> >>
> >> >> Carlo Caione wrote:
> >> >>> > I think that the problem
> >> >>> > here is that endless.squash has been stored in two extents in the
> >> >>> > ISO9660 and GRUB doesn't deal fine with that (also according to
> this
> >> >>> > comment
> >> >>> >
> >> >>> >
> http://git.savannah.gnu.org/cgit/grub.git/tree/grub-core/fs/iso9660.c#n960
> >> >>
> >> >> Andrei Borzenkov wrote:
> >> >>> This comment could be stale and misleading.
> >> >>
> >> >> The code of read_node() in iso9660.c looks ready for multi-extent
> (aka
> >> >> ISO
> >> >> level 3).
> >> >>
> >> >> I have a very densely compressed test ISO for large file support
> >> >>http://scdbackup.webframe.org/large.iso.bz2
> >> >> which bunzip2 inflates from 4.5 KiB to 4+ GiB.
> >> >
> >> > Yeah, I made some more tests and apparently the problem is not in the
> >> > ISO9660 module but in the SQUASH4 module.
> >> > To be perfectly honest I have already a patch that solves my problem
> >> > but I'm having a hard time to understand WHY it solves the issue since
> >> > I'm still not really familiar with squashfs (I casually found it
> >> > during my debugging).
> >> > Of course probably it is only masking a problem somewhere else. Still
> >> > digging.
> >> >
> >> > diff --git a/grub-core/fs/squash4.c b/grub-core/fs/squash4.c
> >> > index b97b344..4fef813 100644
> >> > --- a/grub-core/fs/squash4.c
> >> > +++ b/grub-core/fs/squash4.c
> >> > @@ -895,7 +895,7 @@ grub_squash_read_data (struct grub_squash_data
> >> > *data,
> >> >break;
> >> >  }
> >> >
> >> > -  if (fragment == 0x)
> >> > +  if (fragment == 0x || fragment == 0)
> >> >  return direct_read (data, ino, off, buf, len);
> >> >
> >> >err = read_chunk (data, , sizeof (frag),
> >>
> >> Hey Vladimir,
> >> Any idea about this?
> >
> > How big is the file in question? What is the squash tools version?
>
> The squash image is 4564504576
>
I mean the size of file you try to access

>
> mksquashfs version 4.3-git (2014/06/09)
>
> No options are passed to mksquashfs when creating the image.
>
> --
> Carlo Caione  |  +39.340.80.30.096  |  Endless
>
___
Bug-grub mailing list
Bug-grub@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-grub

Re: Problem with ISO9660 and files stored on multiple extents

2017-02-16 Thread Carlo Caione 
On Thu, Feb 16, 2017 at 4:54 PM, Vladimir 'phcoder' Serbinenko
 wrote:
>
>
> On Thu, Feb 16, 2017, 14:14 Carlo Caione  wrote:
>>
>> On Thu, Feb 16, 2017 at 2:09 PM, Vladimir 'phcoder' Serbinenko
>>  wrote:
>> >
>> >
>> > On Thu, Feb 16, 2017, 14:04 Carlo Caione  wrote:
>> >>
>> >> On Thu, Feb 16, 2017 at 12:37 PM, Carlo Caione 
>> >> wrote:
>> >> > On Thu, Feb 16, 2017 at 12:32 PM, Thomas Schmitt 
>> >> > wrote:
>> >> >> Hi,
>> >> >>
>> >> >> Carlo Caione wrote:
>> >> >>> > I think that the problem
>> >> >>> > here is that endless.squash has been stored in two extents in the
>> >> >>> > ISO9660 and GRUB doesn't deal fine with that (also according to
>> >> >>> > this
>> >> >>> > comment
>> >> >>> >
>> >> >>> >
>> >> >>> > http://git.savannah.gnu.org/cgit/grub.git/tree/grub-core/fs/iso9660.c#n960
>> >> >>
>> >> >> Andrei Borzenkov wrote:
>> >> >>> This comment could be stale and misleading.
>> >> >>
>> >> >> The code of read_node() in iso9660.c looks ready for multi-extent
>> >> >> (aka
>> >> >> ISO
>> >> >> level 3).
>> >> >>
>> >> >> I have a very densely compressed test ISO for large file support
>> >> >>http://scdbackup.webframe.org/large.iso.bz2
>> >> >> which bunzip2 inflates from 4.5 KiB to 4+ GiB.
>> >> >
>> >> > Yeah, I made some more tests and apparently the problem is not in the
>> >> > ISO9660 module but in the SQUASH4 module.
>> >> > To be perfectly honest I have already a patch that solves my problem
>> >> > but I'm having a hard time to understand WHY it solves the issue
>> >> > since
>> >> > I'm still not really familiar with squashfs (I casually found it
>> >> > during my debugging).
>> >> > Of course probably it is only masking a problem somewhere else. Still
>> >> > digging.
>> >> >
>> >> > diff --git a/grub-core/fs/squash4.c b/grub-core/fs/squash4.c
>> >> > index b97b344..4fef813 100644
>> >> > --- a/grub-core/fs/squash4.c
>> >> > +++ b/grub-core/fs/squash4.c
>> >> > @@ -895,7 +895,7 @@ grub_squash_read_data (struct grub_squash_data
>> >> > *data,
>> >> >break;
>> >> >  }
>> >> >
>> >> > -  if (fragment == 0x)
>> >> > +  if (fragment == 0x || fragment == 0)
>> >> >  return direct_read (data, ino, off, buf, len);
>> >> >
>> >> >err = read_chunk (data, , sizeof (frag),
>> >>
>> >> Hey Vladimir,
>> >> Any idea about this?
>> >
>> > How big is the file in question? What is the squash tools version?
>>
>> The squash image is 4564504576
>
> I mean the size of file you try to access

In the squashfs image there is one single file of size 1147128

-- 
Carlo Caione  |  +39.340.80.30.096  |  Endless

___
Bug-grub mailing list
Bug-grub@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-grub

Re: Problem with ISO9660 and files stored on multiple extents

2017-02-16 Thread Carlo Caione 
On Thu, Feb 16, 2017 at 2:09 PM, Vladimir 'phcoder' Serbinenko
 wrote:
>
>
> On Thu, Feb 16, 2017, 14:04 Carlo Caione  wrote:
>>
>> On Thu, Feb 16, 2017 at 12:37 PM, Carlo Caione  wrote:
>> > On Thu, Feb 16, 2017 at 12:32 PM, Thomas Schmitt 
>> > wrote:
>> >> Hi,
>> >>
>> >> Carlo Caione wrote:
>> >>> > I think that the problem
>> >>> > here is that endless.squash has been stored in two extents in the
>> >>> > ISO9660 and GRUB doesn't deal fine with that (also according to this
>> >>> > comment
>> >>> >
>> >>> > http://git.savannah.gnu.org/cgit/grub.git/tree/grub-core/fs/iso9660.c#n960
>> >>
>> >> Andrei Borzenkov wrote:
>> >>> This comment could be stale and misleading.
>> >>
>> >> The code of read_node() in iso9660.c looks ready for multi-extent (aka
>> >> ISO
>> >> level 3).
>> >>
>> >> I have a very densely compressed test ISO for large file support
>> >>http://scdbackup.webframe.org/large.iso.bz2
>> >> which bunzip2 inflates from 4.5 KiB to 4+ GiB.
>> >
>> > Yeah, I made some more tests and apparently the problem is not in the
>> > ISO9660 module but in the SQUASH4 module.
>> > To be perfectly honest I have already a patch that solves my problem
>> > but I'm having a hard time to understand WHY it solves the issue since
>> > I'm still not really familiar with squashfs (I casually found it
>> > during my debugging).
>> > Of course probably it is only masking a problem somewhere else. Still
>> > digging.
>> >
>> > diff --git a/grub-core/fs/squash4.c b/grub-core/fs/squash4.c
>> > index b97b344..4fef813 100644
>> > --- a/grub-core/fs/squash4.c
>> > +++ b/grub-core/fs/squash4.c
>> > @@ -895,7 +895,7 @@ grub_squash_read_data (struct grub_squash_data
>> > *data,
>> >break;
>> >  }
>> >
>> > -  if (fragment == 0x)
>> > +  if (fragment == 0x || fragment == 0)
>> >  return direct_read (data, ino, off, buf, len);
>> >
>> >err = read_chunk (data, , sizeof (frag),
>>
>> Hey Vladimir,
>> Any idea about this?
>
> How big is the file in question? What is the squash tools version?

The squash image is 4564504576

mksquashfs version 4.3-git (2014/06/09)

No options are passed to mksquashfs when creating the image.

-- 
Carlo Caione  |  +39.340.80.30.096  |  Endless

___
Bug-grub mailing list
Bug-grub@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-grub

Re: Problem with ISO9660 and files stored on multiple extents

2017-02-16 Thread Vladimir 'phcoder' Serbinenko 
On Thu, Feb 16, 2017, 14:04 Carlo Caione  wrote:

> On Thu, Feb 16, 2017 at 12:37 PM, Carlo Caione  wrote:
> > On Thu, Feb 16, 2017 at 12:32 PM, Thomas Schmitt 
> wrote:
> >> Hi,
> >>
> >> Carlo Caione wrote:
> >>> > I think that the problem
> >>> > here is that endless.squash has been stored in two extents in the
> >>> > ISO9660 and GRUB doesn't deal fine with that (also according to this
> >>> > comment
> >>> >
> http://git.savannah.gnu.org/cgit/grub.git/tree/grub-core/fs/iso9660.c#n960
> >>
> >> Andrei Borzenkov wrote:
> >>> This comment could be stale and misleading.
> >>
> >> The code of read_node() in iso9660.c looks ready for multi-extent (aka
> ISO
> >> level 3).
> >>
> >> I have a very densely compressed test ISO for large file support
> >>http://scdbackup.webframe.org/large.iso.bz2
> >> which bunzip2 inflates from 4.5 KiB to 4+ GiB.
> >
> > Yeah, I made some more tests and apparently the problem is not in the
> > ISO9660 module but in the SQUASH4 module.
> > To be perfectly honest I have already a patch that solves my problem
> > but I'm having a hard time to understand WHY it solves the issue since
> > I'm still not really familiar with squashfs (I casually found it
> > during my debugging).
> > Of course probably it is only masking a problem somewhere else. Still
> digging.
> >
> > diff --git a/grub-core/fs/squash4.c b/grub-core/fs/squash4.c
> > index b97b344..4fef813 100644
> > --- a/grub-core/fs/squash4.c
> > +++ b/grub-core/fs/squash4.c
> > @@ -895,7 +895,7 @@ grub_squash_read_data (struct grub_squash_data *data,
> >break;
> >  }
> >
> > -  if (fragment == 0x)
> > +  if (fragment == 0x || fragment == 0)
> >  return direct_read (data, ino, off, buf, len);
> >
> >err = read_chunk (data, , sizeof (frag),
>
> Hey Vladimir,
> Any idea about this?
>
How big is the file in question? What is the squash tools version?

>
>
> --
> Carlo Caione  |  +39.340.80.30.096  |  Endless
>
___
Bug-grub mailing list
Bug-grub@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-grub

Re: Problem with ISO9660 and files stored on multiple extents

2017-02-16 Thread Carlo Caione 
On Thu, Feb 16, 2017 at 12:37 PM, Carlo Caione  wrote:
> On Thu, Feb 16, 2017 at 12:32 PM, Thomas Schmitt  wrote:
>> Hi,
>>
>> Carlo Caione wrote:
>>> > I think that the problem
>>> > here is that endless.squash has been stored in two extents in the
>>> > ISO9660 and GRUB doesn't deal fine with that (also according to this
>>> > comment
>>> > http://git.savannah.gnu.org/cgit/grub.git/tree/grub-core/fs/iso9660.c#n960
>>
>> Andrei Borzenkov wrote:
>>> This comment could be stale and misleading.
>>
>> The code of read_node() in iso9660.c looks ready for multi-extent (aka ISO
>> level 3).
>>
>> I have a very densely compressed test ISO for large file support
>>http://scdbackup.webframe.org/large.iso.bz2
>> which bunzip2 inflates from 4.5 KiB to 4+ GiB.
>
> Yeah, I made some more tests and apparently the problem is not in the
> ISO9660 module but in the SQUASH4 module.
> To be perfectly honest I have already a patch that solves my problem
> but I'm having a hard time to understand WHY it solves the issue since
> I'm still not really familiar with squashfs (I casually found it
> during my debugging).
> Of course probably it is only masking a problem somewhere else. Still digging.
>
> diff --git a/grub-core/fs/squash4.c b/grub-core/fs/squash4.c
> index b97b344..4fef813 100644
> --- a/grub-core/fs/squash4.c
> +++ b/grub-core/fs/squash4.c
> @@ -895,7 +895,7 @@ grub_squash_read_data (struct grub_squash_data *data,
>break;
>  }
>
> -  if (fragment == 0x)
> +  if (fragment == 0x || fragment == 0)
>  return direct_read (data, ino, off, buf, len);
>
>err = read_chunk (data, , sizeof (frag),

Hey Vladimir,
Any idea about this?


-- 
Carlo Caione  |  +39.340.80.30.096  |  Endless

___
Bug-grub mailing list
Bug-grub@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-grub

Re: Problem with ISO9660 and files stored on multiple extents

2017-02-16 Thread Carlo Caione 
On Thu, Feb 16, 2017 at 12:32 PM, Thomas Schmitt  wrote:
> Hi,
>
> Carlo Caione wrote:
>> > I think that the problem
>> > here is that endless.squash has been stored in two extents in the
>> > ISO9660 and GRUB doesn't deal fine with that (also according to this
>> > comment
>> > http://git.savannah.gnu.org/cgit/grub.git/tree/grub-core/fs/iso9660.c#n960
>
> Andrei Borzenkov wrote:
>> This comment could be stale and misleading.
>
> The code of read_node() in iso9660.c looks ready for multi-extent (aka ISO
> level 3).
>
> I have a very densely compressed test ISO for large file support
>http://scdbackup.webframe.org/large.iso.bz2
> which bunzip2 inflates from 4.5 KiB to 4+ GiB.

Yeah, I made some more tests and apparently the problem is not in the
ISO9660 module but in the SQUASH4 module.
To be perfectly honest I have already a patch that solves my problem
but I'm having a hard time to understand WHY it solves the issue since
I'm still not really familiar with squashfs (I casually found it
during my debugging).
Of course probably it is only masking a problem somewhere else. Still digging.

diff --git a/grub-core/fs/squash4.c b/grub-core/fs/squash4.c
index b97b344..4fef813 100644
--- a/grub-core/fs/squash4.c
+++ b/grub-core/fs/squash4.c
@@ -895,7 +895,7 @@ grub_squash_read_data (struct grub_squash_data *data,
   break;
 }

-  if (fragment == 0x)
+  if (fragment == 0x || fragment == 0)
 return direct_read (data, ino, off, buf, len);

   err = read_chunk (data, , sizeof (frag),

-- 
Carlo Caione  |  +39.340.80.30.096  |  Endless

___
Bug-grub mailing list
Bug-grub@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-grub

Re: Problem with ISO9660 and files stored on multiple extents

2017-02-16 Thread Thomas Schmitt 
Hi,

Carlo Caione wrote:
> > I think that the problem
> > here is that endless.squash has been stored in two extents in the
> > ISO9660 and GRUB doesn't deal fine with that (also according to this
> > comment
> > http://git.savannah.gnu.org/cgit/grub.git/tree/grub-core/fs/iso9660.c#n960

Andrei Borzenkov wrote:
> This comment could be stale and misleading.

The code of read_node() in iso9660.c looks ready for multi-extent (aka ISO
level 3).

I have a very densely compressed test ISO for large file support
   http://scdbackup.webframe.org/large.iso.bz2
which bunzip2 inflates from 4.5 KiB to 4+ GiB.

grub-fstest 2.02~beta2-36 of my Debian Sid VM works fine with this ISO.
The comment in line 960 of iso9660.c would be outdated therefore.

The ISO contains a data file
  -rw-r--r--1 1000 1000 4329375744 May  6  2014 '/my/large_file'
which consists of two extents
  $ xorriso -indev large.iso -find /my/large_file -exec report_lba
  ...
  Report layout: xt , Startlba ,   Blocks , Filesize , ISO image path
  File data lba:  0 ,   56 ,  2097151 , 4329375744 , '/my/large_file'
  File data lba:  1 ,  2097207 ,16802 , 4329375744 , '/my/large_file'

The file is all zeros. For the given question, i write a few recognizable
bytes into its extents:
  echo Extent1Start | dd conv=notrunc bs=2048 seek=56 count=1 of=large.iso
  echo Extent2Start | dd conv=notrunc bs=2048 seek=2097207 count=1 of=large.iso
  echo RandomTargetInExtent2 | dd conv=notrunc bs=2048 seek=2113916 count=1 
of=large.iso

(2113916 - 56 ) * 2048 = 4329185280 will be used as random access address in
one of my tests. 


Now for grub-fstest. Reading all bytes:

  $ grub-fstest large.iso cat /my/large_file | wc
3   3 4329375744

  $ grub-fstest large.iso cat /my/large_file | od -c
  000   E   x   t   e   n   t   1   S   t   a   r   t  \n  \0  \0  \0
  020  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
  *
  3774000   E   x   t   e   n   t   2   S   t   a   r   t  \n  \0  \0  \0
  3774020  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
  *
  4020242   R   a   n   d   o   m   T   a   r   g   e   t   I   n   E   x
  40202420020   t   e   n   t   2  \n  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
  40202420040  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
  *
  40203204000

(Octal 3774000 is 2048 bytes below 4 GiB. That's the highest block
aligned count which can be expressed in a single extent.)

Random access into the second extent:

  $ grub-fstest --skip=4329185280 --length=16384 large.iso cat /my/large_file | 
wc
1   1   16384

  $ grub-fstest --skip=4329185280 --length=16384 large.iso cat /my/large_file | 
od -c
  000   R   a   n   d   o   m   T   a   r   g   e   t   I   n   E   x
  020   t   e   n   t   2  \n  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
  040  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
  *
  004


Have a nice day :)

Thomas
 

___
Bug-grub mailing list
Bug-grub@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-grub

Re: Problem with ISO9660 and files stored on multiple extents

2017-02-15 Thread Andrei Borzenkov 
16.02.2017 00:30, Carlo Caione пишет:
> Hi,
> in our ISO images in the ISO9660 partition we ship a squashfs image
> containing the full OS image.
> At boot GRUB does something like this:
> 
> loopback loop_squash ($eosimage)/endless/endless.squash
> loopback loop_img (loop_squash)/endless.img
> set root=(loop_img,gpt3)
> 
> This is working fine for our small images without any problem.
> 
> The problem arises when the size of endless.squash is >4GB. In this
> case accessing (loop_img) just fails with:
> 
> No known filesystem detected
> 
> I spent a bit tracking down this issue and I think that the problem
> here is that endless.squash has been stored in two extents in the
> ISO9660 and GRUB doesn't deal fine with that (also according to this
> comment 
> http://git.savannah.gnu.org/cgit/grub.git/tree/grub-core/fs/iso9660.c#n960).
> 

This comment could be stale and misleading.

> Could be this the cause of the problem? Any suggestion besides
> extending the iso9660 module with the support for files stored on
> multiple extents?
> 

Quick look suggests that it does process multi-extents

while (dirent.flags & FLAG_MORE_EXTENTS)
  {

in grub_iso9660_iterate_dir()

I suggest you try to use gdb on grub-fstest binary that allows stepping
through the code to see where it misbehaves.

> This is briefly how I tracked down the issue. When squash_mount() is
> called it tried to access the squashfs image to retrieve the number of
> the fragments, this is done basically reading at sb.unk1offset in the
> squashfs image. This number for images >4GB is clearly wrong. Being
> the image mounted in loopback what happens is that grub is trying to
> read the squashfs file on the ISO9660 partition at file->offset >4GB
> in grub_iso9660_read(). Inspecting the ISO with isoinfo we have:
> 
> --   000  4294965248 Feb 15 2017 [  13356 
> ENDLESS.SQUASH;1
> --   000   269539328 Feb 15 2017 [2110507 00]
> ENDLESS.SQUASH;1
> 
> That is in grub_iso9660_read() we are trying to access the second
> extent, reading rubbish.
> 
> Is this theory correct?
> 
> Thank you,
> 
> 


___
Bug-grub mailing list
Bug-grub@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-grub

Problem with ISO9660 and files stored on multiple extents

2017-02-15 Thread Carlo Caione 
Hi,
in our ISO images in the ISO9660 partition we ship a squashfs image
containing the full OS image.
At boot GRUB does something like this:

loopback loop_squash ($eosimage)/endless/endless.squash
loopback loop_img (loop_squash)/endless.img
set root=(loop_img,gpt3)

This is working fine for our small images without any problem.

The problem arises when the size of endless.squash is >4GB. In this
case accessing (loop_img) just fails with:

No known filesystem detected

I spent a bit tracking down this issue and I think that the problem
here is that endless.squash has been stored in two extents in the
ISO9660 and GRUB doesn't deal fine with that (also according to this
comment 
http://git.savannah.gnu.org/cgit/grub.git/tree/grub-core/fs/iso9660.c#n960).

Could be this the cause of the problem? Any suggestion besides
extending the iso9660 module with the support for files stored on
multiple extents?

This is briefly how I tracked down the issue. When squash_mount() is
called it tried to access the squashfs image to retrieve the number of
the fragments, this is done basically reading at sb.unk1offset in the
squashfs image. This number for images >4GB is clearly wrong. Being
the image mounted in loopback what happens is that grub is trying to
read the squashfs file on the ISO9660 partition at file->offset >4GB
in grub_iso9660_read(). Inspecting the ISO with isoinfo we have:

--   000  4294965248 Feb 15 2017 [  13356 
ENDLESS.SQUASH;1
--   000   269539328 Feb 15 2017 [2110507 00]
ENDLESS.SQUASH;1

That is in grub_iso9660_read() we are trying to access the second
extent, reading rubbish.

Is this theory correct?

Thank you,


-- 
Carlo Caione  |  +39.340.80.30.096  |  Endless

___
Bug-grub mailing list
Bug-grub@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-grub