Re: mkimg used to create gpt image, problem booting

[Andrey V. Elsukov]

On 24.08.2014 19:23, Marcel Moolenaar wrote:
 
 On Aug 24, 2014, at 2:11 AM, Andrey V. Elsukov bu7c...@yandex.ru wrote:
 
 On 24.08.2014 06:14, Craig Rodrigues wrote:
 
 I did some further debugging inside the loader by doing the following.
  - I added CFLAGS += -DPART_DEBUG to sys/boot/common/Makefile.inc
  - I added DEBUG() statements all over sys/boot/common/part.c

 I observed that in sys/boot/common/part.c in the ptbl_gptread() function,
 that in this section:

305 ent = (struct gpt_ent *)tbl;
306 size = MIN(hdr.hdr_entries * hdr.hdr_entsz,
307 MAXTBLSZ * table-sectorsize);
308 for (i = 0; i  size / hdr.hdr_entsz; i++, ent++) {
309 if (uuid_equal(ent-ent_type, gpt_uuid_unused, 
 NULL))
310 continue;

 ent-ent_type is all 0's, which matches gpt_uuid_unused, so it bails
 out of the loop and never adds the gpt partitions to the list of partitions
 that the loader can access.

 Yes, the problem is in the ptable_gptread() function. I'll commit the fix.

 
 Actually, no. There is *a* problem in that function:
 The function does not respect hdr.hdr_entsz when it
 needs the next entry. It simply uses ent++, which
 is fixed our definition of struct gpt_ent and may
 not match the definition of the writer.

Yes, you are right. I'll fix this. Thanks.

 I don't see how the loader is responsible for *the*
 problem. All I see in qemu is that the loader, when
 it reads a sector, isn't getting the actual sector
 data that's in the image.
 
 Just do a ktrace on qemu and you'll see what I mean.
 YMMV of course,

Also there is bootparttest utility in the tools/tools/bootparttest. I
think it can be useful for debugging.

-- 
WBR, Andrey V. Elsukov
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: mkimg used to create gpt image, problem booting

[Andrey V. Elsukov]

On 24.08.2014 20:59, Craig Rodrigues wrote:
 Also, in the gptboot man page, it mentions that gptboot can only boot
 on systems with 128 partitions or less.  This seems like an artificial
 restriction.
 Does the gptboot code really enforce this?  Not that I have a system with more
 than 128 partitions. :)

It's because gptboot uses static buffer to read and write GPT table.

-- 
WBR, Andrey V. Elsukov
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: mkimg used to create gpt image, problem booting

[Andrey V. Elsukov]

On 25.08.2014 03:55, Marcel Moolenaar wrote:
 Though, w/ people dd'ing images onto disks, and using growfs to grow
 as necessary, we might want to allocate a few more more than the
 minimum...  I do agree that we want to keep sizes to a minimum...
 
 One thing I can maybe do is simply fill the empty sectors
 that are there because of alignment. If you add -P 4K to
 mkimg, then the first partition will by 4K aligned and you
 have about 5 sectors unused between the end of the GPT
 table and the first sector of the first partition. I may
 as well extend the table to cover those unued sectors.

IMHO, mkimg should behave like gpart and create images in
gpart-compatible way. Some users may want to copy the partition layout
from the image to real hardware and they will not be able to do it.

Also, now FreeBSD 11.0 uses different first usable LBA. By default it is
4k aligned. And this creates some incompatibility with older versions.
You can't do `gpart restore` and get the same table, as you had on older
system.

 However, this is a pretty side-ways way to end up with a
 GPT table that has some extra room. Maybe having scheme-
 specific options for this kind of thing is not bad. At
 least EBR and APM have the same problem and the BSD
 disk label can also be created with more than just 8
 partitions.

I thought about implementing `gpart modify` or `gpart set` -n entries to
change number of entries when it is possible (i.e. disklabel(8) can do
it, but gpart doesn't).
Also in r228076 I changed APM code to calculate maximum number of
entries depending from available free space.

-- 
WBR, Andrey V. Elsukov
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: mkimg used to create gpt image, problem booting

[Marcel Moolenaar]

On Aug 25, 2014, at 12:48 AM, Andrey V. Elsukov bu7c...@yandex.ru wrote:

 On 25.08.2014 03:55, Marcel Moolenaar wrote:
 Though, w/ people dd'ing images onto disks, and using growfs to grow
 as necessary, we might want to allocate a few more more than the
 minimum...  I do agree that we want to keep sizes to a minimum...
 
 One thing I can maybe do is simply fill the empty sectors
 that are there because of alignment. If you add -P 4K to
 mkimg, then the first partition will by 4K aligned and you
 have about 5 sectors unused between the end of the GPT
 table and the first sector of the first partition. I may
 as well extend the table to cover those unued sectors.
 
 IMHO, mkimg should behave like gpart and create images in
 gpart-compatible way.

It already does. There's s difference between behaving
like something else or behaving exactly identical to
that something. The same applies to compatible. It is
not the same as identical.

There is no compatibility issue. mkimg follows the GPT
specification (modulo bugs) and gpart happily groks the
partition table.

 Some users may want to copy the partition layout
 from the image to real hardware and they will not be able to do it.

I'm inclined to say that generally speaking this is not
possible. The GPT has metadata in the first few sectors
*and* the last few sectors and LBAs of these sectors are
part of the metadata. You cannot blindly copy an image
onto a physical medium unless the image and the physical
medium are of exactly the same size. Odds are they are
not.

To reliably transfer or convert an image (e.g. RAW-VHD)
one must modify the image as part of the process. Not a
hard rule, but best to assume as a rule of thumb. This
seems to warrant a utility all by itself.

 Also, now FreeBSD 11.0 uses different first usable LBA. By default it is
 4k aligned. And this creates some incompatibility with older versions.
 You can't do `gpart restore` and get the same table, as you had on older
 system.

It sounds restore is broken then. The restore command
cannot ever assume anything about the GPT. Including
the tool that created the GPT. In order to restore a
GPT, it must be properly backed-up. The backup header
and table should suffice most of the time for that
purpose as it's a replica, but as soon as meta-data is
missing and the restore command has to guess, things
will go wrong.

-- 
Marcel Moolenaar
mar...@xcllnt.net




signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: mkimg used to create gpt image, problem booting

[Andrey V. Elsukov]

On 25.08.2014 18:40, Marcel Moolenaar wrote:
 Also, now FreeBSD 11.0 uses different first usable LBA. By default it is
 4k aligned. And this creates some incompatibility with older versions.
 You can't do `gpart restore` and get the same table, as you had on older
 system.
 
 It sounds restore is broken then. The restore command
 cannot ever assume anything about the GPT. Including
 the tool that created the GPT. In order to restore a
 GPT, it must be properly backed-up. The backup header
 and table should suffice most of the time for that
 purpose as it's a replica, but as soon as meta-data is
 missing and the restore command has to guess, things
 will go wrong.

`gpart restore` just uses a number of commands to geom_part(4) to create
partition table similar to what was backed up. If your partition table
on the old system had a partition that starts from LBA 34, now `gpart
create` isn't able to create such partition table. Because by default
the first usable LBA is 40.

-- 
WBR, Andrey V. Elsukov
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: mkimg used to create gpt image, problem booting

[John-Mark Gurney]

Andrey V. Elsukov wrote this message on Mon, Aug 25, 2014 at 19:02 +0400:
 On 25.08.2014 18:40, Marcel Moolenaar wrote:
  Also, now FreeBSD 11.0 uses different first usable LBA. By default it is
  4k aligned. And this creates some incompatibility with older versions.
  You can't do `gpart restore` and get the same table, as you had on older
  system.
  
  It sounds restore is broken then. The restore command
  cannot ever assume anything about the GPT. Including
  the tool that created the GPT. In order to restore a
  GPT, it must be properly backed-up. The backup header
  and table should suffice most of the time for that
  purpose as it's a replica, but as soon as meta-data is
  missing and the restore command has to guess, things
  will go wrong.
 
 `gpart restore` just uses a number of commands to geom_part(4) to create
 partition table similar to what was backed up. If your partition table
 on the old system had a partition that starts from LBA 34, now `gpart
 create` isn't able to create such partition table. Because by default
 the first usable LBA is 40.

Luckily, gpart restore won't work:
# gpart backup /dev/md0
GPT 4
1 freebsd-ufs  8 262144  
# gpart restore md1  /tmp/foob.gpt.back 
gpart: entries '4': Invalid argument

So, we're somewhat safe, guess gpart restore needs to learn how to
handle entries properly

We should fix this, since other OS's might not use 128 for entries..

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 All that I will do, has been done, All that I have, has not.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: mkimg used to create gpt image, problem booting

[Andrey V. Elsukov]

On 24.08.2014 06:14, Craig Rodrigues wrote:
 Hi,
 
 I did some more experiments, and found that after /boot/loader runs,
 if I break into the loader prompt and type lsdev, I would get this:
 
 (1)  GPT Disk image which boots under QEMU, made by bsdinstall
 ==
 View from loader
 
 OK lsdev
 cd devices:
 disk devices:
   disk0:BIOS drive A:
   disk1:BIOS drive C:
  disk1p1: FreeBSD boot
  disk1p2: FreeBSD UFS
  disk1p3: FreeBSD swap
 pxe devices:
 
 
 View from gpart, after we mdconfig the disk image
 
 =  34  10485693  md0  GPT  (5.0G)
 34   1281  freebsd-boot  (64K)
162   99592962  freebsd-ufs  (4.7G)
99594585242883  freebsd-swap  (256M)
   10483746  1981   - free -  (991K)
 
 
 (2)  GPT Disk image which fails to boot under QEMU, made by mkimg
 ===
 View from loader
 
 OK lsdev
 cd devices:
 disk devices:
   disk0:BIOS drive A:
   disk1:BIOS drive C:
 pxe devices:
 
 View from gpart, after we mdconfig the disk image
 
 
 =  3  1784944  md1  GPT  (872M)
 3   321  freebsd-boot  (16K)
35  17849122  freebsd-ufs  (872M)
 
 
 
 This leads me to believe that there is logic in /boot/loader,
 which is not in GEOM, that fails to parse the GPT produced by mkimg.
 
 
 I did some further debugging inside the loader by doing the following.
   - I added CFLAGS += -DPART_DEBUG to sys/boot/common/Makefile.inc
   - I added DEBUG() statements all over sys/boot/common/part.c
 
 I observed that in sys/boot/common/part.c in the ptbl_gptread() function,
 that in this section:
 
 305 ent = (struct gpt_ent *)tbl;
 306 size = MIN(hdr.hdr_entries * hdr.hdr_entsz,
 307 MAXTBLSZ * table-sectorsize);
 308 for (i = 0; i  size / hdr.hdr_entsz; i++, ent++) {
 309 if (uuid_equal(ent-ent_type, gpt_uuid_unused, 
 NULL))
 310 continue;
 
 ent-ent_type is all 0's, which matches gpt_uuid_unused, so it bails
 out of the loop and never adds the gpt partitions to the list of partitions
 that the loader can access.
 
 I'm not familiar with the GPT format, nor am I familiar with the
 gpt code inside the loader, and how it differs from GEOM.
 
 Do you have any further ideas of where to hunt for the root cause of
 the problem?

Yes, the problem is in the ptable_gptread() function. I'll commit the fix.

-- 
WBR, Andrey V. Elsukov



signature.asc
Description: OpenPGP digital signature

Re: mkimg used to create gpt image, problem booting

[Marcel Moolenaar]

On Aug 24, 2014, at 2:11 AM, Andrey V. Elsukov bu7c...@yandex.ru wrote:

 On 24.08.2014 06:14, Craig Rodrigues wrote:

 I did some further debugging inside the loader by doing the following.
  - I added CFLAGS += -DPART_DEBUG to sys/boot/common/Makefile.inc
  - I added DEBUG() statements all over sys/boot/common/part.c
 
 I observed that in sys/boot/common/part.c in the ptbl_gptread() function,
 that in this section:
 
305 ent = (struct gpt_ent *)tbl;
306 size = MIN(hdr.hdr_entries * hdr.hdr_entsz,
307 MAXTBLSZ * table-sectorsize);
308 for (i = 0; i  size / hdr.hdr_entsz; i++, ent++) {
309 if (uuid_equal(ent-ent_type, gpt_uuid_unused, 
 NULL))
310 continue;
 
 ent-ent_type is all 0's, which matches gpt_uuid_unused, so it bails
 out of the loop and never adds the gpt partitions to the list of partitions
 that the loader can access.
 
 Yes, the problem is in the ptable_gptread() function. I'll commit the fix.
 

Actually, no. There is *a* problem in that function:
The function does not respect hdr.hdr_entsz when it
needs the next entry. It simply uses ent++, which
is fixed our definition of struct gpt_ent and may
not match the definition of the writer.

I don't see how the loader is responsible for *the*
problem. All I see in qemu is that the loader, when
it reads a sector, isn't getting the actual sector
data that's in the image.

Just do a ktrace on qemu and you'll see what I mean.
YMMV of course,

-- 
Marcel Moolenaar
mar...@xcllnt.net




signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: mkimg used to create gpt image, problem booting

[Craig Rodrigues]

On Sun, Aug 24, 2014 at 2:11 AM, Andrey V. Elsukov bu7c...@yandex.ru wrote:

 Yes, the problem is in the ptable_gptread() function. I'll commit the fix.

Index: head/sys/boot/common/part.c
===
--- head/sys/boot/common/part.c (revision 270444)
+++ head/sys/boot/common/part.c (revision 270445)
@@ -254,8 +254,8 @@
table-sectorsize);
if (phdr != NULL) {
/* Read the primary GPT table. */
-   size = MIN(MAXTBLSZ,
-   phdr-hdr_entries * phdr-hdr_entsz / table-sectorsize);
+   size = MIN(MAXTBLSZ, (phdr-hdr_entries * phdr-hdr_entsz +
+   table-sectorsize - 1) / table-sectorsize);
if (dread(dev, tbl, size, phdr-hdr_lba_table) == 0 
gpt_checktbl(phdr, tbl, size * table-sectorsize,
table-sectors - 1) == 0) {

I can confirm that r270445 fixes the problem for me, where I can
now QEMU boot a GPT partitioned image created with mkimg.

I put some more debugging in the code, and found this:


(1)  GPT IMAGE CREATED WITH MKIMG
===

phdr-hdr_entries = 2, phdr-hdr_entsz = 128, table-sectorsize = 512

(2)  GPT IMAGE CREATED WITH BSDINSTALL
===

phdr-hdr_entries = 128, phdr-hdr_entsz = 128, table-sectorsize = 512


Does gpart  create a fixed partition table with 128 entries?
That would explain a lot.

Also, in the gptboot man page, it mentions that gptboot can only boot
on systems with 128 partitions or less.  This seems like an artificial
restriction.
Does the gptboot code really enforce this?  Not that I have a system with more
than 128 partitions. :)

Should mkimg be changed to create a partition table with 128 entries
by default, to match older versions of FreeBSD which do not have this fix?

Thanks for the fix!

--
Craig
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: mkimg used to create gpt image, problem booting

[Craig Rodrigues]

On Sun, Aug 24, 2014 at 8:23 AM, Marcel Moolenaar mar...@xcllnt.net wrote:

 On Aug 24, 2014, at 2:11 AM, Andrey V. Elsukov bu7c...@yandex.ru wrote:

 On 24.08.2014 06:14, Craig Rodrigues wrote:

 I did some further debugging inside the loader by doing the following.
  - I added CFLAGS += -DPART_DEBUG to sys/boot/common/Makefile.inc
  - I added DEBUG() statements all over sys/boot/common/part.c

 I observed that in sys/boot/common/part.c in the ptbl_gptread() function,
 that in this section:

305 ent = (struct gpt_ent *)tbl;
306 size = MIN(hdr.hdr_entries * hdr.hdr_entsz,
307 MAXTBLSZ * table-sectorsize);
308 for (i = 0; i  size / hdr.hdr_entsz; i++, ent++) {
309 if (uuid_equal(ent-ent_type, gpt_uuid_unused, 
 NULL))
310 continue;

 ent-ent_type is all 0's, which matches gpt_uuid_unused, so it bails
 out of the loop and never adds the gpt partitions to the list of partitions
 that the loader can access.

 Yes, the problem is in the ptable_gptread() function. I'll commit the fix.


 Actually, no. There is *a* problem in that function:
 The function does not respect hdr.hdr_entsz when it
 needs the next entry. It simply uses ent++, which
 is fixed our definition of struct gpt_ent and may
 not match the definition of the writer.

Yes, you are correct.  I looked at the GPT format here:

http://en.wikipedia.org/wiki/GUID_Partition_Table

Although the default size in the specification for a gpt entry is 128 bytes,
which matches the size of our struct gpt_entry,
technically, the gpt header could specify a gpt entry size that is something
other than 128 bytes.  I guess the only restriction seems to be that
you cannot have variable sized gpt entries.they have to match the
size of the gpt entry specified in the gpt header.

I guess we haven't hit this yet because there are probably very few
peopel creating gpt tables with entry sizes other than 128, but in the
future, who knows?

--
Craig
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: mkimg used to create gpt image, problem booting

[Marcel Moolenaar]

On Aug 24, 2014, at 9:59 AM, Craig Rodrigues rodr...@freebsd.org wrote:

 On Sun, Aug 24, 2014 at 2:11 AM, Andrey V. Elsukov bu7c...@yandex.ru wrote:
 
 Yes, the problem is in the ptable_gptread() function. I'll commit the fix.
 

 Should mkimg be changed to create a partition table with 128 entries
 by default, to match older versions of FreeBSD which do not have this fix?

Maybe. 128 is the suggested default. It's not a hard lower
limit. Technically speaking, it's perfectly fine to create
just enough entries to fill a single sector. Then again,
code makes all kinds of assumptions or has all kinds of
bugs -- just like the logic in the loader apparently.

By having mkimg create a large table, even if it's knows
up front that it doesn't have to may prevent broken code
from tripping over, bit it surely bloats the image for
no reason.

-- 
Marcel Moolenaar
mar...@xcllnt.net




signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: mkimg used to create gpt image, problem booting

[Craig Rodrigues]

On Sun, Aug 24, 2014 at 12:05 PM, Marcel Moolenaar mar...@xcllnt.net wrote:

 On Aug 24, 2014, at 9:59 AM, Craig Rodrigues rodr...@freebsd.org wrote:

 On Sun, Aug 24, 2014 at 2:11 AM, Andrey V. Elsukov bu7c...@yandex.ru wrote:

 Yes, the problem is in the ptable_gptread() function. I'll commit the fix.


 Should mkimg be changed to create a partition table with 128 entries
 by default, to match older versions of FreeBSD which do not have this fix?

 Maybe. 128 is the suggested default. It's not a hard lower
 limit. Technically speaking, it's perfectly fine to create
 just enough entries to fill a single sector. Then again,
 code makes all kinds of assumptions or has all kinds of
 bugs -- just like the logic in the loader apparently.

 By having mkimg create a large table, even if it's knows
 up front that it doesn't have to may prevent broken code
 from tripping over, bit it surely bloats the image for
 no reason.

I see what you are saying.  If you have a new device
and do gpart create -s GPT, then this value in sys/geom/part/g_part_gpt.c:

 .gps_minent = 128,

causes the logic in the g_part_ctl_create() function in sys/geom/part/g_part.c
to set the number of partitions to 128, and then it calls g_part_ctl_create()
which creates the partition table with 128 empty entries.

gpart create doesn't know how many partitions you want, so it
needs to allocate some space up front for the partition table,
and then you can do gpart add to add the partitions later.

With mkimg, you know exactly how many partitions you are creating
, so you don't need to specify 128 as the number of partitions.

Since only gpart was available for creating GPT partitions, the side-effect
of always having 128 partitions hid the bug in the loader.

Hopefully Andrey's fix can be backported to at least stable/9, because
the loader bug seems to have been there since at least 2012.

--
Craig
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: mkimg used to create gpt image, problem booting

[John-Mark Gurney]

Craig Rodrigues wrote this message on Sun, Aug 24, 2014 at 15:37 -0700:
 On Sun, Aug 24, 2014 at 12:05 PM, Marcel Moolenaar mar...@xcllnt.net wrote:
 
  On Aug 24, 2014, at 9:59 AM, Craig Rodrigues rodr...@freebsd.org wrote:
 
  On Sun, Aug 24, 2014 at 2:11 AM, Andrey V. Elsukov bu7c...@yandex.ru 
  wrote:
 
  Yes, the problem is in the ptable_gptread() function. I'll commit the fix.
 
 
  Should mkimg be changed to create a partition table with 128 entries
  by default, to match older versions of FreeBSD which do not have this fix?
 
  Maybe. 128 is the suggested default. It's not a hard lower
  limit. Technically speaking, it's perfectly fine to create
  just enough entries to fill a single sector. Then again,
  code makes all kinds of assumptions or has all kinds of
  bugs -- just like the logic in the loader apparently.
 
  By having mkimg create a large table, even if it's knows
  up front that it doesn't have to may prevent broken code
  from tripping over, bit it surely bloats the image for
  no reason.
 
 I see what you are saying.  If you have a new device
 and do gpart create -s GPT, then this value in sys/geom/part/g_part_gpt.c:
 
  .gps_minent = 128,
 
 causes the logic in the g_part_ctl_create() function in sys/geom/part/g_part.c
 to set the number of partitions to 128, and then it calls g_part_ctl_create()
 which creates the partition table with 128 empty entries.
 
 gpart create doesn't know how many partitions you want, so it
 needs to allocate some space up front for the partition table,
 and then you can do gpart add to add the partitions later.
 
 With mkimg, you know exactly how many partitions you are creating
 , so you don't need to specify 128 as the number of partitions.

Though, w/ people dd'ing images onto disks, and using growfs to grow
as necessary, we might want to allocate a few more more than the
minimum...  I do agree that we want to keep sizes to a minimum...

 Since only gpart was available for creating GPT partitions, the side-effect
 of always having 128 partitions hid the bug in the loader.
 
 Hopefully Andrey's fix can be backported to at least stable/9, because
 the loader bug seems to have been there since at least 2012.

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 All that I will do, has been done, All that I have, has not.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: mkimg used to create gpt image, problem booting

[Marcel Moolenaar]

On Aug 24, 2014, at 4:31 PM, John-Mark Gurney j...@funkthat.com wrote:
 
 With mkimg, you know exactly how many partitions you are creating
 , so you don't need to specify 128 as the number of partitions.
 
 Though, w/ people dd'ing images onto disks, and using growfs to grow
 as necessary, we might want to allocate a few more more than the
 minimum...  I do agree that we want to keep sizes to a minimum...

One thing I can maybe do is simply fill the empty sectors
that are there because of alignment. If you add -P 4K to
mkimg, then the first partition will by 4K aligned and you
have about 5 sectors unused between the end of the GPT
table and the first sector of the first partition. I may
as well extend the table to cover those unued sectors.

However, this is a pretty side-ways way to end up with a
GPT table that has some extra room. Maybe having scheme-
specific options for this kind of thing is not bad. At
least EBR and APM have the same problem and the BSD
disk label can also be created with more than just 8
partitions.

Related:
o   Is there a need to create images with empty space at
the end of in between partitions?
o   Is there a need to create partitions with specific
indices (i.e. index 6 for a typical 'f' partition)?

Answers to these questions could help figure this out...

-- 
Marcel Moolenaar
mar...@xcllnt.net




signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: mkimg used to create gpt image, problem booting

[Craig Rodrigues]

On Fri, Aug 22, 2014 at 1:45 PM, Marcel Moolenaar mar...@xcllnt.net wrote:

 If I mdconfig the foo1.img disk image, and do a gpart show, I see:

 =  3  1784944  md0  GPT  (872M)
3   321  freebsd-boot  (16K)
   35  17849122  freebsd-ufs  (872M)

 Any idea what I am doing wrong?

 To the best of my knowledge, qemu is the thing you're
 doing wrong :-)

Hi,

I transferred foo1.img to a Mac with VirtualBox, converted it to VMDK
with VBoxManage convertfromraw --format VMDK, and tried to boot it
in VirtualBox.
I got the same error as in QEMU.  It looks like /boot/loader runs,
but I cannot do ls to see the root file system.

I created another disk image with a GPT layout, but this time
using the FreeBSD bsdinstall inside a bhyve VM.

I noticed the following:

WORKING IMAGE BOOTS IN QEMU, CREATED WITH BSDINSTALL
=
=  34  10485693  md0  GPT  (5.0G)
34   1281  83bd6b9d-7f41-11dc-be0b-001560b84f0f  (64K)
   162   99592962  516e7cb6-6ecf-11d6-8ff8-00022d09712b  (4.7G)
   99594585242883  516e7cb5-6ecf-11d6-8ff8-00022d09712b  (256M)
  10483746  1981   - free -  (991K)

DOES NOT BOOT IN QEMU, CREATED WITH MKIMG
===
=  3  1784944  md1  GPT  (872M)
3   321  83bd6b9d-7f41-11dc-be0b-001560b84f0f  (16K)
   35  17849122  516e7cb6-6ecf-11d6-8ff8-00022d09712b  (872M)


I ran the following crazy experiment, just to see what would happen:

dd if=/dev/md1s2 of=/dev/md0s2 bs=8192

I then tried to boot the first image with QEMU, and it booted successfully,
with my UFS file system that I had previously created with makefs.

I'm not sure where to look for the problem.  I notice that
in the non-working image, the offset starts at block 3,
while in the working image, the offset starts at block 34.

Is that enough to make things not boot?

--
Craig
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: mkimg used to create gpt image, problem booting

[Marcel Moolenaar]

On Aug 23, 2014, at 12:00 PM, Craig Rodrigues rodr...@freebsd.org wrote:
 
 I ran the following crazy experiment, just to see what would happen:
 
 dd if=/dev/md1s2 of=/dev/md0s2 bs=8192
 
 I then tried to boot the first image with QEMU, and it booted successfully,
 with my UFS file system that I had previously created with makefs.
 
 I'm not sure where to look for the problem.  I notice that
 in the non-working image, the offset starts at block 3,
 while in the working image, the offset starts at block 34.
 
 Is that enough to make things not boot?

Could be. Try the -P option to mkimg. It sets the
underlying (unexposed) physical sector size while
still working with the visible 512 bytes sectors.
The net effect is that for the GPT scheme things
get aligned to the physical sector size and that
it also causes the image size to be rounded.

You can also try emitting vmdk directly to see if
that makes a difference. vmdk also has the side-
effect of rounding the image to the grain size.

-- 
Marcel Moolenaar
mar...@xcllnt.net




signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: mkimg used to create gpt image, problem booting

[Craig Rodrigues]

On Sat, Aug 23, 2014 at 12:11 PM, Marcel Moolenaar mar...@xcllnt.net wrote:

 Could be. Try the -P option to mkimg. It sets the
 underlying (unexposed) physical sector size while
 still working with the visible 512 bytes sectors.
 The net effect is that for the GPT scheme things
 get aligned to the physical sector size and that
 it also causes the image size to be rounded.

 You can also try emitting vmdk directly to see if
 that makes a difference. vmdk also has the side-
 effect of rounding the image to the grain size.


I tried the following experiments:

mkimg -v -f vmdk -s gpt -b test1/boot/pmbr -p
freebsd-boot:=test1/boot/gptboot -p freebsd-ufs:=/tmp/file.img -o
/tmp/foo1.vmdk

When I tried to boot the image in QEMU, I had the same problem as before.
It looks like it started writing the image on block 3, same as before.


I also tried adding the -P flag, with different values like 2048 and 4096.
I ran into the same problem.

Hmm.

--
Craig
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: mkimg used to create gpt image, problem booting

[Craig Rodrigues]

Hi,

I did some more experiments, and found that after /boot/loader runs,
if I break into the loader prompt and type lsdev, I would get this:

(1)  GPT Disk image which boots under QEMU, made by bsdinstall
==
View from loader

OK lsdev
cd devices:
disk devices:
  disk0:BIOS drive A:
  disk1:BIOS drive C:
 disk1p1: FreeBSD boot
 disk1p2: FreeBSD UFS
 disk1p3: FreeBSD swap
pxe devices:


View from gpart, after we mdconfig the disk image

=  34  10485693  md0  GPT  (5.0G)
34   1281  freebsd-boot  (64K)
   162   99592962  freebsd-ufs  (4.7G)
   99594585242883  freebsd-swap  (256M)
  10483746  1981   - free -  (991K)


(2)  GPT Disk image which fails to boot under QEMU, made by mkimg
===
View from loader

OK lsdev
cd devices:
disk devices:
  disk0:BIOS drive A:
  disk1:BIOS drive C:
pxe devices:

View from gpart, after we mdconfig the disk image


=  3  1784944  md1  GPT  (872M)
3   321  freebsd-boot  (16K)
   35  17849122  freebsd-ufs  (872M)



This leads me to believe that there is logic in /boot/loader,
which is not in GEOM, that fails to parse the GPT produced by mkimg.


I did some further debugging inside the loader by doing the following.
  - I added CFLAGS += -DPART_DEBUG to sys/boot/common/Makefile.inc
  - I added DEBUG() statements all over sys/boot/common/part.c

I observed that in sys/boot/common/part.c in the ptbl_gptread() function,
that in this section:

305 ent = (struct gpt_ent *)tbl;
306 size = MIN(hdr.hdr_entries * hdr.hdr_entsz,
307 MAXTBLSZ * table-sectorsize);
308 for (i = 0; i  size / hdr.hdr_entsz; i++, ent++) {
309 if (uuid_equal(ent-ent_type, gpt_uuid_unused, NULL))
310 continue;

ent-ent_type is all 0's, which matches gpt_uuid_unused, so it bails
out of the loop and never adds the gpt partitions to the list of partitions
that the loader can access.

I'm not familiar with the GPT format, nor am I familiar with the
gpt code inside the loader, and how it differs from GEOM.

Do you have any further ideas of where to hunt for the root cause of
the problem?

Thanks.

--
Craig
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: mkimg used to create gpt image, problem booting

[Marcel Moolenaar]

On Aug 22, 2014, at 9:49 AM, Craig Rodrigues rodr...@freebsd.org wrote:

 
 (5)  Tried to boot the image with qemu:
 
 qemu-system-x86_64 -m 2048 -hda /tmp/foo1.img

*snip*

 If I mdconfig the foo1.img disk image, and do a gpart show, I see:
 
 =  3  1784944  md0  GPT  (872M)
3   321  freebsd-boot  (16K)
   35  17849122  freebsd-ufs  (872M)
 
 Any idea what I am doing wrong?

To the best of my knowledge, qemu is the thing you're
doing wrong :-)

I have so far not been able to boot an image created
by mkimg with a FreeBSD-hosted qemu.
o   VMware and VirtualBox are fine.
o   A non-FreeBSD hosted qemu also works fine.

If your host is running -current, make sure to set
MALLOC_CONF=junk:false. It improves behaviour on
FreeBSD for boot0/boo1.

HTH (probably not),

-- 
Marcel Moolenaar
mar...@xcllnt.net




signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: mkimg used to create gpt image, problem booting

[Ed Maste]

On 22 August 2014 16:45, Marcel Moolenaar mar...@xcllnt.net wrote:

 I have so far not been able to boot an image created
 by mkimg with a FreeBSD-hosted qemu.
 o   VMware and VirtualBox are fine.
 o   A non-FreeBSD hosted qemu also works fine.

For what it's worth, I have no trouble booting a mkimg-created image
with QEMU on FreeBSD, and have been using it regularly for vt(4) and
UEFI development.

Some particulars:
* Host is amd64 stable/10 with some additional MFC candidates (uname
reports FreeBSD 10.0-STABLE #1 r268946+94ba9c8(stable-10))
* My mkimg command was mkimg -s gpt -b $ROOTDIR/boot/pmbr -p
efi:=$ROOTDIR/boot/boot1.efifat -p freebsd-boot:=$ROOTDIR/boot/gptboot
-p freebsd-ufs:=ufs -p freebsd-swap::1M -o img
* QEMU installed from the package, qemu-devel-2.0.0_2
* SeaBIOS version info printed at startup is the same as Craig's
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org