Re: archdetect guessing on arm* != armel

2014-01-03 Thread Ian Campbell 
On Fri, 2014-01-03 at 04:00 +, Dimitri John Ledkov wrote:
 Hey Ian,
 
 I wonder if I see some mid-air collision =)

It looks like we are working along the same lines, so no harm done.

 I am not sure, so not to waste everyone elses time, maybe you can take
 a look at the current ubuntu proposed patches to see how they are
 similar or different to your proposals.

I guess not to waste everyone elses time is why the CCs were dropped?
Having written my response I think it would be useful to share with the
list so I've added them back. I hope that is OK.

I also Bcc'd the folks who I originally CC'd so they don't wonder what
happened. Although I expect are on one of the two lists anyway so
wouldn't miss it.

 I've opened your gitorious page which informed me that created
 repositories are empty?! =) Is is all fully / correctly pushed?

libdebian-installer.git should have been OK, but it seems like the lack
of a master branch makes the webui for selecting non-default branches
fail to appear. There was a direct link in the Activities list for use
ijc which was to:
https://gitorious.org/ijc-debian/libdebian-installer/commits/4b3ba870c86242a280e2e746544efdb2c8032928
plus I've just pushed a dummy master branch to keep it happy.

When I went to create base-installer.git yesterday there was some sort
of server error, I'm stuck in a loop on Your new repository is being
created. I've mailed their support alias as they requested.

In the meantime the 3 patches to that repo are appended.

 The patches for armel(symlinked as armhf)/arm64 that have been
 recently proposed look like following:
 https://bugs.launchpad.net/ubuntu/+source/base-installer/+bug/1263756
 https://code.launchpad.net/~dannf/ubuntu/trusty/base-installer/arm64+calxeda-subarches/+merge/199977

I think Dann has already pushed the libdebian-installer bits to Debian.

 Which is:
 - for arm64 just return generic

Eventually this should detect and report efi as well. Given the
general intention in the ecosystem to use efi on arm64 this should
perhaps even become the default.

I think there is little value in symlinking subarch-arm64-linux.c to the
32 bit arm ones -- really it just needs to statically return generic
and not do all the DT/cpuinfo parsing and matching etc, which is the
bulk of the 32 bit version.

Maybe there will be some code to share if we add efi support to armhf
too -- but more likely the efi detection in subarch-x86-linux.c should
be refactored into a helper which they can all use.

 - for armel/armhf majority of subarches just return generic

Ack. Where on armhf majority includes all new subarches.

 - for armhf, calxeda - subarch midway return generic-lpae
 - for armhf, calxeda - subarch highbank return generic

These two shouldn't be necessary. Both should report generic and the
selection of armmp vs. armmp-lpae (s/armmp/generic/g in Ubuntu?) should
be done in base-installer.git/kernel/armhf.sh by
inspecting /proc/cpuinfo for the lpae feature. Otherwise we will need to
enumerate every platform/processor which supports LPAE, which would be
very tedious.

I've got the cpuinfo based stuff in my tree/patches appended below
(heavily based on the equivalent x86 code), it seems to work in the test
suite I added but for some reason the installer is still picking armmp
to actually install. Probably the order of something needs reversing...

So, I think the specific detection of midway and highbank (and the virt
flavour, and arm64 xgene  foundation model) should be removed from the
lists and rely on the fallback return generic.

(https://code.launchpad.net/~dannf/ubuntu/trusty/base-installer/arm64
+calxeda-subarches/+merge/199977 seems to add highbank and midwap
(/lpae) suport to armel rather than armhf -- that seems odd to me given
that they are hardfloat capable systems, is this just some Ubuntu
oddity)

 I do hope going forward it will be mostly: ! calxeda-midway ! ads,
 return generic. Or some such.
 
 But doesn't above indicate that some subarch handling is still needed?

I think not given that lpae vs !lpae can be detected via cpuinfo.

 Or can generic-lpae / generic / ads (not sure what's that) kernel
 flavors be merged?

generic^Warmmp-lpae and generic^Warmmp cannot be merged because lpae
address extensions are a static compile time choice -- the page table
entries are different sizes. I principal you could handle this
dynamically (AIUI netware did on x86) but in reality there would be
loads of complexity (boot time patching) and/or overhead (select right
function on every PT update) for not very much gain. We have two
flavours on 32-bit x86 for the same reason.

I don't know what ads is/was but it is armel not armhf and I think for
the most part we should just leave the subarch stuff as is on armel. (At
least for now, probably forever).

Ian.

- Patches to base-installer.git --

From ffdd885d2f17ff3f599df3e2b4055a500331f41a Mon Sep 17 00:00:00 2001
From: Ian Campbell i...@hellion.org.uk
Date: Thu, 3 Oct 2013 20:04:51 +0100

Re: archdetect guessing on arm* != armel

2014-01-03 Thread Dimitri John Ledkov 
On 3 January 2014 10:03, Ian Campbell i...@hellion.org.uk wrote:
 On Fri, 2014-01-03 at 04:00 +, Dimitri John Ledkov wrote:

 I don't know what ads is/was but it is armel not armhf and I think for
 the most part we should just leave the subarch stuff as is on armel. (At
 least for now, probably forever).


armhf is a symlink to armel. I guess they are all armish v7 and
ubuntu only has armhf these days. Which does indeed look backwards
when looking at the diffs =)

Regards,

Dimitri.


-- 
To UNSUBSCRIBE, email to debian-boot-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/CANBHLUg3QS9g_DqxV4Q64fFL-q4jey=brUwVO7V3=jbwcvx...@mail.gmail.com

Re: archdetect guessing on arm* != armel

2014-01-03 Thread Ian Campbell 
On Fri, 2014-01-03 at 11:24 +, Dimitri John Ledkov wrote:
 On 3 January 2014 10:03, Ian Campbell i...@hellion.org.uk wrote:
  On Fri, 2014-01-03 at 04:00 +, Dimitri John Ledkov wrote:
 
  I don't know what ads is/was but it is armel not armhf and I think for
  the most part we should just leave the subarch stuff as is on armel. (At
  least for now, probably forever).
 
 
 armhf is a symlink to armel.

base-installer.git/kernel/arm{el,hf}.sh are separate in Debian, but yes
in libdebian-installer.git subarch-arm{hf,el}-linux are indeed links to
subarch-arm-linux.c.

I mention this since the confusing to me Ubuntu diff was the
base-installer.git kernel/armel.sh one.

 I guess they are all armish v7 and
 ubuntu only has armhf these days.

Right.

My understanding was that in Ubuntu (only) armel was upgraded to v7,
then armhf came along and Ubuntu switched to that for its v7 port and
armel was removed (reverting it to v5 was proposed, but AIUI not
something which happened).

In Debian armel was always v5 and armhf was always v7, I think.

  Which does indeed look backwards
 when looking at the diffs =)

Indeed.

Ian.


-- 
To UNSUBSCRIBE, email to debian-boot-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/1388749412.16958.106.ca...@hastur.hellion.org.uk

Re: archdetect guessing on arm* != armel

2014-01-03 Thread Ian Campbell 
On Fri, 2014-01-03 at 10:03 +, Ian Campbell wrote:
[...]
 When I went to create base-installer.git yesterday there was some sort
 of server error, I'm stuck in a loop on Your new repository is being
 created. I've mailed their support alias as they requested.

And they've now fixed it.

 In the meantime the 3 patches to that repo are appended.

Now pushed to
https://gitorious.org/ijc-debian/base-installer/commits/25cb4b00fec0b1fa4f7eb4431b33cf846c93a294
 plus two additional patches which fixup the lpae support to use lpae by 
default iff the h/w supports it.

Ian.


-- 
To UNSUBSCRIBE, email to debian-boot-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/1388755461.16958.112.ca...@hastur.hellion.org.uk

Re: archdetect guessing on arm* != armel

2014-01-03 Thread dann frazier 
On Thu, Jan 02, 2014 at 10:52:38PM +, Ian Campbell wrote:
 Hi,
 
 In 2010[0,1] support was added to archdetect for guessing the subarch on
 armel platforms by looking at the running kernel versions suffix, this
 was useful because some armel kernel flavours supported multiple boards
 and it meant that support for such platforms could come essentially for
 free without having to edit the installer initrd. This is implemented by
 di_system_subarch_analyze_guess() in libdebianinstaller and exposed via
 archdetect -g.
 
 However I can't actually find anywhere which uses archdetect -g though,
 even for armel, was it never implemented or was it removed? (I couldn't
 find it in a grep through the git log -p of some plausible repos)
 
 Anyway, armel is what it is but this functionality has now also been
 extended to armhf and more recently arm64. I'm not sure this makes sense
 going forward -- the upstream kernel folks have been moving towards a
 single image multiplatform model (i.e. one kernel binary on all new
 platforms, driven by Device Tree) and in Debian we have been following
 suite in the armhf flavour where we have consolidated into just two
 flavours (armmp and armmp-lpae/bigmem). For ARM64 the situation is even
 more clear cut, there upstream has been multiplatform from day one and
 there will never be multiple platforms.
 
 For armhf and arm64 I think we should be moving away from a model which
 requires us to whitelist supported subarchs in the installer towards one
 which takes advantage of the upstream progress to work on most
 platforms. I've been experimenting with having archdetect return
 armhf/generic and handling that in kernel.sh, which seems to work
 fine.

I agree with that - my reasoning for making the changes I did was to
be able to *blacklist* certain subarchitectures which need to behave
different than the default. For instance, knowing that a certain
subarch uses u-boot/flash-kernel vs. grub-efi... which will be the
case for at least one platform. Though perhaps subarch detection is an
incorrect mechanism for this and we should instead look for the
existance of efi interfaces?

  -dann

 On the bootloader side we discussed at the ARM sprint in Cambridge
 (notes at [2]) chainloading grub-uboot on armhf. On arm64 I expect the
 choice will be grub-efi. Both should reduce or eliminate the need for
 subarch specific code.
 
 So, I think di_system_subarch_analyze_guess should be disabled for armhf
 and arm64, and that map_hardware should not have any new armhf platforms
 added to it (or any arm64 ones added at all).
 
 But, I can't quite work out how this infrastructure actually worked in
 practice, because I can't see the consumers anywhere. That makes me wary
 since I think I might be missing something and I don't want to pull the
 rug out on something which is useful.
 
 The only places where I foresee there might be any need for subarch
 specific stuff on armhf are:
 
   * grub-installer or flash-kernel: Something needs to arrange to
 chainload grub. A default of writing /boot/boot.scr (using
 ${kernel_addr_r}?) will help keep this to a minimum I think
   * grub-installer: On armhf needs to be able to select a load
 address for grub (How? TBD, but could look
 into /proc/device-tree/memory* or /sys files perhaps, or even
 better maybe the upstream discussion[3] will result in a runtime
 relocatable grub and eliminate this too)
 
 In all cases this should in any case be based on /proc/device-tree/model
 (which is guaranteed to be present for armhf now that we use armmp).
 
 With EFI on arm64 there should be no need for subarch specific code.
 
 WRT the existing guessing on armel, does it actually work usefully? the
 list of kernel version suffixes which we match today in order to make a
 subarch guess (as opposed to an exact match) from is:
 dove
 omap  (armhf kernel flavour until Wheezy)
 omap4
 mx51
 mx5   (armhf kernel flavour until Wheezy, supported by
flash-kernel, supported by base-installer 
kernel.sh)
 vexpress  (armhf kernel flavour until Wheezy, supported by
base-installer kernel.sh)
 
 AFAICT none of dove, omap4 or mx51 have ever been Debian kernel flavours
 or supported by flash-kernel AFAICT. Perhaps they were Ubuntu kernel
 flavours? Or old armeb flavours? Only mx5 and vexpress are understood by
 the base-installer's kernel selection stuff.
 
 Given that I'm wondering if it might be easiest to just nuke all of this
 stuff across the board even for armel.
 
 I've pushed my WIP to the devel/armhf-armmp branches
 libdebian-installer.git and base-installer.git under
 https://gitorious.org/ijc-debian .
 
 Ian.
 
 [0] https://lists.debian.org/debian-boot/2010/08/msg00641.html
 [1] https://lists.debian.org/debian-boot/2010/09/msg00118.html
 [2]

Re: archdetect guessing on arm* != armel

2014-01-03 Thread Ian Campbell 
On Fri, 2014-01-03 at 08:04 -0700, dann frazier wrote:

  For armhf and arm64 I think we should be moving away from a model which
  requires us to whitelist supported subarchs in the installer towards one
  which takes advantage of the upstream progress to work on most
  platforms. I've been experimenting with having archdetect return
  armhf/generic and handling that in kernel.sh, which seems to work
  fine.
 
 I agree with that - my reasoning for making the changes I did was to
 be able to *blacklist* certain subarchitectures which need to behave
 different than the default. For instance, knowing that a certain
 subarch uses u-boot/flash-kernel vs. grub-efi... which will be the
 case for at least one platform. Though perhaps subarch detection is an
 incorrect mechanism for this and we should instead look for the
 existance of efi interfaces?

Right, I think looking for functional properties (i.e. the presence of
EFI) should be done in preference to lists (black or white) of model
names.

subarch-x86-linux.c for example looks for the existence
of /sys/firmware/efi -- I think we can simply follow suite on ARM.

x86 does have a few hardcoded subarches (support for macs) based on DMI
information (somewhat morally equivalent to the DT model on ARM), but we
should strive on both x86 and arm for these to be the exception not the
rule.

Ian.


-- 
To UNSUBSCRIBE, email to debian-boot-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/1388762196.16958.131.ca...@hastur.hellion.org.uk

archdetect guessing on arm* != armel

2014-01-02 Thread Ian Campbell 
Hi,

In 2010[0,1] support was added to archdetect for guessing the subarch on
armel platforms by looking at the running kernel versions suffix, this
was useful because some armel kernel flavours supported multiple boards
and it meant that support for such platforms could come essentially for
free without having to edit the installer initrd. This is implemented by
di_system_subarch_analyze_guess() in libdebianinstaller and exposed via
archdetect -g.

However I can't actually find anywhere which uses archdetect -g though,
even for armel, was it never implemented or was it removed? (I couldn't
find it in a grep through the git log -p of some plausible repos)

Anyway, armel is what it is but this functionality has now also been
extended to armhf and more recently arm64. I'm not sure this makes sense
going forward -- the upstream kernel folks have been moving towards a
single image multiplatform model (i.e. one kernel binary on all new
platforms, driven by Device Tree) and in Debian we have been following
suite in the armhf flavour where we have consolidated into just two
flavours (armmp and armmp-lpae/bigmem). For ARM64 the situation is even
more clear cut, there upstream has been multiplatform from day one and
there will never be multiple platforms.

For armhf and arm64 I think we should be moving away from a model which
requires us to whitelist supported subarchs in the installer towards one
which takes advantage of the upstream progress to work on most
platforms. I've been experimenting with having archdetect return
armhf/generic and handling that in kernel.sh, which seems to work
fine.

On the bootloader side we discussed at the ARM sprint in Cambridge
(notes at [2]) chainloading grub-uboot on armhf. On arm64 I expect the
choice will be grub-efi. Both should reduce or eliminate the need for
subarch specific code.

So, I think di_system_subarch_analyze_guess should be disabled for armhf
and arm64, and that map_hardware should not have any new armhf platforms
added to it (or any arm64 ones added at all).

But, I can't quite work out how this infrastructure actually worked in
practice, because I can't see the consumers anywhere. That makes me wary
since I think I might be missing something and I don't want to pull the
rug out on something which is useful.

The only places where I foresee there might be any need for subarch
specific stuff on armhf are:

  * grub-installer or flash-kernel: Something needs to arrange to
chainload grub. A default of writing /boot/boot.scr (using
${kernel_addr_r}?) will help keep this to a minimum I think
  * grub-installer: On armhf needs to be able to select a load
address for grub (How? TBD, but could look
into /proc/device-tree/memory* or /sys files perhaps, or even
better maybe the upstream discussion[3] will result in a runtime
relocatable grub and eliminate this too)

In all cases this should in any case be based on /proc/device-tree/model
(which is guaranteed to be present for armhf now that we use armmp).

With EFI on arm64 there should be no need for subarch specific code.

WRT the existing guessing on armel, does it actually work usefully? the
list of kernel version suffixes which we match today in order to make a
subarch guess (as opposed to an exact match) from is:
dove
omap(armhf kernel flavour until Wheezy)
omap4
mx51
mx5 (armhf kernel flavour until Wheezy, supported by
 flash-kernel, supported by base-installer 
 kernel.sh)
vexpress(armhf kernel flavour until Wheezy, supported by
 base-installer kernel.sh)

AFAICT none of dove, omap4 or mx51 have ever been Debian kernel flavours
or supported by flash-kernel AFAICT. Perhaps they were Ubuntu kernel
flavours? Or old armeb flavours? Only mx5 and vexpress are understood by
the base-installer's kernel selection stuff.

Given that I'm wondering if it might be easiest to just nuke all of this
stuff across the board even for armel.

I've pushed my WIP to the devel/armhf-armmp branches
libdebian-installer.git and base-installer.git under
https://gitorious.org/ijc-debian .

Ian.

[0] https://lists.debian.org/debian-boot/2010/08/msg00641.html
[1] https://lists.debian.org/debian-boot/2010/09/msg00118.html
[2] https://lists.debian.org/debian-devel-announce/2013/12/msg1.html
[3] http://lists.gnu.org/archive/html/grub-devel/2013-12/msg00415.html


-- 
To UNSUBSCRIBE, email to debian-boot-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/1388703158.16958.64.ca...@hastur.hellion.org.uk