Re: [Multiarch] armhf on arm64 is not working

[Adam Cécile]

Hi there,

So here's what I did finally:

Got two rental server at https://www.scaleway.com/, one is arm64 
(without armhf support) and Paris datacenter still have old hardware for 
rent that's armhf based.
Both cost me around 6€ a month, so that's quite a good offer. Both 
machines have 4 cores and 2Gb ram (swap needed :/) but with decent I/O 
unlike the RPi and its SD card.
That's my current build setup but I'll be changing for home nvidia tegra 
jetson tx1 board which is on its way... It's 239€ without shipping, 
thanks to a nvidia developper offer (I guess you just need to register 
to get the offer too).


Best regards, Adam.

On 08/29/2017 08:54 PM, Adam Cécile wrote:
[This sender failed our fraud detection checks and may not be who they 
appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing]


Right, 1Gb and slow I/O, that's definitely a block for building Java
stuff...


On 08/29/2017 06:31 PM, Christian Seiler wrote:

Hi there,

On 08/29/2017 06:07 PM, Adam Cecile wrote:

Could be an alternative indeed, but what about the speed compared to
my quad-core i5 with qemu ?

I haven't actually tried that specific comparison, but form my
experience a Pi tends to be a tiny bit faster in pure CPU
performance than qemu on Intel. (But not much.) The RPi3 is
also a quad core, so that is similar. YMMV may vary depending
on the precise workload though.

That said:

  - It has only 1 GiB of RAM. That might be a problem.
  - It doesn't have as much cache as an Intel Core CPU, so if
    you have workloads that require a lot of memory access,
    that'll probably offset any small advantages in pure CPU
    performance.
  - I/O is quite slow. If you compile large things my guess is
    that just because of I/O it'll take longer on the Pi than
    with qemu.

OTOH, it's cheap, so even if it's not the right thing in the
end you're not going to waste a ton of money. You could also
first buy just the board and power supply and only buy a case
and other accessories once you've verified that it's sufficient
for your use case.

Then again, there are also other ARM boards in a similar price
range out there, which might suit your use case better. But I
really am not an expert here, I've just played around with the
Pi a bit in the past...

Regards,
Christian

Re: [Multiarch] armhf on arm64 is not working

[Adam Cécile]

Right, 1Gb and slow I/O, that's definitely a block for building Java 
stuff...



On 08/29/2017 06:31 PM, Christian Seiler wrote:

Hi there,

On 08/29/2017 06:07 PM, Adam Cecile wrote:

Could be an alternative indeed, but what about the speed compared to
my quad-core i5 with qemu ?

I haven't actually tried that specific comparison, but form my
experience a Pi tends to be a tiny bit faster in pure CPU
performance than qemu on Intel. (But not much.) The RPi3 is
also a quad core, so that is similar. YMMV may vary depending
on the precise workload though.

That said:

  - It has only 1 GiB of RAM. That might be a problem.
  - It doesn't have as much cache as an Intel Core CPU, so if
you have workloads that require a lot of memory access,
that'll probably offset any small advantages in pure CPU
performance.
  - I/O is quite slow. If you compile large things my guess is
that just because of I/O it'll take longer on the Pi than
with qemu.

OTOH, it's cheap, so even if it's not the right thing in the
end you're not going to waste a ton of money. You could also
first buy just the board and power supply and only buy a case
and other accessories once you've verified that it's sufficient
for your use case.

Then again, there are also other ARM boards in a similar price
range out there, which might suit your use case better. But I
really am not an expert here, I've just played around with the
Pi a bit in the past...

Regards,
Christian

Re: [Multiarch] armhf on arm64 is not working

[Adam Cecile]

Hi,

I was not aware of this optional 32 bit compatibility. That kinda sucks.
Actually, I'm already using qemu with cowbuilder (I mean, a lot) but my 
biggest problem is not the slowness but the broken thread 
implementation. It's nearly impossible to get something built with Java 
without any crash. That's the reason why I took this dedicated arm box 
online.
Anyway, I'll just resign and see if they have different offer with 
different processor.

If you have any hint about online cheap arm server, let me know...

Thanks a lot for the debugging session, I was really running out of ideas.

Regards, Adam.

On 08/29/2017 04:58 PM, Christian Seiler wrote:

Hi,

32bit ARM compatibility is optional according to the specification,
and if your CPU doesn't support it, you won't be able to natively
run armhf executables. This is in contrast to x86, where all[*]
64bit x86 CPUs also support running old 32bit programs.

From what I've read it appears to be that the vast majority of chips
currently sold do have 32bit ARM compatibility - except for one, the
Cavium Thunder X. And if you look at your /proc/cpuinfo output:

Am 2017-08-29 08:49, schrieb Adam Cecile:

CPU implementer: 0x43
CPU architecture: 8
CPU variant: 0x1
CPU part: 0x0a1
CPU revision: 1


Well, yeah, that's the one.

See also:
https://askubuntu.com/questions/928249/how-to-run-armhf-executables-on-an-arm64-system 



So it appears you're out of luck on your hardware: your CPU
simply doesn't support running armhf executables. You could run them
emulated in qemu-user-static, but that's probably not what you want,
because that's going to be _really_ slow.

Regards,
Christian

[*] There were very, very few exceptions, and nothing marketed as a
general purpose CPU.

Re: [Multiarch] armhf on arm64 is not working

[Christian Seiler]

Hi there,

On 08/29/2017 06:07 PM, Adam Cecile wrote:
> Could be an alternative indeed, but what about the speed compared to
> my quad-core i5 with qemu ?

I haven't actually tried that specific comparison, but form my
experience a Pi tends to be a tiny bit faster in pure CPU
performance than qemu on Intel. (But not much.) The RPi3 is
also a quad core, so that is similar. YMMV may vary depending
on the precise workload though.

That said:

 - It has only 1 GiB of RAM. That might be a problem.
 - It doesn't have as much cache as an Intel Core CPU, so if
   you have workloads that require a lot of memory access,
   that'll probably offset any small advantages in pure CPU
   performance.
 - I/O is quite slow. If you compile large things my guess is
   that just because of I/O it'll take longer on the Pi than
   with qemu.

OTOH, it's cheap, so even if it's not the right thing in the
end you're not going to waste a ton of money. You could also
first buy just the board and power supply and only buy a case
and other accessories once you've verified that it's sufficient
for your use case.

Then again, there are also other ARM boards in a similar price
range out there, which might suit your use case better. But I
really am not an expert here, I've just played around with the
Pi a bit in the past...

Regards,
Christian

Re: [Multiarch] armhf on arm64 is not working

[Adam Cecile]

On 08/29/2017 05:19 PM, Christian Seiler wrote:

Hi,

Am 29. August 2017 17:04:29 MESZ schrieb Adam Cecile :

I was not aware of this optional 32 bit compatibility. That kinda
sucks.

Yeah, especially since you had the misfortune of getting the one chip that is 
sold that doesn't support it.


Actually, I'm already using qemu with cowbuilder (I mean, a lot) but my
biggest problem is not the slowness but the broken thread
implementation.

Yeah, that is a problem indeed. And one that will take a long time to fix, 
because threading is already hard enough when you don't need to juggle hardware 
architectures with different memory coherency models...


If you have any hint about online cheap arm server, let me know...

Does it have to be a server? The raspberry pi 3 has an ARMv8 chip with 32bit 
compat mode. Granted, it's not the fastest, doesn't have that much RAM, and if 
you boot it in 64bit mode some peripherals don't work yet, but for a pure 
compile/build box...? Especially if you're kind of OK with the speed of qemu?

Regards,
Christian
Could be an alternative indeed, but what about the speed compared to my 
quad-core i5 with qemu ?

Re: [Multiarch] armhf on arm64 is not working

[Christian Seiler]

Hi,

Am 29. August 2017 17:04:29 MESZ schrieb Adam Cecile :
>I was not aware of this optional 32 bit compatibility. That kinda
>sucks.

Yeah, especially since you had the misfortune of getting the one chip that is 
sold that doesn't support it.

>Actually, I'm already using qemu with cowbuilder (I mean, a lot) but my
>biggest problem is not the slowness but the broken thread 
>implementation.

Yeah, that is a problem indeed. And one that will take a long time to fix, 
because threading is already hard enough when you don't need to juggle hardware 
architectures with different memory coherency models...

>If you have any hint about online cheap arm server, let me know...

Does it have to be a server? The raspberry pi 3 has an ARMv8 chip with 32bit 
compat mode. Granted, it's not the fastest, doesn't have that much RAM, and if 
you boot it in 64bit mode some peripherals don't work yet, but for a pure 
compile/build box...? Especially if you're kind of OK with the speed of qemu?

Regards,
Christian

Re: [Multiarch] armhf on arm64 is not working

[Christian Seiler]

Hi,

32bit ARM compatibility is optional according to the specification,
and if your CPU doesn't support it, you won't be able to natively
run armhf executables. This is in contrast to x86, where all[*]
64bit x86 CPUs also support running old 32bit programs.

From what I've read it appears to be that the vast majority of chips
currently sold do have 32bit ARM compatibility - except for one, the
Cavium Thunder X. And if you look at your /proc/cpuinfo output:

Am 2017-08-29 08:49, schrieb Adam Cecile:

CPU implementer: 0x43
CPU architecture: 8
CPU variant: 0x1
CPU part: 0x0a1
CPU revision: 1


Well, yeah, that's the one.

See also:
https://askubuntu.com/questions/928249/how-to-run-armhf-executables-on-an-arm64-system

So it appears you're out of luck on your hardware: your CPU
simply doesn't support running armhf executables. You could run them
emulated in qemu-user-static, but that's probably not what you want,
because that's going to be _really_ slow.

Regards,
Christian

[*] There were very, very few exceptions, and nothing marketed as a
general purpose CPU.

Re: [Multiarch] armhf on arm64 is not working

[Adam Cecile]

On 08/29/2017 10:50 AM, Reco wrote:

Hi.

On Tue, Aug 29, 2017 at 08:49:40AM +0200, Adam Cecile wrote:

I'm actually running arm64, not amd64 ;-)

My eyes are failing me then ☺.

Can you provide your /proc/cpuinfo then please?

Sure:

processor: 0
BogoMIPS: 200.00
Features: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics
CPU implementer: 0x43
CPU architecture: 8
CPU variant: 0x1
CPU part: 0x0a1
CPU revision: 1

Ok, I got armhf binary running on ARM64, but the only CPU I have access
to has 'CPU implementer' 0x41 and a different set of features. Unsure if
it matters.

So, I'm proposing to start with something more simple - statically
linked binaries.

Get yourself busybox-static package from armhf and armel. Unpack it with
'dpkg -x', see if either of them works.

Reco


Here we go (nothing changes):

gandalf@arm-builder:~/busybox-static/bin$ file busybox
busybox: ELF 32-bit LSB executable, ARM, EABI5 version 1 (GNU/Linux), 
statically linked, for GNU/Linux 3.2.0, 
BuildID[sha1]=04ed8120414c0945c115ee4f8551b6bfdc1e9e40, stripped


gandalf@arm-builder:~/busybox-static/bin$ ldd busybox
not a dynamic executable

gandalf@arm-builder:~/busybox-static/bin$ ./busybox
-bash: ./busybox: cannot execute binary file: Exec format error

Re: [Multiarch] armhf on arm64 is not working

[Reco]

Hi.

On Tue, Aug 29, 2017 at 08:49:40AM +0200, Adam Cecile wrote:
> > > I'm actually running arm64, not amd64 ;-)
> > My eyes are failing me then ☺.
> > 
> > Can you provide your /proc/cpuinfo then please?
>
> Sure:
> 
> processor: 0
> BogoMIPS: 200.00
> Features: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics
> CPU implementer: 0x43
> CPU architecture: 8
> CPU variant: 0x1
> CPU part: 0x0a1
> CPU revision: 1

Ok, I got armhf binary running on ARM64, but the only CPU I have access
to has 'CPU implementer' 0x41 and a different set of features. Unsure if
it matters.

So, I'm proposing to start with something more simple - statically
linked binaries.

Get yourself busybox-static package from armhf and armel. Unpack it with
'dpkg -x', see if either of them works.

Reco

Re: [Multiarch] armhf on arm64 is not working

[Adam Cecile]

On 08/29/2017 08:25 AM, Reco wrote:

Hi.

On Mon, 28 Aug 2017 23:33:50 +0200
Hitec Adam Cecile  wrote:


Hi,

I'm actually running arm64, not amd64 ;-)

My eyes are failing me then ☺.

Can you provide your /proc/cpuinfo then please?

Reco


Sure:

processor: 0
BogoMIPS: 200.00
Features: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics
CPU implementer: 0x43
CPU architecture: 8
CPU variant: 0x1
CPU part: 0x0a1
CPU revision: 1

processor: 1
BogoMIPS: 200.00
Features: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics
CPU implementer: 0x43
CPU architecture: 8
CPU variant: 0x1
CPU part: 0x0a1
CPU revision: 1

processor: 2
BogoMIPS: 200.00
Features: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics
CPU implementer: 0x43
CPU architecture: 8
CPU variant: 0x1
CPU part: 0x0a1
CPU revision: 1

processor: 3
BogoMIPS: 200.00
Features: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics
CPU implementer: 0x43
CPU architecture: 8
CPU variant: 0x1
CPU part: 0x0a1
CPU revision: 1

Re: [Multiarch] armhf on arm64 is not working

[Reco]

Hi.

On Mon, 28 Aug 2017 23:33:50 +0200
Hitec Adam Cecile  wrote:

> Hi,
> 
> I'm actually running arm64, not amd64 ;-)

My eyes are failing me then ☺.

Can you provide your /proc/cpuinfo then please?

Reco

Re: [Multiarch] armhf on arm64 is not working

[Hitec Adam Cecile]

Hi,

I'm actually running arm64, not amd64 ;-)

Le 28 août 2017 23:25:56 GMT+02:00, Reco  a écrit :
>   Hi.
>
>On Mon, 28 Aug 2017 22:50:17 +0200
>Adam Cécile  wrote:
>
>> Hello,
>> 
>> 
>> I'm trying to set up armhf on arm64 stretch but it does not work:
>> 
>> 
>> file /tmp/bash/bin/bash
>> 
>> /tmp/bash/bin/bash: ELF 32-bit LSB executable, ARM, EABI5 version 1 
>> (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for
>
>> GNU/Linux 3.2.0,
>BuildID[sha1]=e3aedef3dd05738b73e7756ddb9be88dd0f9a8ce, 
>> stripped
>> 
>> 
>>   ldd /tmp/bash/bin/bash
>> 
>>  not a dynamic executable
>
>That's supposed to happen. Try 'objdump -p' instead.
>
>
>> /tmp/bash/bin/bash
>> 
>> -bash: /tmp/bash/bin/bash: cannot execute binary file: Exec format
>error
>
>And that is supposed to happen too.
>
>
>> Any idea about what's wrong ? Everything looks ok to me. Is it
>possible 
>> to be kernel related ?
>
>In a way that's kernel related.
>
>What multiarch is - it's an ability to co-install libraries from
>different architectures (for cross-compilation usually).
>What multiarch is not - it's not an ability to use foreign architecture
>libraries to run foreign architecture binaries.
>
>The reason being - an executable binary contains CPU instructions that
>are suited to CPU of designated architecture only. In your example it's
>armv5. Your CPU is amd64 and its unable to interpret those - it can
>only
>understand i386 and amd64 instruction sets.
>
>Due to this Linux kernel (any reasonable OS kernel that I dealt with,
>in
>fact) refuses to acknowledge that foreign architecture binaries could
>be
>executed at all. Hence 'Exec format error' from the direct execution
>and 'not a dynamic executable' from ldd (which is special form of
>binary execution anyway).
>
>
>There's a way around this limitation, and it involves CPU instruction
>translation via qemu-user. Assuming that you have 'qemu-user-static'
>installed, this should work:
>
>qemu-arm-static /tmp/bash/bin/bash
>
>Reco

-- 
Envoyé de mon appareil Android avec Courriel K-9 Mail. Veuillez excuser ma 
brièveté.

Re: [Multiarch] armhf on arm64 is not working

[Sven Hartge]

Reco  wrote:

> The reason being - an executable binary contains CPU instructions that
> are suited to CPU of designated architecture only. In your example it's
> armv5. Your CPU is amd64 and its unable to interpret those - it can only
> understand i386 and amd64 instruction sets.

No, he is talking about ARM64, the 64bit variant von ARM.

S°

-- 
Sigmentation fault. Core dumped.

Re: [Multiarch] armhf on arm64 is not working

[Reco]

Hi.

On Mon, 28 Aug 2017 22:50:17 +0200
Adam Cécile  wrote:

> Hello,
> 
> 
> I'm trying to set up armhf on arm64 stretch but it does not work:
> 
> 
> file /tmp/bash/bin/bash
> 
> /tmp/bash/bin/bash: ELF 32-bit LSB executable, ARM, EABI5 version 1 
> (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for 
> GNU/Linux 3.2.0, BuildID[sha1]=e3aedef3dd05738b73e7756ddb9be88dd0f9a8ce, 
> stripped
> 
> 
>   ldd /tmp/bash/bin/bash
> 
>  not a dynamic executable

That's supposed to happen. Try 'objdump -p' instead.


> /tmp/bash/bin/bash
> 
> -bash: /tmp/bash/bin/bash: cannot execute binary file: Exec format error

And that is supposed to happen too.


> Any idea about what's wrong ? Everything looks ok to me. Is it possible 
> to be kernel related ?

In a way that's kernel related.

What multiarch is - it's an ability to co-install libraries from
different architectures (for cross-compilation usually).
What multiarch is not - it's not an ability to use foreign architecture
libraries to run foreign architecture binaries.

The reason being - an executable binary contains CPU instructions that
are suited to CPU of designated architecture only. In your example it's
armv5. Your CPU is amd64 and its unable to interpret those - it can only
understand i386 and amd64 instruction sets.

Due to this Linux kernel (any reasonable OS kernel that I dealt with, in
fact) refuses to acknowledge that foreign architecture binaries could be
executed at all. Hence 'Exec format error' from the direct execution
and 'not a dynamic executable' from ldd (which is special form of
binary execution anyway).


There's a way around this limitation, and it involves CPU instruction
translation via qemu-user. Assuming that you have 'qemu-user-static'
installed, this should work:

qemu-arm-static /tmp/bash/bin/bash

Reco

[Multiarch] armhf on arm64 is not working

[Adam Cécile]

Hello,


I'm trying to set up armhf on arm64 stretch but it does not work:


file /tmp/bash/bin/bash

/tmp/bash/bin/bash: ELF 32-bit LSB executable, ARM, EABI5 version 1 
(SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for 
GNU/Linux 3.2.0, BuildID[sha1]=e3aedef3dd05738b73e7756ddb9be88dd0f9a8ce, 
stripped



 ldd /tmp/bash/bin/bash

not a dynamic executable


/tmp/bash/bin/bash

-bash: /tmp/bash/bin/bash: cannot execute binary file: Exec format error


dpkg -S /lib/ld-linux-armhf.so.3

libc6:armhf: /lib/ld-linux-armhf.so.3


Any idea about what's wrong ? Everything looks ok to me. Is it possible 
to be kernel related ?



Thanks in advance, Regards.