Re: Using 4KiB page size by default on buildds

[Thadeu Lima de Souza Cascardo]

On Mon, Nov 14, 2016 at 12:23:17AM +0200, Aaro Koskinen wrote:
> On Sat, Nov 12, 2016 at 09:02:37AM -0200, Thadeu Lima de Souza Cascardo wrote:
> > So, my proposal here is that we decide on a default page size for
> > buildds and kernels shipped with Debian, give the option for the
> > different page size for other users, but state there might be breakages
> > when using it.
> 
> If a program needs to know about page size, it should query/adopt to it
> runtime, not buildtime.
> 
> A.

Sure, that's the long term plan: fixing all software. In the case of
jemalloc, we have identified there is a problem and can start working on
fixing it right away. However, what about all the other hidden bugs we
haven't identified yet?

Even in the case of jemalloc, a quick and dirty fix on Debian sid is
easy. A proper fix acceptable upstream for the old version present in
Jessie that won't impact that much all the other architectures is
another thing.

My point is: I agree with you that we should properly fix the broken
software, I just think that we should have a change now that might
prevent bugs like this from impacting users in the meanwhile.

Cascardo.


signature.asc
Description: PGP signature

Re: Using 4KiB page size by default on buildds

[Thadeu Lima de Souza Cascardo]

On Mon, Nov 14, 2016 at 09:17:56AM +0100, Mathieu Malaterre wrote:
> Hi,
> 
> On Sat, Nov 12, 2016 at 12:02 PM, Thadeu Lima de Souza Cascardo
>  wrote:
> > Hi,
> >
> > While recently investigating the mariadb bug, I found out the culprit is
> > in fact jemalloc. This library assumes the system page size is the one
> > that it found when building it. The version in Jessie was built on a
> > 4KiB-page size system. So, when using it on a 64KiB-page size system, we
> > might see crashes. That's why building mariadb for Jessie on partch
> > fails. mariadb uses jemalloc and during make check, a crash induces a
> > deadlock.
> 
> Nice ! added usertags 'powerpc':
> 
> https://udd.debian.org/cgi-bin/bts-usertags.cgi?tag=powerpc=debian-powerpc%40lists.debian.org

Thanks.

> 
> > To give some context, I used to work fixing drivers for IBM Power
> > Systems, which used 64KiB-page size kernels on most situations. And we
> > found many issues due to people assuming 4KiB page sizes.
> >
> > Even though we could debate about the value of using any page size
> > different from 4KiB, the fact is that there are systems out there using
> > different page sizes, and even Debian ships with some kernels and uses
> > it on its buildds.
> >
> > I, for one, believe we should support such different page sizes on the
> > entire archive. This means fixing jemalloc, device drivers, other memory
> > allocators and any other software that might break or work suboptimally
> > on such systems.
> >
> > However, the current state of things has meant that powerpc has been
> > dropped from squeeze and while I would agree that jemalloc might have
> > been an exception, who knows what else might break because of such
> > differences? We might look for things like page.*size/i in the archive,
> > but upstream fixes might not always be an easy and quick path. Consider
> > the jemalloc case, where upstream has diverged a lot from the current
> > version in Debian, and I am not sure how they might take a fix for the
> > version in Jessie.
> >
> > So, my proposal here is that we decide on a default page size for
> > buildds and kernels shipped with Debian, give the option for the
> > different page size for other users, but state there might be breakages
> > when using it. That doesn't mean it's not supported, and shipping it is
> > a way of giving the chance for it to be tested. It's not broken either,
> > most things should work, but some things here and there are broken, and
> > we could even have a list of known issues (that we want to fix
> > eventually, though in future releases).
> >
> > Given that most users I see on this list are using 10+ year old
> > hardware, which likely have low memory capacity and some support only
> > 32-bit kernels, I'd say we default to 4KiB page size, and that's where
> > most software will work, while 64KiB page size will impact much more
> > broken software.
> >
> > Next steps here are getting a 64-bit kernel built with 4KiB page size
> > available in the archives, and using those kernels on those buildds that
> > currently run a 64-bit kernel.
> 
> The old ppc64 was this way, see
> 
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=826796
> 

Yeah, when I talked about drivers, I was thinking about nouveau, as I
have read before on this list that it was a problem on 64KiB page size
kernels.

> > What are the thoughts of the Debian powerpc community?
> 
> As you know 'powerpc' has been removed from the release archs. So
> instead I would suggest that you get in touch with reproducible people
> instead. Ideally the autopkg test should (could?) be run on a kernel
> with different page size.
> 

Good point bringing up that the package would produce different
binaries, hence the reproducible build folks might take an interest.

But I don't think we should it *instead*. We should it *too*. powerpc
has been removed from Stretch. It may be included in Stretch+1, or at
least I would expect ports.d.o to pick it up. Does it make sense?

> But I fail to see how you can address something like:
> 
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=819711
> 
> The page size is hardcoded (at built time) to 64K, so that it passes
> on both partch.d.o (64K page) as well as my old G4 (4K page)...

You mean that it will produce the same binaries no matter the build
system? But does it run fine on 4KiB page size systems? I agree
scenarios where the software builds assuming a 64KiB page size because
it's PPC will not be handled by my proposal. I still think such software
is broken and must be fixed. But 4KiB as the default size should still
show less broken software for the user.

This particular bug seems to show exactly the kind of problem that we
have because of different page sizes on kernels we use to build
packages. A FTBFS was caused when it was built on partch on a 64KiB-page
size kernel. Had we set the plan above in motion many months ago,
perhaps we would have had a healthier powerpc port?

Best 

Re: Using 4KiB page size by default on buildds

[Mathieu Malaterre]

Hi,

On Sat, Nov 12, 2016 at 12:02 PM, Thadeu Lima de Souza Cascardo
 wrote:
> Hi,
>
> While recently investigating the mariadb bug, I found out the culprit is
> in fact jemalloc. This library assumes the system page size is the one
> that it found when building it. The version in Jessie was built on a
> 4KiB-page size system. So, when using it on a 64KiB-page size system, we
> might see crashes. That's why building mariadb for Jessie on partch
> fails. mariadb uses jemalloc and during make check, a crash induces a
> deadlock.

Nice ! added usertags 'powerpc':

https://udd.debian.org/cgi-bin/bts-usertags.cgi?tag=powerpc=debian-powerpc%40lists.debian.org

> To give some context, I used to work fixing drivers for IBM Power
> Systems, which used 64KiB-page size kernels on most situations. And we
> found many issues due to people assuming 4KiB page sizes.
>
> Even though we could debate about the value of using any page size
> different from 4KiB, the fact is that there are systems out there using
> different page sizes, and even Debian ships with some kernels and uses
> it on its buildds.
>
> I, for one, believe we should support such different page sizes on the
> entire archive. This means fixing jemalloc, device drivers, other memory
> allocators and any other software that might break or work suboptimally
> on such systems.
>
> However, the current state of things has meant that powerpc has been
> dropped from squeeze and while I would agree that jemalloc might have
> been an exception, who knows what else might break because of such
> differences? We might look for things like page.*size/i in the archive,
> but upstream fixes might not always be an easy and quick path. Consider
> the jemalloc case, where upstream has diverged a lot from the current
> version in Debian, and I am not sure how they might take a fix for the
> version in Jessie.
>
> So, my proposal here is that we decide on a default page size for
> buildds and kernels shipped with Debian, give the option for the
> different page size for other users, but state there might be breakages
> when using it. That doesn't mean it's not supported, and shipping it is
> a way of giving the chance for it to be tested. It's not broken either,
> most things should work, but some things here and there are broken, and
> we could even have a list of known issues (that we want to fix
> eventually, though in future releases).
>
> Given that most users I see on this list are using 10+ year old
> hardware, which likely have low memory capacity and some support only
> 32-bit kernels, I'd say we default to 4KiB page size, and that's where
> most software will work, while 64KiB page size will impact much more
> broken software.
>
> Next steps here are getting a 64-bit kernel built with 4KiB page size
> available in the archives, and using those kernels on those buildds that
> currently run a 64-bit kernel.

The old ppc64 was this way, see

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=826796

> What are the thoughts of the Debian powerpc community?

As you know 'powerpc' has been removed from the release archs. So
instead I would suggest that you get in touch with reproducible people
instead. Ideally the autopkg test should (could?) be run on a kernel
with different page size.

But I fail to see how you can address something like:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=819711

The page size is hardcoded (at built time) to 64K, so that it passes
on both partch.d.o (64K page) as well as my old G4 (4K page)...

Thanks anyway for your interest in powerpc.

-M

Re: ppc64(el) failure, haskell-zeromq4-haskell, Error: operand out of domain (2 is not a multiple of 4)

[Breno Leitao]

Hello Clint,

On 11/12/2016 01:39 PM, Clint Adams wrote:
> Hi Breno,
> 
> It's unclear how to contact the ppc64el porters.  Is there a mailing
> list?

Yes. We use the debian-powerpc mailing list. I am copying them to this email.

> Something appears to have broken with GHC's PPC64 code generation.
> When trying to build haskell-zeromq4-haskell[0], we get
> 
> /tmp/ghc8591_0/ghc_6.s:14471:0: error:
>  Error: operand out of domain (2 is not a multiple of 4)
> 
> This appears to be an addis instruction.
> 
> Could you help explain what's wrong with it or how to fix it?

Sure. I will take a look at this issue. I will let you know when I have new
findings.


> [0] 
> https://buildd.debian.org/status/fetch.php?pkg=haskell-zeromq4-haskell=ppc64el=0.6.5-5=1478924753

Where is powerpc now?

[Matti Palmström]

Since the other day I get

N: Skipping acquire of configured file 'main/binary-powerpc/Packages' as 
repository 'http://ftp.us.debian.org/debian stretch InRelease' doesn't 
support architecture 'powerpc'


Is this just a temporary glitch or are we moving to debian-ports?

Regards
/M

Re: Where is powerpc now?

[John Paul Adrian Glaubitz]

On 11/15/2016 01:05 AM, Matti Palmström wrote:
> N: Skipping acquire of configured file 'main/binary-powerpc/Packages' as 
> repository 'http://ftp.us.debian.org/debian stretch InRelease' doesn't support
> architecture 'powerpc'
> 
> Is this just a temporary glitch or are we moving to debian-ports?

powerpc has been removed from testing (Stretch). Your only option is switching 
to unstable.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaub...@debian.org
`. `'   Freie Universitaet Berlin - glaub...@physik.fu-berlin.de
  `-GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Bug#842513: vlc: immediate crash on launch on powerpc

[Lennart Sorensen]

On Sat, Nov 12, 2016 at 10:00:53PM +0200, Adrian Bunk wrote:
> I do not understand this part.
> 
> -maltivec should only be set for the code that is behind the runtime 
> feature check, so this code is only run on hardware that has AltiVec.

Well a mistake in the configure script is making it set for everything.

After fixing that there is the problem of how to build merge.c in
deinterlace with only -maltivec for the altivec merge function, and not
the rest of the file which is supposed to be the generic C implementation.

I guess like the yuv file that has altivec support, the altivec merge
function needs to be built by itself with -maltivec while the rest of
the file is built without it.

I am trying to figure out how to do that but haven't quite figured it
out yet.  Certainly that is the correct way to fix it.

-- 
Len Sorensen