Re: [Tails-dev] Shipping a 686-pae kernel

[Ague Mill]

On Sat, Sep 29, 2012 at 05:50:05PM +0200, intrigeri wrote:
> > What's your opinion? Should I proceed in adding a custom
> > `python-dbus` to the multikernel branch?
> 
> Please proceed: at least so that we can verify that this hack
> workarounds the bug in various settings.

Done. Merged in experimental. My tests are still conclusive and I really
think the branch should be reviewed one more time and merged in devel
now.
 
> I'm fine with shipping a patched python-dbus in Tails (obviously, as
> long as we report the bug and forward the patch at least to Debian).

If the bug is still reproducible in Wheezy, it has to be done. But I am
really unsure on the best place to actually report the bug. Someone
would probably have to come up with a smaller test case, too.

-- 
Ague


pgpJYgGbrgWHq.pgp
Description: PGP signature
___
tails-dev mailing list
tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev

Re: [Tails-dev] Shipping a 686-pae kernel

[Ague Mill]

On Sat, Sep 29, 2012 at 08:42:37PM +0200, intrigeri wrote:
> These Vagrant / memory tweaks made in the feature/multikernel branch
> should be re-done there, as they conflicted with those made in
> feature/vagrant and since then merged into devel, which I've just
> merged back into feature/multikernel.

I don't think they are neeeded anymore after the various improvements to
the build system that was made.

-- 
Ague


pgpTHekbusrg1.pgp
Description: PGP signature
___
tails-dev mailing list
tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev

Re: [Tails-dev] Shipping a 686-pae kernel

[intrigeri]

anonym wrote (06 Sep 2012 09:18:03 GMT) :
> 05/09/12 23:38, anonym wrote:
>> 04/09/12 14:52, intrigeri:
>>> So, I just merged the feature/multikernel branch into experimental.
>> 
>> Built and so far tested in VirtualBox.

> I forgot to say that this pushed the amount of RAM required to build
> Tails in-memory to over 6 GiB (at least when using Vagrant) so I pushed
> commit 974805d to bump it to 7 GiB.

These Vagrant / memory tweaks made in the feature/multikernel branch
should be re-done there, as they conflicted with those made in
feature/vagrant and since then merged into devel, which I've just
merged back into feature/multikernel.

___
tails-dev mailing list
tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev

Re: [Tails-dev] Shipping a 686-pae kernel

[intrigeri]

Hi,

Ague Mill wrote (28 Sep 2012 15:27:18 GMT) :
> So my hack have been to add a call to `dbus_thread_init_default()`
> in the initialization of the Python `dbus` module. And it looks like
> it solve the issue. I have not been able to get the installer to
> crash after installing the modified `python-dbus` package.

Congrats for the debugging and for finding this hack!

> From what I can read from the documentation, except a performance
> hit, there should be no other downsides to it.

> The binary `python-dbus` package is fairly small (226 kB) and at
> this time, it's a patch I feel we can carry on.

> What's your opinion? Should I proceed in adding a custom
> `python-dbus` to the multikernel branch?

Please proceed: at least so that we can verify that this hack
workarounds the bug in various settings.

I'm fine with shipping a patched python-dbus in Tails (obviously, as
long as we report the bug and forward the patch at least to Debian).

Cheers,
-- 
  intrigeri
  | GnuPG key @ https://gaffer.ptitcanardnoir.org/intrigeri/intrigeri.asc
  | OTR fingerprint @ https://gaffer.ptitcanardnoir.org/intrigeri/otr.asc
___
tails-dev mailing list
tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev

Re: [Tails-dev] Shipping a 686-pae kernel

[Ague Mill]

On Thu, Sep 27, 2012 at 11:57:06AM +0200, intrigeri wrote:
> anonym wrote (27 Sep 2012 09:38:32 GMT) :
> > This was in Tails built from feature/multikernel.
> 
> Thank you. I updated the ticket accordingly. So, next step is: somehow
> fix our USB installer vs. the 686-pae kernel. I'm not committing to do
> it any time soon.

After some gdb lifting, I got a stack trace mentioning
`_dbus_watch_invalidate`. This lead me to find this pretty old blog post
 mentioning
that these stack traces were usually because of locking issues. It also
mentions that asking DBus to properly handle multiple threads is as
simple as running `dbus_thread_init_default()`


Unfortunately, while it looks fairly easy to do from Python when using
GLib, according to the answers on this page
,
I have not been able to find a way to easily do something similar for
the Python DBus Qt bindings.

So my hack have been to add a call to `dbus_thread_init_default()` in
the initialization of the Python `dbus` module. And it looks like it
solve the issue. I have not been able to get the installer to crash
after installing the modified `python-dbus` package. From what I can
read from the documentation, except a performance hit, there should be
no other downsides to it.

The binary `python-dbus` package is fairly small (226 kB) and at this
time, it's a patch I feel we can carry on.

It's far from ideal, but it looks like there is very few users of the
Python Qt DBus binding. And our installer is hackish already... which
unfortunately tend to lead to more hacks. :(

What's your opinion? Should I proceed in adding a custom `python-dbus`
to the multikernel branch?

-- 
Ague


pgpiUejoin6pE.pgp
Description: PGP signature
___
tails-dev mailing list
tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev

Re: [Tails-dev] Shipping a 686-pae kernel

[intrigeri]

anonym wrote (27 Sep 2012 09:38:32 GMT) :
> This was in Tails built from feature/multikernel.

Thank you. I updated the ticket accordingly. So, next step is: somehow
fix our USB installer vs. the 686-pae kernel. I'm not committing to do
it any time soon.
___
tails-dev mailing list
tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev

Re: [Tails-dev] Shipping a 686-pae kernel

[anonym]

12/09/12 11:05, intrigeri wrote:
> hi,
> 
> anonym wrote (06 Sep 2012 18:27:37 GMT) :
>> The crash occurs on my (64-bit) laptop. :/
> 
> Do you mean, on your own system (if so: both kernel / userspace
> 64-bits? Debian Wheezy?) or within Tails feature/multikernel with the
> 686-pae kernel?

This was in Tails built from feature/multikernel.



signature.asc
Description: OpenPGP digital signature
___
tails-dev mailing list
tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev

Re: [Tails-dev] Shipping a 686-pae kernel

[alan]

Hi,

> From: intrigeri 
> Date: Wed, 12 Sep 2012 11:05:56 +0200
> 
[...]
> 
> anonym wrote (06 Sep 2012 18:27:37 GMT) :
> > The crash occurs on my (64-bit) laptop. :/
> 
> Do you mean, on your own system (if so: both kernel / userspace
> 64-bits? Debian Wheezy?) or within Tails feature/multikernel with the
> 686-pae kernel?

I can't find an answer to this message. Was it answered somewhere else?

By the way, everybody please consider testing this feature on various
hardware, especially to see if "liveusb-creator crashes on amd64 kernel"
affects systems running the 686-pae kernel.

Cheers


-- 


pgpI8Tko5WcMG.pgp
Description: PGP signature
___
tails-dev mailing list
tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev

Re: [Tails-dev] Shipping a 686-pae kernel

[intrigeri]

hi,

anonym wrote (06 Sep 2012 18:27:37 GMT) :
> The crash occurs on my (64-bit) laptop. :/

Do you mean, on your own system (if so: both kernel / userspace
64-bits? Debian Wheezy?) or within Tails feature/multikernel with the
686-pae kernel?
___
tails-dev mailing list
tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev

Re: [Tails-dev] Shipping a 686-pae kernel

[intrigeri]

anonym wrote (11 Sep 2012 15:32:01 GMT) :
> 11/09/12 14:14, Ague Mill wrote:
>> This does not feel right. The `ifcpu64.c32` module should be copied
>> from the same syslinux package than the one that is going to be
>> used later on by live-build, not by the one installed in the
>> building system.
> [...]

> Agreed. Fixed in commit 14c168f.

Well done, Ague and anonym!

Cheers,
-- 
  intrigeri
  | GnuPG key @ https://gaffer.ptitcanardnoir.org/intrigeri/intrigeri.asc
  | OTR fingerprint @ https://gaffer.ptitcanardnoir.org/intrigeri/otr.asc
___
tails-dev mailing list
tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev

Re: [Tails-dev] Shipping a 686-pae kernel

[anonym]

11/09/12 14:14, Ague Mill wrote:
> On Tue, Sep 04, 2012 at 02:52:53PM +0200, intrigeri wrote:
>> So, I just merged the feature/multikernel branch into experimental.
> 
> Looks like `20-syslinux_detect_cpu` has an issue. In commit 0194501a7,
> syslinux was added as a system-wide dependency (for Vagrant) to build
> Tails.

This situation would be the same in any build system, not just Vagrant.

> This does not feel right. The `ifcpu64.c32` module should be copied from
> the same syslinux package than the one that is going to be used later
> on by live-build, not by the one installed in the building system.
> 
> I really don't think this is a big issue. This COM32 module is not
> probably going to change much over time. But I'd rather not introduce
> the kind of subtle desync. that might happen using two different
> packages.

Agreed. Fixed in commit 14c168f.

Cheers!



signature.asc
Description: OpenPGP digital signature
___
tails-dev mailing list
tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev

Re: [Tails-dev] Shipping a 686-pae kernel

[Ague Mill]

On Tue, Sep 04, 2012 at 02:52:53PM +0200, intrigeri wrote:
> So, I just merged the feature/multikernel branch into experimental.

Looks like `20-syslinux_detect_cpu` has an issue. In commit 0194501a7,
syslinux was added as a system-wide dependency (for Vagrant) to build
Tails.

This does not feel right. The `ifcpu64.c32` module should be copied from
the same syslinux package than the one that is going to be used later
on by live-build, not by the one installed in the building system.

I really don't think this is a big issue. This COM32 module is not
probably going to change much over time. But I'd rather not introduce
the kind of subtle desync. that might happen using two different
packages.

-- 
Ague


pgpJKOqZPJYmB.pgp
Description: PGP signature
___
tails-dev mailing list
tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev

Re: [Tails-dev] Shipping a 686-pae kernel

[Maxim Kammerer]

On Thu, Sep 6, 2012 at 12:38 AM, anonym  wrote:
> This won't work. Even with PAE, processes cannot have more than 4 GB of
> memory (after all, pointers are still 32 bit = 4 GB).
>
> [...]
>
> It's seems we still have to wait for Ague's wipe_memory GRUB module.

You can stop suffering from the NIH syndrome, and use the current
approach in Liberté (choosing the kexec'ed kernel type according to
the CPU type + memtest), or use the old memwipe approach (sequential
chain of processes with each one clearing a reasonable amount of RAM).

I normally ignore endless discussions about trivial issues on this
mailing list, but I just can't watch someone who for once apparently
actually knows programming waste his time on such a useless endeavor.
In the end, you will go back to kernel-based approach, because too
many users will have issues with something custom like GRUB / MemTest
modules, which may have issues with obscure hardware, not see the same
memory that the kernel sees, will not be portable to non-x86 /
non-BIOS architectures, etc.

-- 
Maxim Kammerer
Liberté Linux: http://dee.su/liberte
___
tails-dev mailing list
tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev

Re: [Tails-dev] Shipping a 686-pae kernel

[anonym]

04/09/12 14:52, intrigeri wrote:
> Hi,
> 
> for various reasons (supporting big amounts of RAM, supporting >1 CPU
> core, NX bit), we want to ship a 686-pae kernel in addition to the
> current (486, non-SMP) one we already have, and make the bootloader
> autodetect the most appropriate kernel depending on what the
> CPU supports.
> 
> Work towards this started a while ago, but tremendous progress was
> made these last days, thanks to a proof-of-concept by Ague that
> allowed me to jump over one of the remaining blockers and bring the
> feature into a state that I think is worth testing.
> 
> So, I just merged the feature/multikernel branch into experimental.
> 
> Quoting https://tails.boum.org/todo/nx_bit/ the next steps are:
> 
> 1. heavily test on bare metal, especially to see if "liveusb-creator
>crashes on amd64 kernel" affects systems running the 686-pae kernel

The crash occurs on my (64-bit) laptop. :/

Running 'liveusb-creator-launcher -v' with PAE kernel, clicking "Clone &
Install", "Create Live USB", "Next" --> segfault

Terminal output:

[...]
[creator:348] ['/sbin/sgdisk', '--print', '/dev/sdc']
[creator:348] ['/sbin/sgdisk', '--print', '/dev/sdc']
[creator:348] ['/sbin/sgdisk', '--info', '1', '/dev/sdc']
*** glibc detected *** /usr/bin/python: malloc(): memory corruption:
*** 0x0998d270 ***
=== Backtrace: =
/lib/libc.so.6(+0x6b19a)[0xb74a319a]
/lib/libc.so.6(+0x6dfb7)[0xb74a5fb7]
/lib/libc.so.6(__libc_malloc+0x5c)[0xb74a7bfc]
/usr/lib/libQtCore.so.4(_Z7qMallocj+0x1d)[0xb6ba460d]
Segmentation fault

I saw other variations on the error, like "double free or
corruption". The segfault always occured at the same time though in my
~5 attempts.

Cheers!



signature.asc
Description: OpenPGP digital signature
___
tails-dev mailing list
tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev

Re: [Tails-dev] Shipping a 686-pae kernel

[intrigeri]

anonym wrote (06 Sep 2012 09:18:03 GMT) :
> I forgot to say that this pushed the amount of RAM required to build
> Tails in-memory to over 6 GiB (at least when using Vagrant) so
> I pushed commit 974805d to bump it to 7 GiB.

I don't use Vagrant (yet?) so I don't care that much, but for those
who have 8GiB, going from 6 to 7 makes a huge difference. Perhaps 6.2
or 6.5 or whatever would be enough?

Also, FWIW my libvirt -based build VM is given 6 GiB, no swap, builds
fine inside a 6GB ramdisk, so perhaps the Vagrant build process could
be optimized a bit?
___
tails-dev mailing list
tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev

Re: [Tails-dev] Shipping a 686-pae kernel

[anonym]

05/09/12 23:38, anonym wrote:
> 04/09/12 14:52, intrigeri:
>> So, I just merged the feature/multikernel branch into experimental.
> 
> Built and so far tested in VirtualBox.

I forgot to say that this pushed the amount of RAM required to build
Tails in-memory to over 6 GiB (at least when using Vagrant) so I pushed
commit 974805d to bump it to 7 GiB.

Cheers!



signature.asc
Description: OpenPGP digital signature
___
tails-dev mailing list
tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev

Re: [Tails-dev] Shipping a 686-pae kernel

[anonym]

04/09/12 14:52, intrigeri:
> Hi,
> 
> for various reasons (supporting big amounts of RAM, supporting >1 CPU
> core, NX bit), we want to ship a 686-pae kernel in addition to the
> current (486, non-SMP) one we already have, and make the bootloader
> autodetect the most appropriate kernel depending on what the
> CPU supports.
> 
> Work towards this started a while ago, but tremendous progress was
> made these last days, thanks to a proof-of-concept by Ague that
> allowed me to jump over one of the remaining blockers and bring the
> feature into a state that I think is worth testing.

Awesome!

> So, I just merged the feature/multikernel branch into experimental.

Built and so far tested in VirtualBox. Seems to work just as expected,
e.g. vmlinuz2 is run, I can see my VM's full 8 GB of RAM and dmesg says
that NX protection is active.

> 2. see if kexec'ing a -686-pae kernel (on hardware that supports it)
>fixes "sdmem does not clear all memory"

This won't work. Even with PAE, processes cannot have more than 4 GB of
memory (after all, pointers are still 32 bit = 4 GB). The actual limit
seems to still be ~2 GB though since I had to run 4 fillram instances in
parallel to fill my VM's 8 GB of RAM (and even them they got OOM-killed
before all memory was used for some reason). Here's the results:

Memory containing the pattern before wipe: 7,805,189,024 bytes
Memory containing the pattern after wipe:  5,626,755,520 bytes

It's seems we still have to wait for Ague's wipe_memory GRUB module.

Cheers!



signature.asc
Description: OpenPGP digital signature
___
tails-dev mailing list
tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev

[Tails-dev] Shipping a 686-pae kernel

[intrigeri]

Hi,

for various reasons (supporting big amounts of RAM, supporting >1 CPU
core, NX bit), we want to ship a 686-pae kernel in addition to the
current (486, non-SMP) one we already have, and make the bootloader
autodetect the most appropriate kernel depending on what the
CPU supports.

Work towards this started a while ago, but tremendous progress was
made these last days, thanks to a proof-of-concept by Ague that
allowed me to jump over one of the remaining blockers and bring the
feature into a state that I think is worth testing.

So, I just merged the feature/multikernel branch into experimental.

Quoting https://tails.boum.org/todo/nx_bit/ the next steps are:

1. heavily test on bare metal, especially to see if "liveusb-creator
   crashes on amd64 kernel" affects systems running the 686-pae kernel
2. see if kexec'ing a -686-pae kernel (on hardware that supports it)
   fixes "sdmem does not clear all memory"

Happy testing!

Cheers,
-- 
  intrigeri
  | GnuPG key @ https://gaffer.ptitcanardnoir.org/intrigeri/intrigeri.asc
  | OTR fingerprint @ https://gaffer.ptitcanardnoir.org/intrigeri/otr.asc
___
tails-dev mailing list
tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev