Re: [osv-dev] Re: Pip packages/using Nix

2020-12-29 Thread Matthew Kenigsberg
I have some current work here: https://github.com/mkenigs/osv-apps/commits/nginx-nix. A few things are pretty hacky but I think that's just because of trying to use Make/module.py/nix all at the same time. The image that creates is significantly larger than the current osv nginx one, but I

Re: [osv-dev] Re: Pip packages/using Nix

2020-12-28 Thread Roman Shaposhnik
So is this getting published some place? ;-) Thanks, Roman. On Mon, Dec 14, 2020 at 9:00 AM Matthew Kenigsberg < matthewkenigsb...@gmail.com> wrote: > Thanks for putting so much effort into figuring that out! Really > appreciate it, and glad to get it working! > > On Wednesday, December 9,

Re: [osv-dev] Re: Pip packages/using Nix

2020-12-14 Thread Matthew Kenigsberg
Thanks for putting so much effort into figuring that out! Really appreciate it, and glad to get it working! On Wednesday, December 9, 2020 at 3:45:30 PM UTC-7 Matthew Kenigsberg wrote: > That worked!!! Had to set -z relro -z lazy > > On Wednesday, December 9, 2020 at 12:30:55 PM UTC-7

Re: [osv-dev] Re: Pip packages/using Nix

2020-12-09 Thread Matthew Kenigsberg
That worked!!! Had to set -z relro -z lazy On Wednesday, December 9, 2020 at 12:30:55 PM UTC-7 jwkoz...@gmail.com wrote: > Hi, > > Thanks for uploading the files. It definitely has helped me figure out the > issue. > > In essence, all the .so files like libzfs.so are built with Full RELRO >

Re: [osv-dev] Re: Pip packages/using Nix

2020-12-09 Thread Waldek Kozaczuk
Hi, Thanks for uploading the files. It definitely has helped me figure out the issue. In essence, all the .so files like libzfs.so are built with Full RELRO (run 'readelf -a libzfso.so | grep BIND_NOW) on Nix OS it looks like. Relatedly, some Linux distributions are setup to make gcc

Re: [osv-dev] Re: Pip packages/using Nix

2020-12-08 Thread Matthew Kenigsberg
Not completely sure where libgcc_s.so.1 is coming from, but I uploaded what I have in /nix/store/vran8acwir59772hj4vscr7zribvp7l5-gcc-9.3.0-lib/lib/libgcc_s.so.1: https://drive.google.com/drive/folders/1rM6g-FrzwFpuHr2wX9-J21DzSjyQXGg2 Get a different error if I comment out core/elf.cc:1429:

Re: [osv-dev] Re: Pip packages/using Nix

2020-12-08 Thread Waldek Kozaczuk
I wonder if we have a bug in core/elf.cc::fix_permissions() or logic around. And we might be making the wrong part of the mapping readable based on GNU_RELRO header. I wonder if you are able to create ZFS image by temporarily commenting out the line 1429 of core/elf.cc: ef->fix_permissions();

Re: [osv-dev] Re: Pip packages/using Nix

2020-12-08 Thread Matthew Kenigsberg
[nix-shell:~/osv]$ readelf -l build/release/libzfs-stripped.so Elf file type is DYN (Shared object file) Entry point 0xc8f0 There are 8 program headers, starting at offset 64 Program Headers: Type Offset VirtAddr PhysAddr FileSiz MemSiz Flags Align LOAD 0x 0x

Re: [osv-dev] Re: Pip packages/using Nix

2020-12-08 Thread Waldek Kozaczuk
Back to why it is failing. Based on what you sent us: .. 0x1000b000 0x10016000 [44.0 kB] flags=fmF perm=r offset=0x path=/libzfs.so 0x10016000 0x10035000 [124.0 kB] flags=fmF perm=rx offset=0xb000 path=/libzfs.so 0x10035000 0x1003f000

Re: [osv-dev] Re: Pip packages/using Nix

2020-12-08 Thread Matthew Kenigsberg
My gdb is not the strongest but if I hbreak on arch_relocate_jump_slot looks like _pathname is /libzfs.so, eventually /zpool.so, and then a single /libzfs.so before continue hangs On Tuesday, December 8, 2020 at 9:11:15 AM UTC-7 Matthew Kenigsberg wrote: > Nix is a package manager, and NixOS

Re: [osv-dev] Re: Pip packages/using Nix

2020-12-08 Thread Matthew Kenigsberg
Nix is a package manager, and NixOS is an operating system built completely around the package manager. So every library is stored somewhere in /nix/store, like for example on Nix there is never anything like /lib64/ld-linux-x86-64.so. It would be /nix/store/.../ld-linux-x86-64.so. I could

Re: [osv-dev] Re: Pip packages/using Nix

2020-12-08 Thread Matthew Kenigsberg
(gdb) connect abort (fmt=fmt@entry=0x40645bf0 "Assertion failed: %s (%s: %s: %d)\n") at runtime.cc:105 105 do {} while (true); (gdb) osv syms manifest.find_file: path=/libvdso.so, found file=libvdso.so /home/matthew/osv/build/release.x64/libvdso.so 0x1000 add symbol table from file

Re: [osv-dev] Re: Pip packages/using Nix

2020-12-08 Thread Waldek Kozaczuk
It would be also nice to understand if we are crashing on the 1st arch_relocate_jump_slot() for libfzs.so or is it a specific JUMP_SLOT that causes this crash? On Tuesday, December 8, 2020 at 10:39:06 AM UTC-5 Waldek Kozaczuk wrote: > After you connect with gdb can you run 'osv mmap' and send

Re: [osv-dev] Re: Pip packages/using Nix

2020-12-08 Thread Waldek Kozaczuk
After you connect with gdb can you run 'osv mmap' and send us the output. Make sure you run 'osv syms' before it and dump backtrace after. Please see https://github.com/cloudius-systems/osv/wiki/Debugging-OSv for any details. BTW can you build and run OSv ZFS image on the host without NIX? As

Re: [osv-dev] Re: Pip packages/using Nix

2020-12-07 Thread Matthew Kenigsberg
(gdb) frame 18 #18 0x4039c95a in elf::object::arch_relocate_jump_slot (this=this@entry=0xa110fa00, sym=..., addr=addr@entry=0x10040ca8, addend=addend@entry=0) at arch/x64/arch-elf.cc:172 172*static_cast(addr) = sym.relocated_addr(); (gdb) print _pathname $14

Re: [osv-dev] Re: Pip packages/using Nix

2020-12-06 Thread Waldek Kozaczuk
It might be easier to simply print '_pathname' value if you switch to the right frame in gdb. It would be nice to confirm that the problem we have is with zpool.so and that might lead to understanding why this crash happens. Maybe the is something wrong with building zpool.so. BTW based on

Re: [osv-dev] Re: Pip packages/using Nix

2020-12-06 Thread Waldek Kozaczuk
Can you run the ROFS image you built? Also as I understand it NIX is a package manager but what Linux distribution are you using? As far as ZFS goes could you enable ELF debugging - change this line: conf-debug_elf=0 To conf-debug_elf=1 In conf/base.mk, delete core/elf.o and force rebuild the

Re: [osv-dev] Re: Pip packages/using Nix

2020-12-05 Thread Matthew Kenigsberg
After forcing it to use the right path for libz.so.1, it's working with rofs, but still having the same issue when using zfs, even after I correct the path for libz. On Saturday, December 5, 2020 at 5:18:37 PM UTC-7 Matthew Kenigsberg wrote: > gcc version 9.3.0 (GCC) > QEMU emulator version

Re: [osv-dev] Re: Pip packages/using Nix

2020-12-05 Thread Matthew Kenigsberg
gcc version 9.3.0 (GCC) QEMU emulator version 5.1.0 Running with fs=rofs I get the error: Traceback (most recent call last): File "/home/matthew/osv/scripts/gen-rofs-img.py", line 369, in main() File "/home/matthew/osv/scripts/gen-rofs-img.py", line 366, in main gen_image(outfile,

Re: [osv-dev] Re: Pip packages/using Nix

2020-12-05 Thread Waldek Kozaczuk
I can not reproduce it on Ubuntu 20.20 neither Fedora 33. Here is the code fragment where it happens: 169 bool object::arch_relocate_jump_slot(symbol_module& sym, void *addr, Elf64_Sxword addend) 170 { 171 if (sym.symbol) { 172 *static_cast(addr) = sym.relocated_addr(); 173

Re: [osv-dev] Re: Pip packages/using Nix

2020-12-05 Thread Waldek Kozaczuk
Which version of GCC and QEMU/KV< are you using? On Saturday, December 5, 2020 at 5:44:12 PM UTC-5 Matthew Kenigsberg wrote: > Struggling to get scripts/build to run on NixOS because I'm getting a page > fault. NixOS does keep shared libraries in nonstandard locations, not sure > if that's

Re: [osv-dev] Re: Pip packages/using Nix

2020-12-05 Thread Matthew Kenigsberg
Struggling to get scripts/build to run on NixOS because I'm getting a page fault. NixOS does keep shared libraries in nonstandard locations, not sure if that's breaking something. More details below, but any ideas? As far as I can tell, the error is caused by tools/mkfs/mkfs.cc:71:

Re: [osv-dev] Re: Pip packages/using Nix

2020-11-28 Thread Matthew Kenigsberg
Hi, I'll send something, might take a bit before I find time to work on it though. Thanks, Matthew On Saturday, November 28, 2020 at 1:11:11 PM UTC-7 Roman Shaposhnik wrote: > On Tue, Nov 24, 2020 at 8:03 AM Waldek Kozaczuk > wrote: > > > > Hey, > > > > Send a patch with a new app that

Re: [osv-dev] Re: Pip packages/using Nix

2020-11-28 Thread Roman Shaposhnik
On Tue, Nov 24, 2020 at 8:03 AM Waldek Kozaczuk wrote: > > Hey, > > Send a patch with a new app that could demonstrate it, please, if you can. I > would like to see it. Sounds like a nice improvement. FWIW: I'd love to see it too -- been meaning to play with Nix and this gives me a perfect

[osv-dev] Re: Pip packages/using Nix

2020-11-24 Thread Waldek Kozaczuk
Hey, Send a patch with a new app that could demonstrate it, please, if you can. I would like to see it. Sounds like a nice improvement. Waldek On Monday, November 23, 2020 at 7:36:49 PM UTC-5 Matthew Kenigsberg wrote: > Hi, > > That definitely helped, thanks for the response! > > Haven't had

[osv-dev] Re: Pip packages/using Nix

2020-11-23 Thread Matthew Kenigsberg
Hi, That definitely helped, thanks for the response! Haven't had time to look at this in depth, so I feel like I'm not qualified to know if my own suggestion is actually helpful. But looking at osv-apps/python-from-host/Makefile, it seems like nix could do things a bit more cleanly. Nix does

[osv-dev] Re: Pip packages/using Nix

2020-11-07 Thread Waldek Kozaczuk
Hi Matthew, I am not familiar with nix and how exactly it would fit. If you look at the osv-apps repo there are many examples of python 2/3 apps. All of those are driven by module.py and optional makefiles to do a job of collecting relevant files to the final OSv image as scripts/build,