It's probably something customizable in musl, since it can run in 32 and 64 bit systems. Probably for emscripten we defined it as 32-bit since memory is 32-bit anyhow. So if you want to change this, just defining it as 64-bit and fixing up the syscalls would be enough.
Do you really use files larger than you can fit in memory all at once, though? :) On Sun, Mar 4, 2018 at 3:05 PM, Sören Balko <[email protected]> wrote: > Thanks, Alon - very helpful! Having unsigned 32bit ints would help, but > not necessarily a lot. We process video files that can occasionally be > huge, especially when dealing with poorly compressed video streams such as > motion JPEGs. The fact that off_t is declared as a 32 bit int (signed or > not) strikes me as odd. Is that a musl limitation? > > > On Monday, 5 March 2018 04:58:55 UTC+10, Alon Zakai wrote: >> >> About the 31 bit issue, there's a chance the issue is that the asm.js FFI >> boundary is treated as signed (an asm.js function returning a 32-bit >> integer will use | 0). In that case, what might be the bug is that when JS >> calls a function returning an unsigned value it should use >>> 0. Another >> possibility is that the loads/stores of that struct value (makeSetValue >> etc.) may need to be marked as unsigned. >> >> On Sun, Mar 4, 2018 at 10:55 AM, Alon Zakai <[email protected]> wrote: >> >>> C_STRUCTS is generated from the headers in gen_struct_info.py, basically >>> by compiling small C programs to see what the offsets are. I believe it >>> does not look at sizes, though (except for __size__ which is computed for >>> the entire struct). The numbers there are the offsets, not the sizes. So >>> st_size is at offset 36. >>> >>> The stat.h says >>> >>> off_t st_size; >>> blksize_t st_blksize; >>> >>> I'm not sure how to easily find the definition of off_t, but looking in >>> the offsets, st_size is 36 and st_blksize which is after it is 40, so the >>> size must be 4. So it's not big enough if you need more then 32 bits, off_t >>> would need to be redefined. (Do you really need more than 32 bits, though?) >>> >>> A separate question is if 32-bit values work - I think you said 31 bits >>> seems to be the limit. That could be due to treating the value as signed >>> somewhere ( | 0 will do that). If 32 unsigned bits are enough for you, >>> finding that bug might be practical. >>> >>> >>> On Sat, Mar 3, 2018 at 12:31 AM, Soeren Balko <[email protected]> wrote: >>> >>>> In struct_info.compiled.json, the "stat" struct is declared like so: >>>> >>>> "stat":{ >>>> >>>> "st_rdev":28, >>>> "st_mtim":{ >>>> "tv_sec":56, >>>> "tv_nsec":60, >>>> "__size__":8 >>>> }, >>>> "st_blocks":44, >>>> "st_atim":{ >>>> "tv_sec":48, >>>> "tv_nsec":52, >>>> "__size__":8 >>>> }, >>>> "st_nlink":16, >>>> "__st_ino_truncated":8, >>>> "st_ctim": { >>>> >>>> "tv_sec":64, >>>> "tv_nsec":68, >>>> "__size__":8 >>>> }, >>>> "st_mode":12, >>>> "st_blksize":40, >>>> "__st_dev_padding":4, >>>> "st_dev":0, >>>> "st_size":36, >>>> "st_gid":24, >>>> "__st_rdev_padding":32, >>>> "st_uid":20, >>>> "st_ino":72, >>>> "__size__":76 >>>> } >>>> >>>> >>>> I assume the properties are the bit widths of the various fields (?). >>>> According to this, st_size is 36 bits, which is enough to cater even for >>>> very large files. >>>> >>>> Can you please confirm, Alon? >>>> >>>> >>>> On Saturday, March 3, 2018 at 6:18:09 PM UTC+10, Soeren Balko wrote: >>>> >>>>> Thanks, Alon! This does indeed seem to be the issue. In >>>>> library_syscall.js, the "st_size" member is considered am i32 (see below). >>>>> I do not yet fully understand how C_STRUCTS is generated. I can see that >>>>> compiler.js receives a JSON object STRUCT_INFO that contains the type >>>>> definitions. Is this generated from the musl headers? >>>>> >>>>> doStat: function(func, path, buf) { >>>>> try { >>>>> var stat = func(path); >>>>> } catch (e) { >>>>> if (e && e.node && PATH.normalize(path) !== PATH.normalize(FS.getPath( >>>>> e.node))) { >>>>> // an error occurred while trying to look up the path; we should just >>>>> report ENOTDIR >>>>> return -ERRNO_CODES.ENOTDIR; >>>>> } >>>>> throw e; >>>>> } >>>>> {{{ makeSetValue('buf', C_STRUCTS.stat.st_dev, 'stat.dev', 'i32') }}}; >>>>> {{{ makeSetValue('buf', C_STRUCTS.stat.__st_dev_padding, '0', 'i32') >>>>> }}}; >>>>> {{{ makeSetValue('buf', C_STRUCTS.stat.__st_ino_truncated, 'stat.ino', >>>>> 'i32') }}}; >>>>> {{{ makeSetValue('buf', C_STRUCTS.stat.st_mode, 'stat.mode', 'i32') >>>>> }}}; >>>>> {{{ makeSetValue('buf', C_STRUCTS.stat.st_nlink, 'stat.nlink', 'i32') >>>>> }}}; >>>>> {{{ makeSetValue('buf', C_STRUCTS.stat.st_uid, 'stat.uid', 'i32') }}}; >>>>> {{{ makeSetValue('buf', C_STRUCTS.stat.st_gid, 'stat.gid', 'i32') }}}; >>>>> {{{ makeSetValue('buf', C_STRUCTS.stat.st_rdev, 'stat.rdev', 'i32') >>>>> }}}; >>>>> {{{ makeSetValue('buf', C_STRUCTS.stat.__st_rdev_padding, '0', 'i32') >>>>> }}}; >>>>> *{{{ makeSetValue('buf', C_STRUCTS.stat.st_size, 'stat.size', 'i32') >>>>> }}};* >>>>> {{{ makeSetValue('buf', C_STRUCTS.stat.st_blksize, '4096', 'i32') }}}; >>>>> {{{ makeSetValue('buf', C_STRUCTS.stat.st_blocks, 'stat.blocks', 'i32') >>>>> }}}; >>>>> {{{ makeSetValue('buf', C_STRUCTS.stat.st_atim.tv_sec, >>>>> '(stat.atime.getTime() >>>>> / 1000)|0', 'i32') }}}; >>>>> {{{ makeSetValue('buf', C_STRUCTS.stat.st_atim.tv_nsec, '0', 'i32') >>>>> }}}; >>>>> {{{ makeSetValue('buf', C_STRUCTS.stat.st_mtim.tv_sec, >>>>> '(stat.mtime.getTime() >>>>> / 1000)|0', 'i32') }}}; >>>>> <td id="LC61" class="blob-code blob-code-inner js-file-line" >>>>> style="text-align: left; box-sizing: border-box; padding-right: 10px; >>>>> padding-lef >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "emscripten-discuss" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >> -- > You received this message because you are subscribed to the Google Groups > "emscripten-discuss" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "emscripten-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
