Hi Richard, you were right. I have made another test with wiping out whole sstate-cache for acpid and rerunning the whole reproducible test. It occurs that the package /media/plobacz/9582beb9-8774-4d4d-976f-63a6f1c87c13/poky-contrib/build/sstate-cache/a7/41/sstate:acpid:core2-64-poky-linux:2.0.34:r0:core2-64:11:a741afe7c0bfc218a844a546735ddc4dcaf578bff5d6a3762b1bdedf0819fc9b_package.tar.zst was already containing wrong timestamps.
> I'd guess you need to make the sstate archives of do_package store the > acl and extended attribute data and when that is done, the millisecond > timestamp data will probably be saved too? So I'm guessing that commands which are producing this package_name_package.tar.zst are producing it in gnu format. Question is where can I find it? Is it being produced in this sstate.bbclass or elsewhere? BR Piotr Od: [email protected] <[email protected]> w imieniu użytkownika Piotr Łobacz via lists.openembedded.org <[email protected]> Wysłane: wtorek, 1 sierpnia 2023 15:27 Do: Martin Jansa <[email protected]> DW: Richard Purdie <[email protected]>; Joshua Watt <[email protected]>; Alexandre Belloni <[email protected]>; [email protected] <[email protected]>; Alex Stewart <[email protected]> Temat: ODP: [OE-Core][PATCH v11][master-next 1/5] package_ipk.bbclass: add support for ACLs and xattr Hi Martin, thx for the link, actually, this error: $ diffoscope rootfs.tumbleweed.posix.tar rootfs.ubuntu.posix.tar 016a4c00: 2e2f 7573 722f 6269 6e2f 5061 7848 6561 ./usr/bin/PaxHea -016a4c10: 6465 7273 2f63 6861 7474 722e 6532 6673 ders/chattr.e2fs -016a4c20: 7072 6f67 7300 0000 0000 0000 0000 0000 progs........... +016a4c10: 6465 7273 2e32 322f 6368 6174 7472 2e65 ders.22/chattr.e +016a4c20: 3266 7370 726f 6773 0000 0000 0000 0000 2fsprogs........ ... 03937000: 2e2f 7573 722f 6269 6e2f 5061 7848 6561 ./usr/bin/PaxHea -03937010: 6465 7273 2f63 6f6e 7461 696e 6572 642d ders/containerd- -03937020: 6374 7200 0000 0000 0000 0000 0000 0000 ctr............. +03937010: 6465 7273 2e32 322f 636f 6e74 6169 6e65 ders.22/containe +03937020: 7264 2d63 7472 0000 0000 0000 0000 0000 rd-ctr.......... for posix, is due to missing params "--pax-option=exthdr.name=%d/PaxHeaders/%f,atime:=0,ctime:=0" which can be read in here https://www.gnu.org/software/tar/manual/html_node/PAX-keywords.html in the last paragraph and I do know that this information is well hidden :P And we have currently a talk about that on bug-tar mailing list... Unfortunately the issue is unrelevant, because I already know that... the problem is because as Richard pointed it out correctly: > What is likely happening is that we have two cases: > > a) do_package is run and do_package_write_ipk uses the files directly > from disk which has the millisecond data. do_package writes an sstate > tarball using tar in non-posix mode. > > b) do_package_setscene runs instead of do_package. This restores the > tarball written into sstate in a). The millisecond data is therefore > lost. do_package_write_ipk would then be written without it. > > I'd guess you need to make the sstate archives of do_package store the > acl and extended attribute data and when that is done, the millisecond > timestamp data will probably be saved too? I'm not sure this is going > to be a generally useful thing in the general case though and zeroing > the millisecond portion would really be easier in general if we could. > Certainly, SOURCE_DATE_EPOCH doesn't have millisecond precision. And this is exactly what is happening between reproducibleA and reproducibleB, which we can observe in both directories e.g. acpid: plobacz@latitude:/media/plobacz/9582beb9-8774-4d4d-976f-63a6f1c87c13/poky-contrib$ ls --full-time /media/plobacz/9582beb9-8774-4d4d-976f-63a6f1c87c13/poky-contrib/build-st/reproducibleA/tmp/work/core2-64-poky-linux/acpid/2.0.34-r0/packages-split/acpid-src/usr/src/debug/acpid/2.0.34-r0/ razem 192 -rw-r--r-- 2 plobacz plobacz 10934 2019-07-31 14:21:00.000000000 +0200 acpid.c -rw-r--r-- 2 plobacz plobacz 1670 2018-04-11 01:51:39.000000000 +0200 acpid.h -rw-r--r-- 2 plobacz plobacz 878 2009-04-29 16:37:13.000000000 +0200 acpi_genetlink.h -rw-r--r-- 2 plobacz plobacz 7339 2012-02-16 21:24:42.000000000 +0100 acpi_ids.c -rw-r--r-- 2 plobacz plobacz 1118 2012-02-16 21:24:42.000000000 +0100 acpi_ids.h -rw-r--r-- 2 plobacz plobacz 5264 2015-07-29 00:32:35.000000000 +0200 acpi_listen.c -rw-r--r-- 2 plobacz plobacz 4583 2013-08-16 00:49:18.000000000 +0200 connection_list.c -rw-r--r-- 2 plobacz plobacz 2725 2013-08-15 22:24:07.000000000 +0200 connection_list.h -rw-r--r-- 2 plobacz plobacz 17402 2018-03-29 17:07:23.000000000 +0200 event.c -rw-r--r-- 2 plobacz plobacz 1288 2013-12-19 03:42:46.000000000 +0100 event.h -rw-r--r-- 2 plobacz plobacz 1656 2009-04-29 16:37:13.000000000 +0200 genetlink.h -rw-r--r-- 2 plobacz plobacz 4943 2021-08-29 18:46:00.000000000 +0200 inotify_handler.c -rw-r--r-- 2 plobacz plobacz 1094 2009-11-09 23:03:27.000000000 +0100 inotify_handler.h -rw-r--r-- 2 plobacz plobacz 18544 2022-08-15 16:38:03.000000000 +0200 input_layer.c -rw-r--r-- 2 plobacz plobacz 1257 2009-11-10 00:16:59.000000000 +0100 input_layer.h drwxr-xr-x 2 plobacz plobacz 4096 2022-09-16 03:24:42.000000000 +0200 kacpimon -rw-r--r-- 2 plobacz plobacz 13746 2021-09-20 03:31:56.000000000 +0200 libnetlink.c -rw-r--r-- 2 plobacz plobacz 3028 2021-09-20 03:30:25.000000000 +0200 libnetlink.h -rw-r--r-- 2 plobacz plobacz 1401 2018-03-29 17:08:05.000000000 +0200 log.c -rw-r--r-- 2 plobacz plobacz 1269 2018-03-29 17:08:02.000000000 +0200 log.h -rw-r--r-- 2 plobacz plobacz 6155 2021-09-20 03:32:03.000000000 +0200 netlink.c -rw-r--r-- 2 plobacz plobacz 1066 2009-04-29 16:37:13.000000000 +0200 netlink.h -rw-r--r-- 2 plobacz plobacz 4876 2015-07-29 00:32:35.000000000 +0200 proc.c -rw-r--r-- 2 plobacz plobacz 1066 2012-02-16 21:25:02.000000000 +0100 proc.h -rw-r--r-- 2 plobacz plobacz 5018 2022-09-16 03:24:42.000000000 +0200 sock.c -rw-r--r-- 2 plobacz plobacz 1198 2012-02-16 21:25:02.000000000 +0100 sock.h -rw-r--r-- 2 plobacz plobacz 3007 2016-01-27 17:13:18.000000000 +0100 ud_socket.c -rw-r--r-- 2 plobacz plobacz 352 2012-03-21 00:54:18.000000000 +0100 ud_socket.h plobacz@latitude:/media/plobacz/9582beb9-8774-4d4d-976f-63a6f1c87c13/poky-contrib$ ls --full-time /media/plobacz/9582beb9-8774-4d4d-976f-63a6f1c87c13/poky-contrib/build-st/reproducibleB/tmp/work/core2-64-poky-linux/acpid/2.0.34-r0/packages-split/acpid-src/usr/src/debug/acpid/2.0.34-r0/ razem 192 -rw-r--r-- 3 plobacz plobacz 10934 2019-07-31 14:21:00.861236271 +0200 acpid.c -rw-r--r-- 3 plobacz plobacz 1670 2018-04-11 01:51:39.467499913 +0200 acpid.h -rw-r--r-- 3 plobacz plobacz 878 2009-04-29 16:37:13.000000000 +0200 acpi_genetlink.h -rw-r--r-- 3 plobacz plobacz 7339 2012-02-16 21:24:42.515701450 +0100 acpi_ids.c -rw-r--r-- 3 plobacz plobacz 1118 2012-02-16 21:24:42.544701444 +0100 acpi_ids.h -rw-r--r-- 3 plobacz plobacz 5264 2015-07-29 00:32:35.499948465 +0200 acpi_listen.c -rw-r--r-- 3 plobacz plobacz 4583 2013-08-16 00:49:18.764134072 +0200 connection_list.c -rw-r--r-- 3 plobacz plobacz 2725 2013-08-15 22:24:07.562169686 +0200 connection_list.h -rw-r--r-- 3 plobacz plobacz 17402 2018-03-29 17:07:23.716843024 +0200 event.c -rw-r--r-- 3 plobacz plobacz 1288 2013-12-19 03:42:46.529591162 +0100 event.h -rw-r--r-- 3 plobacz plobacz 1656 2009-04-29 16:37:13.000000000 +0200 genetlink.h -rw-r--r-- 3 plobacz plobacz 4943 2021-08-29 18:46:00.558827438 +0200 inotify_handler.c -rw-r--r-- 3 plobacz plobacz 1094 2009-11-09 23:03:27.000000000 +0100 inotify_handler.h -rw-r--r-- 3 plobacz plobacz 18544 2022-08-15 16:38:03.122942190 +0200 input_layer.c -rw-r--r-- 3 plobacz plobacz 1257 2009-11-10 00:16:59.000000000 +0100 input_layer.h drwxr-xr-x 2 plobacz plobacz 4096 2022-09-16 03:24:42.000000000 +0200 kacpimon -rw-r--r-- 3 plobacz plobacz 13746 2021-09-20 03:31:56.196781952 +0200 libnetlink.c -rw-r--r-- 3 plobacz plobacz 3028 2021-09-20 03:30:25.930632950 +0200 libnetlink.h -rw-r--r-- 3 plobacz plobacz 1401 2018-03-29 17:08:05.842574608 +0200 log.c -rw-r--r-- 3 plobacz plobacz 1269 2018-03-29 17:08:02.566518016 +0200 log.h -rw-r--r-- 3 plobacz plobacz 6155 2021-09-20 03:32:03.418793923 +0200 netlink.c -rw-r--r-- 3 plobacz plobacz 1066 2009-04-29 16:37:13.000000000 +0200 netlink.h -rw-r--r-- 3 plobacz plobacz 4876 2015-07-29 00:32:35.453948235 +0200 proc.c -rw-r--r-- 3 plobacz plobacz 1066 2012-02-16 21:25:02.924701352 +0100 proc.h -rw-r--r-- 3 plobacz plobacz 5018 2022-09-16 03:24:42.000000000 +0200 sock.c -rw-r--r-- 3 plobacz plobacz 1198 2012-02-16 21:25:02.925701357 +0100 sock.h -rw-r--r-- 3 plobacz plobacz 3007 2016-01-27 17:13:18.836936981 +0100 ud_socket.c -rw-r--r-- 3 plobacz plobacz 352 2012-03-21 00:54:18.929518037 +0100 ud_socket.h plobacz@latitude:/media/plobacz/9582beb9-8774-4d4d-976f-63a6f1c87c13/poky-contrib$ And you see that for some reason for reproducibleA which uses sstate-cache miliseconds are lost, due to exactly what Richard has written. Now I need to check one more thing how and were is that exactly happening. BR Piotr Od: Martin Jansa <[email protected]> Wysłane: wtorek, 1 sierpnia 2023 15:04 Do: Piotr Łobacz <[email protected]> DW: Richard Purdie <[email protected]>; Joshua Watt <[email protected]>; Alexandre Belloni <[email protected]>; [email protected] <[email protected]>; Alex Stewart <[email protected]> Temat: Re: [OE-Core][PATCH v11][master-next 1/5] package_ipk.bbclass: add support for ACLs and xattr Just FYI: Not sure if it's still relevant, but last time I was comparing some posix archives from different host OSes there were some differences other than timestamps I believe, this was 2.5 years ago showing differences between tar 1.29 on ubuntu 18.04 and 1.33 on tumbleweed: https://git.openembedded.org/openembedded-core/commit/?id=3ecea58f2a3382d9f4b410d6ad7089111334cb6f Cheers, On Mon, Jul 31, 2023 at 11:28 PM Piotr Łobacz <[email protected]> wrote: > I'd guess you need to make the sstate archives of do_package store the > acl and extended attribute data and when that is done, the millisecond > timestamp data will probably be saved too? I'm not sure this is going > to be a generally useful thing in the general case though and zeroing > the millisecond portion would really be easier in general if we could. > Certainly, SOURCE_DATE_EPOCH doesn't have millisecond precision. This is what we were talking about on tar mailing list, to remove this precision for posix format. Additionally in case of acl and extended attribute data they need to be also added to this sstate archives, which still means that it needs to be extended with posix format. BR Piotr
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#185250): https://lists.openembedded.org/g/openembedded-core/message/185250 Mute This Topic: https://lists.openembedded.org/mt/100485786/21656 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
