Hi all, On 17-06-15 21:19:12, Georg Faerber wrote: > I really would like to make the build of schleuder, a gpg enabled > mailing list, reproducible. However, I'm a bit lost on my own, that's > why I'm searching for input with this mail:
tl;dr: Due to recent changes and fixes upstream, schleuder now does build reproducible. For details, please read on. -- Well, it took me quite a while and some time, but it was worth it, it seems [1]. Here's a short summary of the changes: - Sometimes, gpg-agent gets stuck, killing it before and after running each test helps [2]. - Some tests expected specific dates (like 2016-12-06) to be returned by gpg. However, these dates are non deterministic, they depend on locale settings. Quoting a comment in the corresponding upstream merge request: "Oh my... I wondered before why gpg doesn't show timezone-information anywhere but assumed that they just don't store it or hide it because it doesn't add actual information. TIL gpg knows about timezones and also converts dates but doesn't tell about it... So actually in order to compare a key-date I have to remember (or guess) which timezone it was created in, and calulate the time-difference to my local timezone, in order to guess if a off-by-one-day-mismatch is maybe not an actual mismatch? I'm shaking my head." Therefore, we've relaxed the expected output, to not match specific dates, but a specific format, like YYYY-MM-DD. [3] - Some tests check if getting and / or refreshing keys via the keyserver mechanism works. For this to work, a local keyserver is started, which mocks the behaviour of the keyservers on the Internets. In the past, we just started the keyserver before running the test and assumed it to work. Sometimes this failed, because the keyserver took longer to start, leading to failed tests, because it was in fact unreachable. This was solved via introducing a check for the keyserver to be up, before running the test. [4] - I've never encountered the following during the "official" tests run by jenkins, but I've run into this three times while using reprotest. I'm not sure if this is a bug, race or a feature, but during the second build, the build dir was read only. In the past, the database in which schleuder stores list information was just below the build dir, read only in these cases, leading to a hanging test suite. This was solved via introducing erb code parsing in the schleuder config [5], and using this feature via env vars in Debian to move the database dir below /tmp/ [6]. @dkg: It seems, there is still a bug / race in dirmngr, which leads to errors like "can't connect to '127.0.0.1': no IP address for host" and in turn "marking host '127.0.0.1' as dead". See the attached debug log for details, the log was taken on October 1st with dirmrngr out of unstable. I'm happy to debug this further, if needed. That's all for now! Thanks for this initiative and the work all of you're putting into it, highly appreciated! Cheers, Georg [1] https://tests.reproducible-builds.org/debian/history/schleuder.html [2] https://0xacab.org/schleuder/schleuder/commit/8ecd8a4cdc76bc8fcdee54397fd0928d338146c1 [3] https://0xacab.org/schleuder/schleuder/commit/e52b6851b30d2578d2eb6c451425549a630d2ba6 [4] https://0xacab.org/schleuder/schleuder/commit/5acd2df470e420ef54ac1bf1193638ec8a23bbba [5] https://0xacab.org/schleuder/schleuder/commit/d188224ce60f8850001825dc94a7d1fa470342ae [6] https://anonscm.debian.org/cgit/pkg-ruby-extras/schleuder.git/commit/?id=f1cafc243766ecd0ad3ec84d8b1adf26ef2e0c66
2017-10-01 06:16:42 dirmngr[32131] listening on socket '/tmp/schleuder-test/example.org/list62/S.dirmngr' 2017-10-01 06:16:42 dirmngr[32132.0] permanently loaded certificates: 149 2017-10-01 06:16:42 dirmngr[32132.0] runtime cached certificates: 0 2017-10-01 06:16:42 dirmngr[32132.0] trusted certificates: 149 (148,0,0,1) 2017-10-01 06:16:42 dirmngr[32132.0] failed to open cache dir file '/tmp/schleuder-test/example.org/list62/crls.d/DIR.txt': No such file or directory 2017-10-01 06:16:42 dirmngr[32132.0] creating directory '/tmp/schleuder-test/example.org/list62/crls.d' 2017-10-01 06:16:42 dirmngr[32132.0] new cache dir file '/tmp/schleuder-test/example.org/list62/crls.d/DIR.txt' created 2017-10-01 06:16:42 dirmngr[32132.6] handler for fd 6 started 2017-10-01 06:16:42 dirmngr[32132.6] DBG: chan_6 -> # Home: /tmp/schleuder-test/example.org/list62 2017-10-01 06:16:42 dirmngr[32132.6] DBG: chan_6 -> # Config: /tmp/schleuder-test/example.org/list62/dirmngr.conf 2017-10-01 06:16:42 dirmngr[32132.6] DBG: chan_6 -> OK Dirmngr 2.2.1 at your service 2017-10-01 06:16:42 dirmngr[32132.6] connection from process 32128 (0:0) 2017-10-01 06:16:42 dirmngr[32132.6] DBG: chan_6 <- GETINFO version 2017-10-01 06:16:42 dirmngr[32132.6] DBG: chan_6 -> D 2.2.1 2017-10-01 06:16:42 dirmngr[32132.6] DBG: chan_6 -> OK 2017-10-01 06:16:42 dirmngr[32132.6] DBG: chan_6 <- KEYSERVER --clear hkp://127.0.0.1:9999 2017-10-01 06:16:42 dirmngr[32132.6] DBG: chan_6 -> OK 2017-10-01 06:16:42 dirmngr[32132.6] DBG: chan_6 <- KEYSERVER 2017-10-01 06:16:42 dirmngr[32132.6] DBG: chan_6 -> S KEYSERVER hkp://127.0.0.1:9999 2017-10-01 06:16:42 dirmngr[32132.6] DBG: chan_6 -> OK 2017-10-01 06:16:42 dirmngr[32132.6] DBG: chan_6 <- KEYSERVER --clear hkp://127.0.0.1:9999 2017-10-01 06:16:42 dirmngr[32132.6] DBG: chan_6 -> OK 2017-10-01 06:16:42 dirmngr[32132.6] DBG: chan_6 <- KS_GET -- [email protected] 2017-10-01 06:16:43 dirmngr[32132.6] DBG: dns: libdns initialized (tor mode) 2017-10-01 06:16:43 dirmngr[32132.6] DBG: dns: resolve_dns_name(127.0.0.1): Success 2017-10-01 06:16:43 dirmngr[32132.6] DBG: dns: libdns initialized (tor mode) 2017-10-01 06:16:43 dirmngr[32132.6] DBG: dns: resolve_dns_addr(): Success 2017-10-01 06:16:43 dirmngr[32132.6] resolve_dns_addr for '127.0.0.1': 'localhost' [already known] 2017-10-01 06:16:43 dirmngr[32132.6] DBG: dns: resolve_dns_name(127.0.0.1): Success 2017-10-01 06:16:43 dirmngr[32132.6] DBG: dns: resolve_dns_addr(): Success 2017-10-01 06:16:43 dirmngr[32132.6] number of system provided CAs: 148 2017-10-01 06:16:43 dirmngr[32132.6] DBG: http.c:connect_server: trying name='127.0.0.1' port=9999 2017-10-01 06:16:43 dirmngr[32132.6] DBG: dns: resolve_dns_name(127.0.0.1): Success 2017-10-01 06:16:43 dirmngr[32132.6] DBG: http.c:1819:socket_new: object 0x00007f9f403a6b20 for fd 7 created 2017-10-01 06:16:43 dirmngr[32132.6] DBG: http.c:request: 2017-10-01 06:16:43 dirmngr[32132.6] DBG: >> GET /pks/lookup?op=get&options=mr&[email protected]&exact=on HTTP/1.0\r\n 2017-10-01 06:16:43 dirmngr[32132.6] DBG: >> Host: localhost:9999\r\n 2017-10-01 06:16:43 dirmngr[32132.6] DBG: http.c:request-header: 2017-10-01 06:16:43 dirmngr[32132.6] DBG: >> \r\n 2017-10-01 06:16:43 dirmngr[32132.6] DBG: http.c:response: 2017-10-01 06:16:43 dirmngr[32132.6] DBG: >> HTTP/1.1 200 OK\r\n 2017-10-01 06:16:43 dirmngr[32132.6] http.c:RESP: 'Content-Type: text/html;charset=utf-8' 2017-10-01 06:16:43 dirmngr[32132.6] http.c:RESP: 'Content-Length: 1762' 2017-10-01 06:16:43 dirmngr[32132.6] http.c:RESP: 'X-XSS-Protection: 1; mode=block' 2017-10-01 06:16:43 dirmngr[32132.6] http.c:RESP: 'X-Content-Type-Options: nosniff' 2017-10-01 06:16:43 dirmngr[32132.6] http.c:RESP: 'X-Frame-Options: SAMEORIGIN' 2017-10-01 06:16:43 dirmngr[32132.6] http.c:RESP: 'Connection: close' 2017-10-01 06:16:43 dirmngr[32132.6] http.c:RESP: 'Server: thin' 2017-10-01 06:16:43 dirmngr[32132.6] http.c:RESP: '' 2017-10-01 06:16:43 dirmngr[32132.6] DBG: chan_6 -> S SOURCE http://127.0.0.1:9999 2017-10-01 06:16:43 dirmngr[32132.6] DBG: (1762 bytes sent via D lines not shown) 2017-10-01 06:16:43 dirmngr[32132.6] DBG: chan_6 -> OK 2017-10-01 06:16:44 dirmngr[32132.6] DBG: chan_6 <- BYE 2017-10-01 06:16:44 dirmngr[32132.6] DBG: chan_6 -> OK closing connection 2017-10-01 06:16:44 dirmngr[32132.6] handler for fd 6 terminated 2017-10-01 06:16:45 dirmngr[32132.0] socket file has been removed - shutting down 2017-10-01 06:16:45 dirmngr[32132.0] dirmngr (GnuPG) 2.2.1 stopped 2017-10-01 06:16:46 dirmngr[32169] listening on socket '/tmp/schleuder-test/example.org/list63/S.dirmngr' 2017-10-01 06:16:46 dirmngr[32170.0] permanently loaded certificates: 149 2017-10-01 06:16:46 dirmngr[32170.0] runtime cached certificates: 0 2017-10-01 06:16:46 dirmngr[32170.0] trusted certificates: 149 (148,0,0,1) 2017-10-01 06:16:46 dirmngr[32170.0] failed to open cache dir file '/tmp/schleuder-test/example.org/list63/crls.d/DIR.txt': No such file or directory 2017-10-01 06:16:46 dirmngr[32170.0] creating directory '/tmp/schleuder-test/example.org/list63/crls.d' 2017-10-01 06:16:46 dirmngr[32170.0] new cache dir file '/tmp/schleuder-test/example.org/list63/crls.d/DIR.txt' created 2017-10-01 06:16:47 dirmngr[32170.6] handler for fd 6 started 2017-10-01 06:16:47 dirmngr[32170.6] DBG: chan_6 -> # Home: /tmp/schleuder-test/example.org/list63 2017-10-01 06:16:47 dirmngr[32170.6] DBG: chan_6 -> # Config: /tmp/schleuder-test/example.org/list63/dirmngr.conf 2017-10-01 06:16:47 dirmngr[32170.6] DBG: chan_6 -> OK Dirmngr 2.2.1 at your service 2017-10-01 06:16:47 dirmngr[32170.6] connection from process 32166 (0:0) 2017-10-01 06:16:47 dirmngr[32170.6] DBG: chan_6 <- GETINFO version 2017-10-01 06:16:47 dirmngr[32170.6] DBG: chan_6 -> D 2.2.1 2017-10-01 06:16:47 dirmngr[32170.6] DBG: chan_6 -> OK 2017-10-01 06:16:47 dirmngr[32170.6] DBG: chan_6 <- KS_FETCH -- http://127.0.0.1:9999/keys/example.asc 2017-10-01 06:16:47 dirmngr[32170.6] number of system provided CAs: 148 2017-10-01 06:16:47 dirmngr[32170.6] DBG: http.c:connect_server: trying name='127.0.0.1' port=9999 2017-10-01 06:16:47 dirmngr[32170.6] DBG: dns: libdns initialized (tor mode) 2017-10-01 06:16:47 dirmngr[32170.6] DBG: dns: resolve_dns_name(127.0.0.1): Success 2017-10-01 06:16:47 dirmngr[32170.6] DBG: http.c:1819:socket_new: object 0x00007f5c38007c60 for fd 7 created 2017-10-01 06:16:47 dirmngr[32170.6] DBG: http.c:request: 2017-10-01 06:16:47 dirmngr[32170.6] DBG: >> GET /keys/example.asc HTTP/1.0\r\n 2017-10-01 06:16:47 dirmngr[32170.6] DBG: >> Host: 127.0.0.1:9999\r\n 2017-10-01 06:16:47 dirmngr[32170.6] DBG: http.c:request-header: 2017-10-01 06:16:47 dirmngr[32170.6] DBG: >> \r\n 2017-10-01 06:16:47 dirmngr[32170.6] DBG: http.c:response: 2017-10-01 06:16:47 dirmngr[32170.6] DBG: >> HTTP/1.1 200 OK\r\n 2017-10-01 06:16:47 dirmngr[32170.6] http.c:RESP: 'Content-Type: text/html;charset=utf-8' 2017-10-01 06:16:47 dirmngr[32170.6] http.c:RESP: 'Content-Length: 1762' 2017-10-01 06:16:47 dirmngr[32170.6] http.c:RESP: 'X-XSS-Protection: 1; mode=block' 2017-10-01 06:16:47 dirmngr[32170.6] http.c:RESP: 'X-Content-Type-Options: nosniff' 2017-10-01 06:16:47 dirmngr[32170.6] http.c:RESP: 'X-Frame-Options: SAMEORIGIN' 2017-10-01 06:16:47 dirmngr[32170.6] http.c:RESP: 'Connection: close' 2017-10-01 06:16:47 dirmngr[32170.6] http.c:RESP: 'Server: thin' 2017-10-01 06:16:47 dirmngr[32170.6] http.c:RESP: '' 2017-10-01 06:16:47 dirmngr[32170.6] DBG: (1762 bytes sent via D lines not shown) 2017-10-01 06:16:47 dirmngr[32170.6] DBG: chan_6 -> OK 2017-10-01 06:16:48 dirmngr[32170.6] DBG: chan_6 <- BYE 2017-10-01 06:16:48 dirmngr[32170.6] DBG: chan_6 -> OK closing connection 2017-10-01 06:16:48 dirmngr[32170.6] handler for fd 6 terminated 2017-10-01 06:16:48 dirmngr[32170.0] socket file has been removed - shutting down 2017-10-01 06:16:48 dirmngr[32170.0] dirmngr (GnuPG) 2.2.1 stopped 2017-10-01 06:16:49 dirmngr[32207] listening on socket '/tmp/schleuder-test/example.org/list64/S.dirmngr' 2017-10-01 06:16:49 dirmngr[32208.0] permanently loaded certificates: 149 2017-10-01 06:16:49 dirmngr[32208.0] runtime cached certificates: 0 2017-10-01 06:16:49 dirmngr[32208.0] trusted certificates: 149 (148,0,0,1) 2017-10-01 06:16:49 dirmngr[32208.0] failed to open cache dir file '/tmp/schleuder-test/example.org/list64/crls.d/DIR.txt': No such file or directory 2017-10-01 06:16:49 dirmngr[32208.0] creating directory '/tmp/schleuder-test/example.org/list64/crls.d' 2017-10-01 06:16:49 dirmngr[32208.0] new cache dir file '/tmp/schleuder-test/example.org/list64/crls.d/DIR.txt' created 2017-10-01 06:16:50 dirmngr[32208.6] handler for fd 6 started 2017-10-01 06:16:50 dirmngr[32208.6] DBG: chan_6 -> # Home: /tmp/schleuder-test/example.org/list64 2017-10-01 06:16:50 dirmngr[32208.6] DBG: chan_6 -> # Config: /tmp/schleuder-test/example.org/list64/dirmngr.conf 2017-10-01 06:16:50 dirmngr[32208.6] DBG: chan_6 -> OK Dirmngr 2.2.1 at your service 2017-10-01 06:16:50 dirmngr[32208.6] connection from process 32204 (0:0) 2017-10-01 06:16:50 dirmngr[32208.6] DBG: chan_6 <- GETINFO version 2017-10-01 06:16:50 dirmngr[32208.6] DBG: chan_6 -> D 2.2.1 2017-10-01 06:16:50 dirmngr[32208.6] DBG: chan_6 -> OK 2017-10-01 06:16:50 dirmngr[32208.6] DBG: chan_6 <- KEYSERVER --clear hkp://127.0.0.1:9999 2017-10-01 06:16:50 dirmngr[32208.6] DBG: chan_6 -> OK 2017-10-01 06:16:50 dirmngr[32208.6] DBG: chan_6 <- KS_GET -- 0x98769E8A1091F36BD88403ECF71A3F8412D83889 2017-10-01 06:16:50 dirmngr[32208.6] DBG: dns: libdns initialized (tor mode) 2017-10-01 06:16:50 dirmngr[32208.6] DBG: dns: resolve_dns_name(127.0.0.1): Success 2017-10-01 06:16:50 dirmngr[32208.6] DBG: dns: libdns initialized (tor mode) 2017-10-01 06:16:58 dirmngr[32208.6] DBG: dns: resolve_dns_addr(): Connection closed in DNS 2017-10-01 06:16:58 dirmngr[32208.6] resolve_dns_addr failed while checking '127.0.0.1': Connection closed in DNS 2017-10-01 06:16:58 dirmngr[32208.6] DBG: dns: resolve_dns_name(127.0.0.1): Success 2017-10-01 06:17:00 dirmngr[32208.6] DBG: dns: resolve_dns_addr(): Success 2017-10-01 06:17:00 dirmngr[32208.6] number of system provided CAs: 148 2017-10-01 06:17:00 dirmngr[32208.6] DBG: http.c:connect_server: trying name='127.0.0.1' port=9999 2017-10-01 06:17:00 dirmngr[32208.6] DBG: dns: resolve_dns_name(127.0.0.1): Success 2017-10-01 06:17:00 dirmngr[32208.6] can't connect to '127.0.0.1': no IP address for host 2017-10-01 06:17:00 dirmngr[32208.6] error connecting to 'http://127.0.0.1:9999': Unknown host 2017-10-01 06:17:00 dirmngr[32208.6] marking host '127.0.0.1' as dead 2017-10-01 06:17:00 dirmngr[32208.6] DBG: dns: resolve_dns_name(127.0.0.1): Success 2017-10-01 06:17:01 dirmngr[32208.6] DBG: dns: resolve_dns_addr(): Success 2017-10-01 06:17:01 dirmngr[32208.6] host '127.0.0.1' marked as dead 2017-10-01 06:17:01 dirmngr[32208.6] command 'KS_GET' failed: No keyserver available 2017-10-01 06:17:01 dirmngr[32208.6] DBG: chan_6 -> ERR 167772346 No keyserver available <Dirmngr> 2017-10-01 06:17:01 dirmngr[32208.6] DBG: chan_6 <- BYE 2017-10-01 06:17:01 dirmngr[32208.6] DBG: chan_6 -> OK closing connection 2017-10-01 06:17:01 dirmngr[32208.6] handler for fd 6 terminated 2017-10-01 06:17:02 dirmngr[32208.6] handler for fd 6 started 2017-10-01 06:17:02 dirmngr[32208.6] DBG: chan_6 -> # Home: /tmp/schleuder-test/example.org/list64 2017-10-01 06:17:02 dirmngr[32208.6] DBG: chan_6 -> # Config: /tmp/schleuder-test/example.org/list64/dirmngr.conf 2017-10-01 06:17:02 dirmngr[32208.6] DBG: chan_6 -> OK Dirmngr 2.2.1 at your service 2017-10-01 06:17:02 dirmngr[32208.6] connection from process 32243 (0:0) 2017-10-01 06:17:02 dirmngr[32208.6] DBG: chan_6 <- KILLDIRMNGR 2017-10-01 06:17:02 dirmngr[32208.6] DBG: chan_6 -> ERR 167788543 End of file <Dirmngr> - [closing connection] 2017-10-01 06:17:02 dirmngr[32208.0] socket file has been removed - shutting down
signature.asc
Description: Digital signature
_______________________________________________ Reproducible-builds mailing list [email protected] http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
