Hi, I'm trying this out – I can't reproduce the zookeeper failure (on main as of today) but I ran into the same problem on the TSDB test.
I think what's happening is that the test actually takes too long. The test <https://github.com/prometheus/prometheus/blob/eb1617ebd5131c9f5db9c9aafb60d044798eef2c/tsdb/db_test.go#L1269> loops 20 times, and in each loop writes out 20 blocks. I don't know what macOS is doing there, it takes >1s on my (not old, not new) macbook to write one block. I don't know if there is a more efficient way to run this test on macOS (or to speed up tsdb on macOS in general?), but I think in the meantime I think we can avoid hitting the timeout <https://github.com/prometheus/prometheus/pull/10671>. /MR On Fri, Apr 22, 2022 at 8:16 PM Mayur R <[email protected]> wrote: > > *make build succeeds then I did following* > MacBook-Pro-2:prometheus mraleras$ *make test* > >> running all tests > GO111MODULE=on go test -race ./... > ok github.com/prometheus/prometheus/cmd/prometheus 111.481s > ok github.com/prometheus/prometheus/cmd/promtool 130.848s > ok github.com/prometheus/prometheus/config 2.780s > ok github.com/prometheus/prometheus/discovery 5.684s > ? github.com/prometheus/prometheus/discovery/aws [no test files] > ok github.com/prometheus/prometheus/discovery/azure 0.515s > ok github.com/prometheus/prometheus/discovery/consul 3.499s > ok github.com/prometheus/prometheus/discovery/digitalocean 1.938s > ok github.com/prometheus/prometheus/discovery/dns 1.441s > ok github.com/prometheus/prometheus/discovery/eureka 1.135s > ok github.com/prometheus/prometheus/discovery/file 3.167s > ? github.com/prometheus/prometheus/discovery/gce [no test files] > ok github.com/prometheus/prometheus/discovery/hetzner 0.650s > ok github.com/prometheus/prometheus/discovery/http 1.572s > ? github.com/prometheus/prometheus/discovery/install [no test > files] > ok github.com/prometheus/prometheus/discovery/kubernetes 27.412s > ok github.com/prometheus/prometheus/discovery/legacymanager > 3.735s > ok github.com/prometheus/prometheus/discovery/linode 0.618s > ok github.com/prometheus/prometheus/discovery/marathon 0.368s > ok github.com/prometheus/prometheus/discovery/moby 6.517s > ok github.com/prometheus/prometheus/discovery/openstack 0.521s > ok github.com/prometheus/prometheus/discovery/puppetdb 1.199s > ok github.com/prometheus/prometheus/discovery/refresh 0.387s > ok github.com/prometheus/prometheus/discovery/scaleway 1.810s > ok github.com/prometheus/prometheus/discovery/targetgroup 0.261s > ok github.com/prometheus/prometheus/discovery/triton 0.494s > ok github.com/prometheus/prometheus/discovery/uyuni 0.532s > ok github.com/prometheus/prometheus/discovery/xds 2.572s > --- FAIL: TestNewDiscoveryError (0.07s) > zookeeper_test.go:35: expected error, got nil > FAIL > FAIL github.com/prometheus/prometheus/discovery/zookeeper 0.369s > ok > github.com/prometheus/prometheus/documentation/examples/custom-sd/adapter > 0.591s > ? > github.com/prometheus/prometheus/documentation/examples/custom-sd/adapter-usage > [no test files] > ? github.com/prometheus/prometheus/model/exemplar [no test files] > ok github.com/prometheus/prometheus/model/labels 0.370s > ok github.com/prometheus/prometheus/model/relabel 0.251s > ok github.com/prometheus/prometheus/model/rulefmt 1.043s > ok github.com/prometheus/prometheus/model/textparse 0.395s > ? github.com/prometheus/prometheus/model/timestamp [no test > files] > ? github.com/prometheus/prometheus/model/value [no test files] > ok github.com/prometheus/prometheus/notifier 0.955s > ? github.com/prometheus/prometheus/plugins [no test files] > ? github.com/prometheus/prometheus/prompb [no test files] > ok github.com/prometheus/prometheus/promql 29.842s > ok github.com/prometheus/prometheus/promql/parser 8.617s > ok github.com/prometheus/prometheus/rules 44.925s > ok github.com/prometheus/prometheus/scrape 20.093s > ok github.com/prometheus/prometheus/storage 1.526s > ok github.com/prometheus/prometheus/storage/remote 14.953s > ok github.com/prometheus/prometheus/template 0.438s > ok github.com/prometheus/prometheus/tracing 0.516s > level=info msg="Replaying on-disk memory mappable chunks if any" > level=info msg="On-disk memory mappable chunks replay completed" > duration=4.315µs > level=info msg="Replaying WAL, this may take a while" > level=warn msg="Unknown series references" samples=3803 exemplars=0 > level=info msg="WAL segment loaded" segment=0 maxSegment=1 > level=info msg="WAL segment loaded" segment=1 maxSegment=1 > level=info msg="WAL replay completed" checkpoint_replay_duration=235.938µs > wal_replay_duration=2.655315502s total_replay_duration=2.655577365s > level=info msg="Replaying on-disk memory mappable chunks if any" > level=info msg="On-disk memory mappable chunks replay completed" > duration=10.581575ms > level=info msg="Replaying WAL, this may take a while" > level=warn msg="Unknown series references" samples=6473 exemplars=0 > level=info msg="WAL segment loaded" segment=0 maxSegment=1 > level=info msg="WAL segment loaded" segment=1 maxSegment=1 > level=info msg="WAL replay completed" checkpoint_replay_duration=376.876µs > wal_replay_duration=2.77993431s total_replay_duration=2.790944711s > panic: test timed out after 10m0s > > goroutine 9479 [running]: > testing.(*M).startAlarm.func1() > /usr/local/Cellar/go/1.17.8/libexec/src/testing/testing.go:1788 > +0xbb > created by time.goFunc > /usr/local/Cellar/go/1.17.8/libexec/src/time/sleep.go:180 +0x4a > > goroutine 1 [chan receive, 7 minutes]: > testing.(*T).Run(0xc0002d4d00, {0x1a6320c, 0x25}, 0x1a82618) > /usr/local/Cellar/go/1.17.8/libexec/src/testing/testing.go:1307 > +0x752 > testing.runTests.func1(0x0) > /usr/local/Cellar/go/1.17.8/libexec/src/testing/testing.go:1598 > +0x9a > testing.tRunner(0xc0002d4d00, 0xc00033fa60) > /usr/local/Cellar/go/1.17.8/libexec/src/testing/testing.go:1259 > +0x230 > testing.runTests(0xc0002c2280, {0x2a75bc0, 0x96, 0x96}, {0x203000, > 0x19395e2, 0x2a7b700}) > /usr/local/Cellar/go/1.17.8/libexec/src/testing/testing.go:1596 > +0x7cb > testing.(*M).Run(0xc0002c2280) > /usr/local/Cellar/go/1.17.8/libexec/src/testing/testing.go:1504 > +0x9d2 > go.uber.org/goleak.VerifyTestMain({0x25b9cc0 > <http://go.uber.org/goleak.VerifyTestMain(%7B0x25b9cc0>, 0xc0002c2280}, > {0xc0000b5e70, 0x2, 0x2}) > /Users/mraleras/go/pkg/mod/ > go.uber.org/[email protected]/testmain.go:53 +0x5a > github.com/prometheus/prometheus/tsdb.TestMain(0x480) > /Users/mraleras/go/src/ > github.com/prometheus/prometheus/tsdb/db_test.go:63 +0x372 > main.main() > _testmain.go:385 +0x265 > > goroutine 6659 [semacquire]: > sync.runtime_Semacquire(0xc0004e0204) > /usr/local/Cellar/go/1.17.8/libexec/src/runtime/sema.go:56 +0x25 > sync.(*WaitGroup).Wait(0xc0004e0204) > /usr/local/Cellar/go/1.17.8/libexec/src/sync/waitgroup.go:130 +0xea > > github.com/prometheus/prometheus/tsdb.TestTombstoneCleanRetentionLimitsRace(0xc001942b60) > /Users/mraleras/go/src/ > github.com/prometheus/prometheus/tsdb/db_test.go:1310 +0x306 > testing.tRunner(0xc001942b60, 0x1a82618) > /usr/local/Cellar/go/1.17.8/libexec/src/testing/testing.go:1259 > +0x230 > created by testing.(*T).Run > /usr/local/Cellar/go/1.17.8/libexec/src/testing/testing.go:1306 > +0x727 > > goroutine 9047 [select]: > github.com/prometheus/prometheus/tsdb.(*DB).run(0xc0001ea270) > /Users/mraleras/go/src/ > github.com/prometheus/prometheus/tsdb/db.go:811 +0x2ef > created by github.com/prometheus/prometheus/tsdb.open > /Users/mraleras/go/src/ > github.com/prometheus/prometheus/tsdb/db.go:758 +0x1a78 > > goroutine 8935 [select]: > github.com/prometheus/prometheus/tsdb/wal.(*WAL).run(0xc0002ee000) > /Users/mraleras/go/src/ > github.com/prometheus/prometheus/tsdb/wal/wal.go:332 +0x129 > created by github.com/prometheus/prometheus/tsdb/wal.NewSize > /Users/mraleras/go/src/ > github.com/prometheus/prometheus/tsdb/wal/wal.go:301 +0x531 > > goroutine 9455 [syscall]: > syscall.syscall(0x10c9aa0, 0xc000114000, 0x1000000, 0x0) > /usr/local/Cellar/go/1.17.8/libexec/src/runtime/sys_darwin.go:22 > +0x3b > syscall.Open({0xc000066320, 0x9d}, 0x2a7a8a0, 0x0) > > /usr/local/Cellar/go/1.17.8/libexec/src/syscall/zsyscall_darwin_amd64.go:1088 > +0x10e > os.openFileNolog({0xc000066320, 0x9d}, 0x0, 0x0) > /usr/local/Cellar/go/1.17.8/libexec/src/os/file_unix.go:217 +0x125 > os.OpenFile({0xc000066320, 0x9d}, 0x0, 0x400000) > /usr/local/Cellar/go/1.17.8/libexec/src/os/file.go:338 +0x5d > os.Open(...) > /usr/local/Cellar/go/1.17.8/libexec/src/os/file.go:318 > > github.com/prometheus/prometheus/tsdb/fileutil.OpenMmapFileWithSize({0xc000066320 > <http://github.com/prometheus/prometheus/tsdb/fileutil.OpenMmapFileWithSize(%7B0xc000066320>, > 0x9d}, 0x0) > /Users/mraleras/go/src/ > github.com/prometheus/prometheus/tsdb/fileutil/mmap.go:32 +0x7d > github.com/prometheus/prometheus/tsdb/fileutil.OpenMmapFile(...) > /Users/mraleras/go/src/ > github.com/prometheus/prometheus/tsdb/fileutil/mmap.go:28 > > github.com/prometheus/prometheus/tsdb/index.(*Writer).writeLabelIndices(0xc0000ffa20) > /Users/mraleras/go/src/ > github.com/prometheus/prometheus/tsdb/index/index.go:577 +0xe5 > > github.com/prometheus/prometheus/tsdb/index.(*Writer).ensureStage(0xc0000ffa20, > 0x3) > /Users/mraleras/go/src/ > github.com/prometheus/prometheus/tsdb/index/index.go:382 +0x1b6 > github.com/prometheus/prometheus/tsdb/index.(*Writer).Close(0xc0000ffa20) > /Users/mraleras/go/src/ > github.com/prometheus/prometheus/tsdb/index/index.go:1027 +0x3f > > github.com/prometheus/prometheus/tsdb.(*LeveledCompactor).write(0xc001b56360, > {0xc0000262a0, 0x64}, 0xc0001770e0, {0xc000131aa8, 0x1, 0x1}) > /Users/mraleras/go/src/ > github.com/prometheus/prometheus/tsdb/compact.go:613 +0x11c7 > > github.com/prometheus/prometheus/tsdb.(*LeveledCompactor).Write(0xc001b56360, > {0xc0000262a0, 0x64}, {0x25cb178, 0xc0000e72c0}, 0xe, 0xf, 0xc000177050) > /Users/mraleras/go/src/ > github.com/prometheus/prometheus/tsdb/compact.go:504 +0x446 > github.com/prometheus/prometheus/tsdb.(*Block).CleanTombstones(0xc0000e72c0, > {0xc0000262a0, 0x64}, {0x25c2800, 0xc001b56360}) > /Users/mraleras/go/src/ > github.com/prometheus/prometheus/tsdb/block.go:606 +0x29c > github.com/prometheus/prometheus/tsdb.(*DB).CleanTombstones(0xc0001ea270) > /Users/mraleras/go/src/ > github.com/prometheus/prometheus/tsdb/db.go:1684 +0x3bd > > github.com/prometheus/prometheus/tsdb.TestTombstoneCleanRetentionLimitsRace.func2() > /Users/mraleras/go/src/ > github.com/prometheus/prometheus/tsdb/db_test.go:1307 +0xd8 > created by > github.com/prometheus/prometheus/tsdb.TestTombstoneCleanRetentionLimitsRace > /Users/mraleras/go/src/ > github.com/prometheus/prometheus/tsdb/db_test.go:1304 +0x2f9 > FAIL github.com/prometheus/prometheus/tsdb 600.490s > ok github.com/prometheus/prometheus/tsdb/agent 33.503s > ok github.com/prometheus/prometheus/tsdb/chunkenc 0.721s > ok github.com/prometheus/prometheus/tsdb/chunks 36.631s > ? github.com/prometheus/prometheus/tsdb/encoding [no test files] > ? github.com/prometheus/prometheus/tsdb/errors [no test files] > ok github.com/prometheus/prometheus/tsdb/fileutil 0.597s > ok github.com/prometheus/prometheus/tsdb/goversion 0.318s > ok github.com/prometheus/prometheus/tsdb/index 83.811s > ok github.com/prometheus/prometheus/tsdb/record 0.706s > ok github.com/prometheus/prometheus/tsdb/test 0.602s [no tests > to run] > ok github.com/prometheus/prometheus/tsdb/tombstones 0.634s > ok github.com/prometheus/prometheus/tsdb/tsdbutil 0.323s > ok github.com/prometheus/prometheus/tsdb/wal 70.393s > ? github.com/prometheus/prometheus/util/gate [no test files] > ok github.com/prometheus/prometheus/util/httputil 0.504s > ok github.com/prometheus/prometheus/util/logging 0.784s > ? github.com/prometheus/prometheus/util/osutil [no test files] > ok github.com/prometheus/prometheus/util/pool 0.448s > ? github.com/prometheus/prometheus/util/runtime [no test files] > ok github.com/prometheus/prometheus/util/stats 0.747s > ok github.com/prometheus/prometheus/util/strutil 0.846s > ? github.com/prometheus/prometheus/util/teststorage [no test > files] > ? github.com/prometheus/prometheus/util/testutil [no test files] > ? github.com/prometheus/prometheus/util/treecache [no test files] > ok github.com/prometheus/prometheus/web 17.538s > ok github.com/prometheus/prometheus/web/api/v1 4.295s > ? github.com/prometheus/prometheus/web/ui [no test files] > FAIL > make: *** [common-test] Error 1 > MacBook-Pro-2:prometheus mraleras$ > > > *is this expected on macos Monterey 12.2.1, Golang version 1.17.8?I am on > main branch with commit-id fc3f1b8ded810a1263f72d37db9db4e8ce726120* > with no changes of my own on top it. > > > > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-developers/c8f0e38e-495e-4224-be16-5c3d5ac2633fn%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-developers/c8f0e38e-495e-4224-be16-5c3d5ac2633fn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAMV%3D_gY1L_u0tXegvNoB%2BD5EaM0PFm9WiKtR_1fFAwv1JOWoAg%40mail.gmail.com.
