Problem is to send a larger log into this mailing list :/
Anyway: uname -a
Linux tevva-server 4.8.7-1.el7.elrepo.x86_64 #1 SMP Thu Nov 10
20:47:24 EST 2016 x86_64 x86_64 x86_64 GNU/Linux
cut from messages (bear in mind that this is a single cut with a bit
cut from inside of it to fit it in the email)
Feb 10 00:17:14 server journal: ==>
/var/log/gitlab/gitlab-shell/gitlab-shell.log <==
Feb 10 00:17:30 server journal: 192.168.1.253 - wally_tm
[10/Feb/2017:00:17:29 +0000] "PROPFIND /remote.php/webdav/Pictures
HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
Feb 10 00:18:00 server kernel: BTRFS info (device sdc): found 22 extents
Feb 10 00:18:01 server journal: 192.168.1.253 - wally_tm
[10/Feb/2017:00:17:59 +0000] "PROPFIND /remote.php/webdav/Pictures
HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
Feb 10 00:18:05 server kernel: BTRFS info (device sdc): found 22 extents
Feb 10 00:18:06 server kernel: BTRFS info (device sdc): relocating
block group 12353563131904 flags 65
Feb 10 00:18:06 server journal:
Feb 10 00:18:06 server journal: ==> /var/log/gitlab/sidekiq/current <==
Feb 10 00:18:06 server journal: 2017-02-10_00:18:06.99341
2017-02-10T00:18:06.993Z 382 TID-otrr6ws48 PruneOldEventsWorker
JID-99d3a4fb69be748c8674b5e1 INFO: start
Feb 10 00:18:06 server journal: 2017-02-10_00:18:06.99571
2017-02-10T00:18:06.995Z 382 TID-otrr6wqok INFO: Cron Jobs - add job
with name: prune_old_events_worker
Feb 10 00:18:07 server journal: 2017-02-10_00:18:07.00454
2017-02-10T00:18:07.004Z 382 TID-otrr6ws48 PruneOldEventsWorker
JID-99d3a4fb69be748c8674b5e1 INFO: done: 0.011 sec
Feb 10 00:18:30 server journal: 192.168.1.253 - wally_tm
[10/Feb/2017:00:18:29 +0000] "PROPFIND /remote.php/webdav/Pictures
HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
Feb 10 00:18:43 server kernel: BTRFS info (device sdc): found 32 extents
Feb 10 00:18:48 server kernel: BTRFS info (device sdc): found 32 extents
Feb 10 00:18:49 server kernel: BTRFS info (device sdc): relocating
block group 12349268164608 flags 65
Feb 10 00:19:01 server journal: 192.168.1.253 - wally_tm
[10/Feb/2017:00:19:00 +0000] "PROPFIND /remote.php/webdav/Pictures
HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
Feb 10 00:19:02 server journal: 2017-02-10_00:19:02.51409
2017-02-10T00:19:02.513Z 382 TID-otrr6wqok INFO: Cron Jobs - add job
with name: prune_old_events_worker
Feb 10 00:19:02 server journal: 2017-02-10_00:19:02.51449
2017-02-10T00:19:02.514Z 382 TID-otrspth10 PruneOldEventsWorker
JID-4a162ace334771baf4befbb7 INFO: start
Feb 10 00:19:02 server journal: 2017-02-10_00:19:02.52994
2017-02-10T00:19:02.529Z 382 TID-otrspth10 PruneOldEventsWorker
JID-4a162ace334771baf4befbb7 INFO: done: 0.015 sec
Feb 10 00:19:26 server kernel: BTRFS info (device sdc): found 33 extents
Feb 10 00:19:31 server kernel: BTRFS info (device sdc): found 33 extents
Feb 10 00:19:31 server journal: 192.168.1.253 - wally_tm
[10/Feb/2017:00:19:29 +0000] "PROPFIND /remote.php/webdav/Pictures
HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
Feb 10 00:19:32 server kernel: BTRFS info (device sdc): relocating
block group 12344973197312 flags 65
Feb 10 00:19:51 server kernel: perf: interrupt took too long (2513 >
2500), lowering kernel.perf_event_max_sample_rate to 79000
Feb 10 00:20:00 server journal: 192.168.1.253 - wally_tm
[10/Feb/2017:00:19:59 +0000] "PROPFIND /remote.php/webdav/Pictures
HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
Feb 10 00:20:10 server kernel: BTRFS info (device sdc): found 32 extents
Feb 10 00:20:10 server journal: 2017-02-10_00:20:10.15695
2017-02-10T00:20:10.156Z 382 TID-otrsptg48
RepositoryCheck::BatchWorker JID-a315de601bca406340583585 INFO: start
Feb 10 00:20:10 server journal: 2017-02-10_00:20:10.15968
2017-02-10T00:20:10.159Z 382 TID-otrr6wqok INFO: Cron Jobs - add job
with name: repository_check_worker
Feb 10 00:20:10 server journal: 2017-02-10_00:20:10.17180
2017-02-10T00:20:10.171Z 382 TID-otrsptilo PruneOldEventsWorker
JID-4fa75dc5a3d36957d1034f56 INFO: start
Feb 10 00:20:10 server journal: 2017-02-10_00:20:10.17430
2017-02-10T00:20:10.174Z 382 TID-otrr6wqok INFO: Cron Jobs - add job
with name: prune_old_events_worker
Feb 10 00:20:10 server journal: 2017-02-10_00:20:10.18948
2017-02-10T00:20:10.189Z 382 TID-otrsptilo PruneOldEventsWorker
JID-4fa75dc5a3d36957d1034f56 INFO: done: 0.018 sec
Feb 10 00:20:11 server journal: 2017-02-10_00:20:11.00073
2017-02-10T00:20:11.000Z 382 TID-otrsptg48
RepositoryCheck::BatchWorker JID-a315de601bca406340583585 INFO: done:
0.844 sec
Feb 10 00:20:14 server kernel: BTRFS info (device sdc): found 32 extents
Feb 10 00:20:15 server kernel: BTRFS info (device sdc): relocating
block group 12340678230016 flags 65
Feb 10 00:20:30 server journal: 192.168.1.253 - wally_tm
[10/Feb/2017:00:20:29 +0000] "PROPFIND /remote.php/webdav/Pictures
HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
Feb 10 00:20:41 server kernel: systemd-tmpfile: 127 output lines
suppressed due to ratelimiting
Feb 10 00:20:49 server kernel: BTRFS info (device sdc): found 34 extents
Feb 10 00:20:54 server kernel: BTRFS info (device sdc): found 34 extents
Feb 10 00:20:54 server kernel: BTRFS info (device sdc): relocating
block group 12336383262720 flags 65
Feb 10 00:21:01 server journal: 192.168.1.253 - wally_tm
[10/Feb/2017:00:20:59 +0000] "PROPFIND /remote.php/webdav/Pictures
HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
Feb 10 00:21:04 server journal: 2017-02-10_00:21:04.95882
2017-02-10T00:21:04.958Z 382 TID-otrsptep4 PruneOldEventsWorker
JID-4bfca965ac38fafae6d9f88e INFO: start
Feb 10 00:21:04 server journal: 2017-02-10_00:21:04.96159
2017-02-10T00:21:04.961Z 382 TID-otrr6wqok INFO: Cron Jobs - add job
with name: prune_old_events_worker
Feb 10 00:21:04 server journal: 2017-02-10_00:21:04.97151
2017-02-10T00:21:04.971Z 382 TID-otrsptep4 PruneOldEventsWorker
JID-4bfca965ac38fafae6d9f88e INFO: done: 0.013 sec
Feb 10 00:21:07 server ntpd[3349]: 0.0.0.0 c612 02 freq_set kernel -21.061 PPM
Feb 10 00:21:07 server ntpd[3349]: 0.0.0.0 c615 05 clock_sync
Feb 10 00:21:30 server journal: 192.168.1.253 - wally_tm
[10/Feb/2017:00:21:29 +0000] "PROPFIND /remote.php/webdav/Pictures
HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
Feb 10 00:21:31 server kernel: BTRFS info (device sdc): found 33 extents
Feb 10 00:21:35 server kernel: BTRFS info (device sdc): found 33 extents
Feb 10 00:21:36 server kernel: BTRFS info (device sdc): relocating
block group 12332088295424 flags 65
Feb 10 00:22:00 server journal: 192.168.1.253 - wally_tm
[10/Feb/2017:00:21:59 +0000] "PROPFIND /remote.php/webdav/Pictures
HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
Feb 10 00:22:04 server journal:
Feb 10 00:22:04 server journal: ==> /var/log/gitlab/redis/current <==
Feb 10 00:22:04 server journal: 2017-02-10_00:22:04.03365 378:M 10 Feb
00:22:04.033 * 10 changes in 300 seconds. Saving...
Feb 10 00:22:04 server journal: 2017-02-10_00:22:04.03514 378:M 10 Feb
00:22:04.034 * Background saving started by pid 1480
Feb 10 00:22:04 server journal: 2017-02-10_00:22:04.48684 1480:C 10
Feb 00:22:04.486 * DB saved on disk
Feb 10 00:22:04 server journal: 2017-02-10_00:22:04.48789 1480:C 10
Feb 00:22:04.487 * RDB: 8 MB of memory used by copy-on-write
Feb 10 00:22:04 server journal: 2017-02-10_00:22:04.53761 378:M 10 Feb
00:22:04.537 * Background saving terminated with success
Feb 10 00:22:11 server journal:
Feb 10 00:22:11 server journal: ==> /var/log/gitlab/sidekiq/current <==
Feb 10 00:22:11 server journal: 2017-02-10_00:22:11.58099
2017-02-10T00:22:11.580Z 382 TID-otrsptep4 PruneOldEventsWorker
JID-2b322e11d397491e806fdf9a INFO: start
Feb 10 00:22:11 server journal: 2017-02-10_00:22:11.58323
2017-02-10T00:22:11.583Z 382 TID-otrr6wqok INFO: Cron Jobs - add job
with name: prune_old_events_worker
Feb 10 00:22:11 server journal: 2017-02-10_00:22:11.59561
2017-02-10T00:22:11.595Z 382 TID-otrsptep4 PruneOldEventsWorker
JID-2b322e11d397491e806fdf9a INFO: done: 0.015 sec
Feb 10 00:22:14 server kernel: BTRFS info (device sdc): found 32 extents
Feb 10 00:22:19 server kernel: BTRFS info (device sdc): found 32 extents
Feb 10 00:22:20 server kernel: BTRFS info (device sdc): relocating
block group 12327793328128 flags 65
Feb 10 00:22:31 server journal: 192.168.1.253 - wally_tm
[10/Feb/2017:00:22:29 +0000] "PROPFIND /remote.php/webdav/Pictures
HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
Feb 10 00:22:54 server kernel: BTRFS info (device sdc): found 33 extents
Feb 10 00:22:59 server kernel: BTRFS info (device sdc): found 33 extents
Feb 10 00:22:59 server kernel: BTRFS info (device sdc): relocating
block group 12323498360832 flags 65
Feb 10 00:23:01 server journal: 192.168.1.253 - wally_tm
[10/Feb/2017:00:22:59 +0000] "PROPFIND /remote.php/webdav/Pictures
HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
Feb 10 00:23:05 server journal: 2017-02-10_00:23:05.86133
2017-02-10T00:23:05.861Z 382 TID-otrsptep4 PruneOldEventsWorker
JID-95691f21edd3bd28b116508f INFO: start
Feb 10 00:23:05 server journal: 2017-02-10_00:23:05.86497
2017-02-10T00:23:05.864Z 382 TID-otrr6wqok INFO: Cron Jobs - add job
with name: prune_old_events_worker
Feb 10 00:23:05 server journal: 2017-02-10_00:23:05.87282
2017-02-10T00:23:05.872Z 382 TID-otrsptep4 PruneOldEventsWorker
JID-95691f21edd3bd28b116508f INFO: done: 0.011 sec
Feb 10 00:23:30 server journal: 192.168.1.253 - wally_tm
[10/Feb/2017:00:23:29 +0000] "PROPFIND /remote.php/webdav/Pictures
HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
Feb 10 00:23:34 server kernel: BTRFS info (device sdc): found 32 extents
Feb 10 00:23:39 server kernel: BTRFS info (device sdc): found 32 extents
Feb 10 00:23:40 server kernel: BTRFS info (device sdc): relocating
block group 12319203393536 flags 65
Feb 10 00:24:01 server journal: 192.168.1.253 - wally_tm
[10/Feb/2017:00:23:59 +0000] "PROPFIND /remote.php/webdav/Pictures
HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
Feb 10 00:24:09 server journal: 2017-02-10_00:24:09.25079
2017-02-10T00:24:09.250Z 382 TID-otrr6wsw0 PruneOldEventsWorker
JID-9f8cf009d88fd3ba6b5c41c8 INFO: start
Feb 10 00:24:09 server journal: 2017-02-10_00:24:09.25230
2017-02-10T00:24:09.252Z 382 TID-otrr6wqok INFO: Cron Jobs - add job
with name: prune_old_events_worker
Feb 10 00:24:09 server journal: 2017-02-10_00:24:09.26504
2017-02-10T00:24:09.264Z 382 TID-otrr6wsw0 PruneOldEventsWorker
JID-9f8cf009d88fd3ba6b5c41c8 INFO: done: 0.014 sec
Feb 10 00:24:15 server kernel: BTRFS info (device sdc): found 33 extents
Feb 10 00:24:19 server kernel: BTRFS info (device sdc): found 33 extents
Feb 10 00:24:20 server kernel: BTRFS info (device sdc): relocating
block group 12314908426240 flags 65
Feb 10 00:24:30 server journal: 192.168.1.253 - wally_tm
[10/Feb/2017:00:24:29 +0000] "PROPFIND /remote.php/webdav/Pictures
HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
Feb 10 00:24:54 server kernel: BTRFS info (device sdc): found 32 extents
Feb 10 00:24:59 server kernel: BTRFS info (device sdc): found 32 extents
Feb 10 00:24:59 server kernel: BTRFS info (device sdc): relocating
block group 12310613458944 flags 65
Feb 10 00:25:00 server journal: 192.168.1.253 - wally_tm
[10/Feb/2017:00:24:59 +0000] "PROPFIND /remote.php/webdav/Pictures
HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
Feb 10 00:25:13 server journal: 2017-02-10_00:25:13.43533
2017-02-10T00:25:13.435Z 382 TID-otrsptep4 PruneOldEventsWorker
JID-0bafb305f0549fbfaedd0f21 INFO: start
Feb 10 00:25:13 server journal: 2017-02-10_00:25:13.43790
2017-02-10T00:25:13.437Z 382 TID-otrr6wqok INFO: Cron Jobs - add job
with name: prune_old_events_worker
Feb 10 00:25:13 server journal: 2017-02-10_00:25:13.44842
2017-02-10T00:25:13.448Z 382 TID-otrsptep4 PruneOldEventsWorker
JID-0bafb305f0549fbfaedd0f21 INFO: done: 0.013 sec
Feb 10 00:25:34 server journal: 192.168.1.253 - wally_tm
[10/Feb/2017:00:25:32 +0000] "PROPFIND /remote.php/webdav/Pictures
HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
Feb 10 00:25:34 server kernel: BTRFS info (device sdc): found 37 extents
Feb 10 00:25:39 server kernel: BTRFS info (device sdc): found 37 extents
Feb 10 00:25:40 server kernel: BTRFS info (device sdc): relocating
block group 12306318491648 flags 65
Feb 10 00:26:00 server journal: 192.168.1.253 - wally_tm
[10/Feb/2017:00:25:59 +0000] "PROPFIND /remote.php/webdav/Pictures
HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
Feb 10 00:26:14 server kernel: BTRFS info (device sdc): found 33 extents
Feb 10 00:26:18 server journal: 2017-02-10_00:26:18.73291
2017-02-10T00:26:18.732Z 382 TID-otrr6wry4 PruneOldEventsWorker
JID-444ab4e6f7b42090f23cfca2 INFO: start
Feb 10 00:26:18 server journal: 2017-02-10_00:26:18.73405
2017-02-10T00:26:18.734Z 382 TID-otrr6wqok INFO: Cron Jobs - add job
with name: prune_old_events_worker
Feb 10 00:26:18 server journal: 2017-02-10_00:26:18.74593
2017-02-10T00:26:18.745Z 382 TID-otrr6wry4 PruneOldEventsWorker
JID-444ab4e6f7b42090f23cfca2 INFO: done: 0.013 sec
Feb 10 00:26:18 server kernel: BTRFS info (device sdc): found 33 extents
Feb 10 00:26:19 server kernel: BTRFS info (device sdc): relocating
block group 12302023524352 flags 65
Feb 10 00:26:30 server journal: 192.168.1.253 - wally_tm
[10/Feb/2017:00:26:29 +0000] "PROPFIND /remote.php/webdav/Pictures
HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
Feb 10 00:26:54 server kernel: BTRFS info (device sdc): found 33 extents
Feb 10 00:26:58 server kernel: BTRFS info (device sdc): found 33 extents
Feb 10 00:26:59 server kernel: BTRFS info (device sdc): relocating
block group 12297728557056 flags 65
Feb 10 00:27:05 server journal:
Feb 10 00:27:05 server journal: ==> /var/log/gitlab/redis/current <==
Feb 10 00:27:05 server journal: 2017-02-10_00:27:05.05277 378:M 10 Feb
00:27:05.052 * 10 changes in 300 seconds. Saving...
Feb 10 00:27:05 server journal: 2017-02-10_00:27:05.05399 378:M 10 Feb
00:27:05.053 * Background saving started by pid 1796
Feb 10 00:27:05 server journal: 2017-02-10_00:27:05.75682 1796:C 10
Feb 00:27:05.756 * DB saved on disk
Feb 10 00:27:05 server journal: 2017-02-10_00:27:05.75733 1796:C 10
Feb 00:27:05.757 * RDB: 8 MB of memory used by copy-on-write
Feb 10 00:27:05 server journal: 2017-02-10_00:27:05.85460 378:M 10 Feb
00:27:05.854 * Background saving terminated with success
Feb 10 00:27:08 server journal: 192.168.1.253 - wally_tm
[10/Feb/2017:00:27:06 +0000] "PROPFIND /remote.php/webdav/Pictures
HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
Feb 10 00:27:11 server journal:
Feb 10 00:27:11 server journal: ==> /var/log/gitlab/sidekiq/current <==
Feb 10 00:27:11 server journal: 2017-02-10_00:27:11.01598
2017-02-10T00:27:11.015Z 382 TID-otrsptskk PruneOldEventsWorker
JID-9c0a9bd8da1b04e7fec41b09 INFO: start
Feb 10 00:27:11 server journal: 2017-02-10_00:27:11.01854
2017-02-10T00:27:11.018Z 382 TID-otrr6wqok INFO: Cron Jobs - add job
with name: prune_old_events_worker
Feb 10 00:27:11 server journal: 2017-02-10_00:27:11.02791
2017-02-10T00:27:11.027Z 382 TID-otrsptskk PruneOldEventsWorker
JID-9c0a9bd8da1b04e7fec41b09 INFO: done: 0.012 sec
Feb 10 00:27:30 server journal: 192.168.1.253 - wally_tm
[10/Feb/2017:00:27:29 +0000] "PROPFIND /remote.php/webdav/Pictures
HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
Feb 10 00:27:35 server kernel: BTRFS info (device sdc): found 33 extents
Feb 10 00:27:40 server kernel: BTRFS info (device sdc): found 33 extents
Feb 10 00:27:40 server kernel: BTRFS info (device sdc): relocating
block group 12293433589760 flags 65
Feb 10 00:28:01 server journal: 192.168.1.253 - wally_tm
[10/Feb/2017:00:27:59 +0000] "PROPFIND /remote.php/webdav/Pictures
HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
Feb 10 00:28:02 server journal: 2017-02-10_00:28:02.23927
2017-02-10T00:28:02.239Z 382 TID-otrsptep4 PruneOldEventsWorker
JID-59ab152a8efd6e7386e82f5a INFO: start
Feb 10 00:28:02 server journal: 2017-02-10_00:28:02.24045
2017-02-10T00:28:02.240Z 382 TID-otrr6wqok INFO: Cron Jobs - add job
with name: prune_old_events_worker
Feb 10 00:28:02 server journal: 2017-02-10_00:28:02.25251
2017-02-10T00:28:02.252Z 382 TID-otrsptep4 PruneOldEventsWorker
JID-59ab152a8efd6e7386e82f5a INFO: done: 0.013 sec
Feb 10 00:28:16 server kernel: BTRFS info (device sdc): found 32 extents
Feb 10 00:28:21 server kernel: BTRFS info (device sdc): found 32 extents
Feb 10 00:28:21 server kernel: BTRFS info (device sdc): relocating
block group 12289138622464 flags 65
Feb 10 00:28:32 server journal: 192.168.1.253 - wally_tm
[10/Feb/2017:00:28:29 +0000] "PROPFIND /remote.php/webdav/Pictures
HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
Feb 10 00:28:58 server kernel: BTRFS info (device sdc): found 33 extents
Feb 10 00:29:00 server journal: 192.168.1.253 - wally_tm
[10/Feb/2017:00:28:59 +0000] "PROPFIND /remote.php/webdav/Pictures
HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
Feb 10 00:29:02 server kernel: BTRFS info (device sdc): found 33 extents
Feb 10 00:29:03 server kernel: BTRFS info (device sdc): relocating
block group 12288064880640 flags 68
Feb 10 00:29:10 server journal: 2017-02-10_00:29:10.54068
2017-02-10T00:29:10.540Z 382 TID-otrsptsws PruneOldEventsWorker
JID-3da257e9bdbdb05b1966f9b6 INFO: start
Feb 10 00:29:10 server journal: 2017-02-10_00:29:10.54251
2017-02-10T00:29:10.542Z 382 TID-otrr6wqok INFO: Cron Jobs - add job
with name: prune_old_events_worker
Feb 10 00:29:10 server journal: 2017-02-10_00:29:10.55457
2017-02-10T00:29:10.554Z 382 TID-otrsptsws PruneOldEventsWorker
JID-3da257e9bdbdb05b1966f9b6 INFO: done: 0.014 sec
Feb 10 00:29:11 server kernel: ------------[ cut here ]------------
Feb 10 00:29:11 server kernel: WARNING: CPU: 7 PID: 27932 at
fs/btrfs/extent-tree.c:6945 __btrfs_free_extent.isra.69+0xb62/0xca0
[btrfs]
Feb 10 00:29:11 server kernel: Modules linked in: xt_nat veth
xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat
nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype
iptable_filter nf_nat nf_conntrack ext4 jbd2 mbcache ipmi_devintf
coretemp iTCO_wdt iTCO_vendor_support gpio_ich kvm_intel kvm irqbypass
intel_cstate input_leds pcspkr hpwdt hpilo lpc_ich mfd_core
i7core_edac edac_core ses ioatdma enclosure sg ipmi_si ipmi_msghandler
acpi_power_meter shpchp pcc_cpufreq acpi_cpufreq nfsd auth_rpcgss
nfs_acl lockd grace sunrpc ip_tables btrfs xor raid6_pq sd_mod
crc32c_intel amdkfd amd_iommu_v2 serio_raw radeon drm_kms_helper
syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci libahci
fjes libata mpt3sas raid_class scsi_transport_sas igb ptp pps_core dca
i2c_algo_bit
Feb 10 00:29:11 server kernel: CPU: 7 PID: 27932 Comm: btrfs Tainted:
G I 4.8.7-1.el7.elrepo.x86_64 #1
Feb 10 00:29:11 server kernel: Hardware name: HP ProLiant SE326M1 ,
BIOS R02 12/07/2010
Feb 10 00:29:11 server kernel: 0000000000000286 00000000e89bd9a3
ffff8805c9b67720 ffffffff8135406c
Feb 10 00:29:11 server kernel: 0000000000000000 0000000000000000
ffff8805c9b67760 ffffffff810817b1
Feb 10 00:29:11 server kernel: 00001b21fffffffe ffff8804d6f47850
00000b2d1dfc4000 00000000fffffffe
Feb 10 00:29:11 server kernel: Call Trace:
Feb 10 00:29:11 server kernel: [<ffffffff8135406c>] dump_stack+0x63/0x87
Feb 10 00:29:11 server kernel: [<ffffffff810817b1>] __warn+0xd1/0xf0
Feb 10 00:29:11 server kernel: [<ffffffff810818ed>] warn_slowpath_null+0x1d/0x20
Feb 10 00:29:11 server kernel: [<ffffffffa0467722>]
__btrfs_free_extent.isra.69+0xb62/0xca0 [btrfs]
Feb 10 00:29:11 server kernel: [<ffffffffa046bd61>]
__btrfs_run_delayed_refs.constprop.78+0xa11/0x1250 [btrfs]
Feb 10 00:29:11 server kernel: [<ffffffffa046f46e>]
btrfs_run_delayed_refs+0x8e/0x2c0 [btrfs]
Feb 10 00:29:11 server kernel: [<ffffffffa04abddb>] ?
free_extent_buffer+0x4b/0x90 [btrfs]
Feb 10 00:29:11 server kernel: [<ffffffffa0485079>]
btrfs_should_end_transaction+0x49/0x60 [btrfs]
Feb 10 00:29:11 server kernel: [<ffffffffa046db87>]
btrfs_drop_snapshot+0x3f7/0x7f0 [btrfs]
Feb 10 00:29:11 server kernel: [<ffffffffa04e0a0a>]
merge_reloc_roots+0x10a/0x260 [btrfs]
Feb 10 00:29:11 server kernel: [<ffffffffa04e0f34>]
relocate_block_group+0x3d4/0x670 [btrfs]
Feb 10 00:29:11 server kernel: [<ffffffffa04e137b>]
btrfs_relocate_block_group+0x1ab/0x290 [btrfs]
Feb 10 00:29:11 server kernel: [<ffffffffa04b1df6>]
btrfs_relocate_chunk.isra.40+0x46/0xd0 [btrfs]
Feb 10 00:29:11 server kernel: [<ffffffffa04b369e>]
__btrfs_balance+0x85e/0xc50 [btrfs]
Feb 10 00:29:11 server kernel: [<ffffffff810ac538>] ? try_to_wake_up+0x58/0x3c0
Feb 10 00:29:11 server kernel: [<ffffffffa04b3d4c>]
btrfs_balance+0x2bc/0x630 [btrfs]
Feb 10 00:29:11 server kernel: [<ffffffffa04bfa81>]
btrfs_ioctl_balance+0x3f1/0x540 [btrfs]
Feb 10 00:29:11 server kernel: [<ffffffffa04c3c38>]
btrfs_ioctl+0xab8/0x21e0 [btrfs]
Feb 10 00:29:11 server kernel: [<ffffffff8120d8d5>] ?
mem_cgroup_commit_charge+0x85/0x100
Feb 10 00:29:11 server kernel: [<ffffffff811d17d9>] ?
page_add_new_anon_rmap+0x89/0xc0
Feb 10 00:29:11 server kernel: [<ffffffff8119f5d5>] ?
lru_cache_add_active_or_unevictable+0x35/0xb0
Feb 10 00:29:11 server kernel: [<ffffffff811c6c60>] ?
handle_mm_fault+0xed0/0x1240
Feb 10 00:29:11 server kernel: [<ffffffff811f4407>] ?
kmem_cache_alloc+0xd7/0x1a0
Feb 10 00:29:11 server kernel: [<ffffffff8122ffc7>] do_vfs_ioctl+0xa7/0x5f0
Feb 10 00:29:11 server kernel: [<ffffffff8112d3ff>] ?
__audit_syscall_entry+0xaf/0x100
Feb 10 00:29:11 server kernel: [<ffffffff8100365d>] ?
syscall_trace_enter+0x1dd/0x2c0
Feb 10 00:29:11 server kernel: [<ffffffff81230589>] SyS_ioctl+0x79/0x90
Feb 10 00:29:11 server kernel: [<ffffffff81003a47>] do_syscall_64+0x67/0x160
Feb 10 00:29:11 server kernel: [<ffffffff8173bbe1>]
entry_SYSCALL64_slow_path+0x25/0x25
Feb 10 00:29:11 server kernel: ---[ end trace 9b04f696ec89affd ]---
Feb 10 00:29:11 server kernel: BTRFS info (device sdc): leaf
5224642887680 total ptrs 280 free space 43
Feb 10 00:29:11 server kernel: #011item 0 key (12288253624320 169 0)
itemoff 16250 itemsize 33
Feb 10 00:29:11 server kernel: #011#011extent refs 1 gen 142939 flags 258
Feb 10 00:29:11 server kernel: #011#011shared block backref parent 5224641380352
Feb 10 00:29:11 server kernel: #011item 1 key (12288253640704 169 0)
itemoff 16217 itemsize 33
Feb 10 00:29:11 server kernel: #011#011extent refs 1 gen 142939 flags 258
Feb 10 00:29:11 server kernel: #011#011shared block backref parent 5224641380352
Feb 10 00:29:11 server kernel: #011item 2 key (12288253804544 169 0)
itemoff 16184 itemsize 33
Feb 10 00:29:11 server kernel: #011#011extent refs 1 gen 142939 flags 258
Feb 10 00:29:11 server kernel: #011#011shared block backref parent 5224641380352
Feb 10 00:29:11 server kernel: #011item 3 key (12288253919232 169 0)
itemoff 16151 itemsize 33
Feb 10 00:29:11 server kernel: #011#011extent refs 1 gen 142939 flags 258
Feb 10 00:29:11 server kernel: #011#011shared block backref parent 5224641380352
Feb 10 00:29:11 server kernel: #011item 4 key (12288253935616 169 0)
itemoff 16118 itemsize 33
Feb 10 00:29:11 server kernel: #011#011extent refs 1 gen 142939 flags 258
Feb 10 00:29:11 server kernel: #011#011shared block backref parent 5224641380352
Feb 10 00:29:11 server kernel: #011item 5 key (12288253952000 169 0)
itemoff 16085 itemsize 33
Feb 10 00:29:11 server kernel: #011#011extent refs 1 gen 142939 flags 258
Feb 10 00:29:11 server kernel: #011#011shared block backref parent 5224641380352
Feb 10 00:29:11 server kernel: #011item 6 key (12288254050304 169 0)
itemoff 16052 itemsize 33
Feb 10 00:29:11 server kernel: #011#011extent refs 1 gen 142939 flags 258
Feb 10 00:29:11 server kernel: #011#011shared block backref parent 5224641380352
Feb 10 00:29:11 server kernel: #011item 7 key (12288254115840 169 0)
itemoff 16019 itemsize 33
Feb 10 00:29:11 server kernel: #011#011extent refs 1 gen 142939 flags 258
Feb 10 00:29:11 server kernel: #011#011shared block backref parent 5224641380352
Feb 10 00:29:11 server kernel: #011item 8 key (12288254361600 169 0)
itemoff 15986 itemsize 33
Feb 10 00:29:11 server kernel: #011#011extent refs 1 gen 142939 flags 258
Feb 10 00:29:11 server kernel: #011#011shared block backref parent 5224641380352
Feb 10 00:29:11 server kernel: #011item 9 key (12288254640128 169 0)
itemoff 15953 itemsize 33
Feb 10 00:29:11 server kernel: #011#011extent refs 1 gen 142939 flags 258
Feb 10 00:29:11 server kernel: #011#011shared block backref parent 5224641380352
Feb 10 00:29:11 server kernel: #011item 10 key (12288255098880 169 0)
itemoff 15920 itemsize 33
Feb 10 00:29:11 server kernel: #011#011extent refs 1 gen 142939 flags 258
Feb 10 00:29:11 server kernel: #011#011shared block backref parent 5224641380352
Feb 10 00:29:11 server kernel: #011item 11 key (12288258097152 169 0)
itemoff 15887 itemsize 33
Feb 10 00:29:11 server kernel: #011#011extent refs 1 gen 142940 flags 258
Feb 10 00:29:11 server kernel: #011#011shared block backref parent 5224641380352
Feb 10 00:29:11 server kernel: #011item 12 key (12288258162688 169 0)
itemoff 15854 itemsize 33
Feb 10 00:29:11 server kernel: #011#011extent refs 1 gen 142940 flags 258
Feb 10 00:29:11 server kernel: #011#011shared block backref parent 5224641380352
Feb 10 00:29:11 server kernel: #011item 13 key (12288258179072 169 0)
itemoff 15821 itemsize 33
Feb 10 00:29:11 server kernel: #011#011extent refs 1 gen 142940 flags 258
Feb 10 00:29:11 server kernel: #011#011shared block backref parent 5224641380352
Feb 10 00:29:11 server kernel: #011item 14 key (12288258375680 169 0)
itemoff 15788 itemsize 33
Feb 10 00:29:11 server kernel: #011#011extent refs 1 gen 142940 flags 258
On 13 Feb 2017, at 00:49, Qu Wenruo <quwen...@cn.fujitsu.com> wrote:
At 02/12/2017 09:17 AM, Tomasz Kusmierz wrote:
Hi all,
So my main storage filesystem got some sort of veird corruption (that
I can gather). Everything seems to work OK, but when I try to create a
snapshot or run balance (no filters) it will get remounted read only.
Kernel version please.
Fun part is that balance seems to be running even on read only FS, and
I continuously get kernel traces in /var/log/messages .... so it might
as well in the back ground silently eat my data away :/
Kernel backtrace please.
It would be better if you could paste the *first* kernel backtrace, as
that could be the cause, and following kernel backtrace is just
warning from btrfs_abort_transaction() without meaningful output.
I just see some normal messages, but no kernel backtrace.
UPDATE:
Yeah, after rebooting the system it does not even mount the FS,
mount.btrfs sits in some sort of spinlock and consumes 100% of singe
core.
UPDATE 2:
System is completelly cooked :/
[root@server ~]# btrfs fi show
Label: 'rockstor_server' uuid: 5581a647-40ef-4a7a-9d73-847bf35a142b
Total devices 1 FS bytes used 5.72GiB
devid 1 size 53.17GiB used 7.03GiB path /dev/sda2
Label: 'broken_pool' uuid: 26095277-a234-455b-8c97-8dac8ad934c8
Total devices 2 FS bytes used 193.52GiB
devid 1 size 1.82TiB used 196.03GiB path /dev/sdb
devid 2 size 1.82TiB used 196.03GiB path /dev/sdi
Label: 'main_pool' uuid: 0576d577-8954-4a60-a02b-9492b3c29318
Total devices 8 FS bytes used 5.83TiB
devid 1 size 1.82TiB used 1.50TiB path /dev/sdc
devid 2 size 1.82TiB used 1.50TiB path /dev/sdd
devid 3 size 1.82TiB used 1.50TiB path /dev/sde
devid 4 size 1.82TiB used 1.50TiB path /dev/sdf
devid 5 size 1.82TiB used 1.50TiB path /dev/sdg
devid 6 size 1.82TiB used 1.50TiB path /dev/sdh
devid 7 size 1.82TiB used 1.50TiB path /dev/sdj
devid 8 size 1.82TiB used 1.50TiB path /dev/sdk
[root@server ~]# mount /dev/sdc /mnt2/main_pool/
mount: wrong fs type, bad option, bad superblock on /dev/sdc,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so.
[root@server ~]# mount /dev/sdd /mnt2/main_pool/
mount: wrong fs type, bad option, bad superblock on /dev/sdd,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so.
[root@server ~]# mount /dev/sde /mnt2/main_pool/
mount: wrong fs type, bad option, bad superblock on /dev/sde,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so.
dmesg tail retuns:
[ 9507.835629] systemd-udevd[1873]: Validate module index
[ 9507.835656] systemd-udevd[1873]: Check if link configuration needs reloading.
[ 9507.835690] systemd-udevd[1873]: seq 3698 queued, 'add' 'bdi'
[ 9507.835873] systemd-udevd[1873]: seq 3698 forked new worker [13858]
[ 9507.836202] BTRFS info (device sdd): disk space caching is enabled
[ 9507.836204] BTRFS info (device sdd): has skinny extents
[ 9507.836322] systemd-udevd[13858]: seq 3698 running
[ 9507.836443] systemd-udevd[13858]: no db file to read
/run/udev/data/+bdi:btrfs-4: No such file or directory
[ 9507.836474] systemd-udevd[13858]: RUN '/bin/mknod
/dev/btrfs-control c 10 234' /etc/udev/rules.d/64-btrfs.rules:1
[ 9507.837366] systemd-udevd[13861]: starting '/bin/mknod
/dev/btrfs-control c 10 234'
[ 9507.837833] BTRFS error (device sdd): failed to read the system array: -5
[ 9507.838231] systemd-udevd[13858]: '/bin/mknod /dev/btrfs-control c
10 234'(err) '/bin/mknod: '/dev/btrfs-control': File exists'
[ 9507.838262] systemd-udevd[13858]: '/bin/mknod /dev/btrfs-control c
10 234' [13861] exit with return code 1
[ 9507.854757] BTRFS: open_ctree failed
[ 9511.370878] BTRFS info (device sdd): disk space caching is enabled
[ 9511.370881] BTRFS info (device sdd): has skinny extents
[ 9511.375097] BTRFS error (device sdd): failed to read the system array: -5
Btrfs failed to read system chunk array from super block.
Normally this means your primary super block is cooked up.
There may still be chance to recover your fs using backup superblocks.
Please paste the output of "btrfs inspect-internal dump-super -af <your fs>"
Thanks,
Qu
[ 9511.392792] BTRFS: open_ctree failed
[ 9514.233627] BTRFS: device label main_pool devid 3 transid 150680 /dev/sde
[ 9514.234399] systemd-udevd[1873]: Validate module index
[ 9514.234431] systemd-udevd[1873]: Check if link configuration needs reloading.
[ 9514.234465] systemd-udevd[1873]: seq 3702 queued, 'add' 'bdi'
[ 9514.234522] systemd-udevd[1873]: passed 142 bytes to netlink
monitor 0x5628f65d40d0
[ 9514.234554] systemd-udevd[13882]: seq 3702 running
[ 9514.234780] systemd-udevd[13882]: no db file to read
/run/udev/data/+bdi:btrfs-6: No such file or directory
[ 9514.234790] BTRFS info (device sde): disk space caching is enabled
[ 9514.234792] BTRFS info (device sde): has skinny extents
[ 9514.234798] systemd-udevd[13882]: RUN '/bin/mknod
/dev/btrfs-control c 10 234' /etc/udev/rules.d/64-btrfs.rules:1
[ 9514.235181] systemd-udevd[13906]: starting '/bin/mknod
/dev/btrfs-control c 10 234'
[ 9514.236448] systemd-udevd[13882]: '/bin/mknod /dev/btrfs-control c
10 234'(err) '/bin/mknod: '/dev/btrfs-control': File exists'
[ 9514.236514] systemd-udevd[13882]: '/bin/mknod /dev/btrfs-control c
10 234' [13906] exit with return code 1
[ 9514.238726] BTRFS error (device sde): failed to read the system array: -5
[ 9514.255472] BTRFS: open_ctree failed
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
