Is it this bug https://bugzilla.redhat.com/show_bug.cgi?id=1651246
On Thu, Feb 14, 2019 at 11:50 AM Jayme <jay...@gmail.com> wrote: > [2019-02-14 02:20:29.611099] I [login.c:110:gf_auth] 0-auth/login: allowed > user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3 > [2019-02-14 02:20:29.611131] I [MSGID: 115029] > [server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted > client from > CTX_ID:ee716e24-e187-4b57-a371-cab544f41162-GRAPH_ID:0-PID:30671-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 > (version: 5.3) > [2019-02-14 02:20:29.619521] I [MSGID: 115036] > [server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting > connection from > CTX_ID:ee716e24-e187-4b57-a371-cab544f41162-GRAPH_ID:0-PID:30671-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 > [2019-02-14 02:20:29.619867] I [MSGID: 101055] > [client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down > connection > CTX_ID:ee716e24-e187-4b57-a371-cab544f41162-GRAPH_ID:0-PID:30671-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 > The message "E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler" repeated 19988 times between [2019-02-14 02:19:31.377315] and > [2019-02-14 02:21:14.033991] > [2019-02-14 02:21:30.303440] E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler > The message "E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler" repeated 10 times between [2019-02-14 02:21:30.303440] and > [2019-02-14 02:23:20.421140] > [2019-02-14 02:23:33.142281] E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler > The message "E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler" repeated 34 times between [2019-02-14 02:23:33.142281] and > [2019-02-14 02:25:29.115156] > [2019-02-14 02:25:30.326469] E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler > [2019-02-14 02:25:53.973830] I [addr.c:54:compare_addr_and_update] > 0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr = > "10.11.0.222" > [2019-02-14 02:25:53.973896] I [login.c:110:gf_auth] 0-auth/login: allowed > user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3 > [2019-02-14 02:25:53.973928] I [MSGID: 115029] > [server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted > client from > CTX_ID:4a6b8860-8274-4b3b-b400-d66cbfb97349-GRAPH_ID:0-PID:33522-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 > (version: 5.3) > [2019-02-14 02:25:54.627728] I [MSGID: 115036] > [server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting > connection from > CTX_ID:4a6b8860-8274-4b3b-b400-d66cbfb97349-GRAPH_ID:0-PID:33522-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 > [2019-02-14 02:25:54.628149] I [MSGID: 101055] > [client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down > connection > CTX_ID:4a6b8860-8274-4b3b-b400-d66cbfb97349-GRAPH_ID:0-PID:33522-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 > [2019-02-14 02:25:56.396855] I [addr.c:54:compare_addr_and_update] > 0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr = > "10.11.0.220" > [2019-02-14 02:25:56.396926] I [login.c:110:gf_auth] 0-auth/login: allowed > user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3 > [2019-02-14 02:25:56.396957] I [MSGID: 115029] > [server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted > client from > CTX_ID:963c2196-108c-485d-aca6-a236906d2acf-GRAPH_ID:0-PID:33635-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 > (version: 5.3) > [2019-02-14 02:25:56.404566] I [MSGID: 115036] > [server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting > connection from > CTX_ID:963c2196-108c-485d-aca6-a236906d2acf-GRAPH_ID:0-PID:33635-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 > [2019-02-14 02:25:56.404866] I [MSGID: 101055] > [client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down > connection > CTX_ID:963c2196-108c-485d-aca6-a236906d2acf-GRAPH_ID:0-PID:33635-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 > The message "E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler" repeated 25 times between [2019-02-14 02:25:30.326469] and > [2019-02-14 02:27:25.965601] > [2019-02-14 02:28:10.538374] E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler > The message "E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler" repeated 38 times between [2019-02-14 02:28:10.538374] and > [2019-02-14 02:29:22.622679] > [2019-02-14 02:29:48.891040] E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler > [2019-02-14 02:29:56.026002] E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler > [2019-02-14 02:31:22.494824] I [addr.c:54:compare_addr_and_update] > 0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr = > "10.11.0.221" > [2019-02-14 02:31:22.494914] I [login.c:110:gf_auth] 0-auth/login: allowed > user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3 > [2019-02-14 02:31:22.494946] I [MSGID: 115029] > [server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted > client from > CTX_ID:210b837a-1c98-4d88-aab8-5ce895398d81-GRAPH_ID:0-PID:18765-HOST:host1.replaced.domain.comm-PC_NAME:non_prod_b-client-2-RECON_NO:-0 > (version: 5.3) > [2019-02-14 02:31:22.503920] I [MSGID: 115036] > [server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting > connection from > CTX_ID:210b837a-1c98-4d88-aab8-5ce895398d81-GRAPH_ID:0-PID:18765-HOST:host1.replaced.domain.comm-PC_NAME:non_prod_b-client-2-RECON_NO:-0 > [2019-02-14 02:31:22.504205] I [MSGID: 101055] > [client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down > connection > CTX_ID:210b837a-1c98-4d88-aab8-5ce895398d81-GRAPH_ID:0-PID:18765-HOST:host1.replaced.domain.comm-PC_NAME:non_prod_b-client-2-RECON_NO:-0 > The message "E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler" repeated 15301 times between [2019-02-14 02:29:48.891040] and > [2019-02-14 02:31:28.963054] > [2019-02-14 02:31:31.016063] E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler > The message "E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler" repeated 628 times between [2019-02-14 02:31:31.016063] and > [2019-02-14 02:33:27.936726] > [2019-02-14 02:33:35.878655] E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler > The message "E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler" repeated 1776 times between [2019-02-14 02:33:35.878655] and > [2019-02-14 02:35:28.143659] > [2019-02-14 02:35:29.607848] E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler > [2019-02-14 02:36:00.483996] I [addr.c:54:compare_addr_and_update] > 0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr = > "10.11.0.220" > [2019-02-14 02:36:00.484066] I [login.c:110:gf_auth] 0-auth/login: allowed > user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3 > [2019-02-14 02:36:00.484098] I [MSGID: 115029] > [server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted > client from > CTX_ID:9f2cd861-23c9-412b-be2c-8c45658bb155-GRAPH_ID:0-PID:39820-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 > (version: 5.3) > [2019-02-14 02:36:00.779203] I [MSGID: 115036] > [server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting > connection from > CTX_ID:9f2cd861-23c9-412b-be2c-8c45658bb155-GRAPH_ID:0-PID:39820-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 > [2019-02-14 02:36:00.779662] I [MSGID: 101055] > [client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down > connection > CTX_ID:9f2cd861-23c9-412b-be2c-8c45658bb155-GRAPH_ID:0-PID:39820-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 > [2019-02-14 02:36:49.892372] I [addr.c:54:compare_addr_and_update] > 0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr = > "10.11.0.222" > [2019-02-14 02:36:49.892451] I [login.c:110:gf_auth] 0-auth/login: allowed > user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3 > [2019-02-14 02:36:49.892481] I [MSGID: 115029] > [server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted > client from > CTX_ID:91cc19de-30ae-4639-a407-c6fce4682fa4-GRAPH_ID:0-PID:39353-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 > (version: 5.3) > [2019-02-14 02:36:49.900530] I [MSGID: 115036] > [server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting > connection from > CTX_ID:91cc19de-30ae-4639-a407-c6fce4682fa4-GRAPH_ID:0-PID:39353-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 > [2019-02-14 02:36:49.900863] I [MSGID: 101055] > [client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down > connection > CTX_ID:91cc19de-30ae-4639-a407-c6fce4682fa4-GRAPH_ID:0-PID:39353-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 > The message "E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler" repeated 15772 times between [2019-02-14 02:35:29.607848] and > [2019-02-14 02:37:24.862502] > [2019-02-14 02:38:11.704945] E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler > The message "E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler" repeated 23 times between [2019-02-14 02:38:11.704945] and > [2019-02-14 02:39:24.469485] > [2019-02-14 02:39:47.550186] E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler > The message "E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler" repeated 34 times between [2019-02-14 02:39:47.550186] and > [2019-02-14 02:41:28.642587] > [2019-02-14 02:41:29.746302] E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler > [2019-02-14 02:42:15.811205] I [addr.c:54:compare_addr_and_update] > 0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr = > "10.11.0.220" > [2019-02-14 02:42:15.811284] I [login.c:110:gf_auth] 0-auth/login: allowed > user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3 > [2019-02-14 02:42:15.811317] I [MSGID: 115029] > [server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted > client from > CTX_ID:6de46e97-ef84-43c8-a472-4e942471a8f6-GRAPH_ID:0-PID:43752-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 > (version: 5.3) > [2019-02-14 02:42:15.819841] I [MSGID: 115036] > [server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting > connection from > CTX_ID:6de46e97-ef84-43c8-a472-4e942471a8f6-GRAPH_ID:0-PID:43752-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 > [2019-02-14 02:42:15.820156] I [MSGID: 101055] > [client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down > connection > CTX_ID:6de46e97-ef84-43c8-a472-4e942471a8f6-GRAPH_ID:0-PID:43752-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 > The message "E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler" repeated 5147 times between [2019-02-14 02:41:29.746302] and > [2019-02-14 02:43:28.777825] > [2019-02-14 02:43:35.252750] E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler > The message "E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler" repeated 17909 times between [2019-02-14 02:43:35.252750] and > [2019-02-14 02:45:29.331082] > [2019-02-14 02:45:29.497786] E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler > [2019-02-14 02:46:08.243167] I [addr.c:54:compare_addr_and_update] > 0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr = > "10.11.0.222" > [2019-02-14 02:46:08.243230] I [login.c:110:gf_auth] 0-auth/login: allowed > user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3 > [2019-02-14 02:46:08.243259] I [MSGID: 115029] > [server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted > client from > CTX_ID:56fb6f47-3e05-4f07-abdf-cb49213cf5c0-GRAPH_ID:0-PID:44348-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 > (version: 5.3) > [2019-02-14 02:46:09.512844] I [MSGID: 115036] > [server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting > connection from > CTX_ID:56fb6f47-3e05-4f07-abdf-cb49213cf5c0-GRAPH_ID:0-PID:44348-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 > [2019-02-14 02:46:09.513302] I [MSGID: 101055] > [client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down > connection > CTX_ID:56fb6f47-3e05-4f07-abdf-cb49213cf5c0-GRAPH_ID:0-PID:44348-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 > The message "E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler" repeated 2892 times between [2019-02-14 02:45:29.497786] and > [2019-02-14 02:47:23.957871] > [2019-02-14 02:47:30.087716] E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler > [2019-02-14 02:47:42.722682] I [addr.c:54:compare_addr_and_update] > 0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr = > "10.11.0.221" > [2019-02-14 02:47:42.722779] I [login.c:110:gf_auth] 0-auth/login: allowed > user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3 > [2019-02-14 02:47:42.722810] I [MSGID: 115029] > [server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted > client from > CTX_ID:34bb8b68-3483-4744-bd33-7a47cd4214b2-GRAPH_ID:0-PID:27248-HOST:host1.replaced.domain.comm-PC_NAME:non_prod_b-client-2-RECON_NO:-0 > (version: 5.3) > [2019-02-14 02:47:42.731133] I [MSGID: 115036] > [server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting > connection from > CTX_ID:34bb8b68-3483-4744-bd33-7a47cd4214b2-GRAPH_ID:0-PID:27248-HOST:host1.replaced.domain.comm-PC_NAME:non_prod_b-client-2-RECON_NO:-0 > [2019-02-14 02:47:42.731447] I [MSGID: 101055] > [client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down > connection > CTX_ID:34bb8b68-3483-4744-bd33-7a47cd4214b2-GRAPH_ID:0-PID:27248-HOST:host1.replaced.domain.comm-PC_NAME:non_prod_b-client-2-RECON_NO:-0 > The message "E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler" repeated 564 times between [2019-02-14 02:47:30.087716] and > [2019-02-14 02:49:28.480491] > [2019-02-14 02:49:32.129061] E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler > The message "E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler" repeated 15058 times between [2019-02-14 02:49:32.129061] and > [2019-02-14 02:51:26.667738] > [2019-02-14 02:52:35.362594] E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler > [2019-02-14 02:53:08.127443] I [addr.c:54:compare_addr_and_update] > 0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr = > "10.11.0.222" > [2019-02-14 02:53:08.127528] I [login.c:110:gf_auth] 0-auth/login: allowed > user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3 > [2019-02-14 02:53:08.127562] I [MSGID: 115029] > [server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted > client from > CTX_ID:3fbc7931-7ac7-4dfd-9169-6271a18fc0f9-GRAPH_ID:0-PID:48068-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 > (version: 5.3) > [2019-02-14 02:53:08.135384] I [MSGID: 115036] > [server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting > connection from > CTX_ID:3fbc7931-7ac7-4dfd-9169-6271a18fc0f9-GRAPH_ID:0-PID:48068-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 > [2019-02-14 02:53:08.135682] I [MSGID: 101055] > [client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down > connection > CTX_ID:3fbc7931-7ac7-4dfd-9169-6271a18fc0f9-GRAPH_ID:0-PID:48068-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 > The message "E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler" repeated 11 times between [2019-02-14 02:52:35.362594] and > [2019-02-14 02:53:24.927967] > [2019-02-14 02:53:56.793284] E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler > The message "E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler" repeated 38 times between [2019-02-14 02:53:56.793284] and > [2019-02-14 02:55:29.101365] > [2019-02-14 02:55:30.445077] E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler > [2019-02-14 02:56:14.259883] I [addr.c:54:compare_addr_and_update] > 0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr = > "10.11.0.222" > [2019-02-14 02:56:14.259948] I [login.c:110:gf_auth] 0-auth/login: allowed > user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3 > [2019-02-14 02:56:14.259980] I [MSGID: 115029] > [server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted > client from > CTX_ID:d7f3996c-4c68-4e15-b0e2-0197bc358ddd-GRAPH_ID:0-PID:885-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 > (version: 5.3) > [2019-02-14 02:56:14.336000] I [MSGID: 115036] > [server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting > connection from > CTX_ID:d7f3996c-4c68-4e15-b0e2-0197bc358ddd-GRAPH_ID:0-PID:885-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 > [2019-02-14 02:56:14.336327] I [MSGID: 101055] > [client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down > connection > CTX_ID:d7f3996c-4c68-4e15-b0e2-0197bc358ddd-GRAPH_ID:0-PID:885-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 > The message "E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler" repeated 19 times between [2019-02-14 02:55:30.445077] and > [2019-02-14 02:57:28.589205] > [2019-02-14 02:57:35.071899] E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler > [2019-02-14 02:58:33.855724] I [addr.c:54:compare_addr_and_update] > 0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr = > "10.11.0.220" > [2019-02-14 02:58:33.855805] I [login.c:110:gf_auth] 0-auth/login: allowed > user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3 > [2019-02-14 02:58:33.855838] I [MSGID: 115029] > [server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted > client from > CTX_ID:868382a8-7e38-481c-b5a3-80c1bdd21f28-GRAPH_ID:0-PID:4631-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 > (version: 5.3) > [2019-02-14 02:58:33.864814] I [MSGID: 115036] > [server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting > connection from > CTX_ID:868382a8-7e38-481c-b5a3-80c1bdd21f28-GRAPH_ID:0-PID:4631-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 > [2019-02-14 02:58:33.865124] I [MSGID: 101055] > [client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down > connection > CTX_ID:868382a8-7e38-481c-b5a3-80c1bdd21f28-GRAPH_ID:0-PID:4631-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 > The message "E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler" repeated 35 times between [2019-02-14 02:57:35.071899] and > [2019-02-14 02:59:29.398638] > [2019-02-14 02:59:33.501166] E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler > [2019-02-14 03:00:01.622537] E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler > The message "E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler" repeated 21104 times between [2019-02-14 02:59:33.501166] and > [2019-02-14 03:01:29.325701] > [2019-02-14 03:01:30.635130] E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler > The message "E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler" repeated 640 times between [2019-02-14 03:01:30.635130] and > [2019-02-14 03:03:29.020755] > [2019-02-14 03:03:31.238543] E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler > [2019-02-14 03:04:00.722382] I [addr.c:54:compare_addr_and_update] > 0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr = > "10.11.0.221" > [2019-02-14 03:04:00.722466] I [login.c:110:gf_auth] 0-auth/login: allowed > user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3 > [2019-02-14 03:04:00.722499] I [MSGID: 115029] > [server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted > client from > CTX_ID:ca4e5703-7129-4242-9fbf-9b119de9e0eb-GRAPH_ID:0-PID:35692-HOST:host1.replaced.domain.comm-PC_NAME:non_prod_b-client-2-RECON_NO:-0 > (version: 5.3) > > On Thu, Feb 14, 2019 at 11:52 AM Sahina Bose <sab...@redhat.com> wrote: > >> >> >> On Thu, Feb 14, 2019 at 8:24 PM Jayme <jay...@gmail.com> wrote: >> >>> https://bugzilla.redhat.com/show_bug.cgi?id=1677160 doesn't seem >>> relevant to me? Is that the correct link? >>> >>> Like I mentioned in a previous email I'm also having problems with >>> Gluster bricks going offline since upgrading to oVirt 4.3 yesterday >>> (previously I've never had a single issue with gluster nor have had a brick >>> ever go down). I suspect this will continue to happen daily as some other >>> users on this group have suggested. I was able to pull some logs from >>> engine and gluster from around the time the brick dropped. My setup is 3 >>> node HCI and I was previously running the latest 4.2 updates (before >>> upgrading to 4.3). My hardware is has a lot of overhead and I'm on 10Gbe >>> gluster backend (the servers were certainly not under any significant >>> amount of load when the brick went offline). To recover I had to place the >>> host in maintenance mode and reboot (although I suspect I could have simply >>> unmounted and remounted gluster mounts). >>> >> >> Anything in the brick logs..the below logs only indicate that engine >> detected that brick was down. To get to why the brick was marked down, the >> bricks logs would help >> >> >>> grep "2019-02-14" engine.log-20190214 | grep >>> "GLUSTER_BRICK_STATUS_CHANGED" >>> 2019-02-14 02:41:48,018-04 WARN >>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>> (DefaultQuartzScheduler1) [5ff5b093] EVENT_ID: >>> GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick >>> host2.replaced.domain.com:/gluster_bricks/non_prod_b/non_prod_b of >>> volume non_prod_b of cluster Default from UP to DOWN via cli. >>> 2019-02-14 03:20:11,189-04 WARN >>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>> (DefaultQuartzScheduler3) [760f7851] EVENT_ID: >>> GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick >>> host2.replaced.domain.com:/gluster_bricks/engine/engine of volume >>> engine of cluster Default from DOWN to UP via cli. >>> 2019-02-14 03:20:14,819-04 WARN >>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>> (DefaultQuartzScheduler3) [760f7851] EVENT_ID: >>> GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick >>> host2.replaced.domain.com:/gluster_bricks/prod_b/prod_b of volume >>> prod_b of cluster Default from DOWN to UP via cli. >>> 2019-02-14 03:20:19,692-04 WARN >>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>> (DefaultQuartzScheduler3) [760f7851] EVENT_ID: >>> GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick >>> host2.replaced.domain.com:/gluster_bricks/isos/isos of volume isos of >>> cluster Default from DOWN to UP via cli. >>> 2019-02-14 03:20:25,022-04 WARN >>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>> (DefaultQuartzScheduler3) [760f7851] EVENT_ID: >>> GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick >>> host2.replaced.domain.com:/gluster_bricks/prod_a/prod_a of volume >>> prod_a of cluster Default from DOWN to UP via cli. >>> 2019-02-14 03:20:29,088-04 WARN >>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>> (DefaultQuartzScheduler3) [760f7851] EVENT_ID: >>> GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick >>> host2.replaced.domain.com:/gluster_bricks/non_prod_b/non_prod_b of >>> volume non_prod_b of cluster Default from DOWN to UP via cli. >>> 2019-02-14 03:20:34,099-04 WARN >>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>> (DefaultQuartzScheduler3) [760f7851] EVENT_ID: >>> GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick >>> host2.replaced.domain.com:/gluster_bricks/non_prod_a/non_prod_a of >>> volume non_prod_a of cluster Default from DOWN to UP via cli >>> >>> glusterd.log >>> >>> # grep -B20 -A20 "2019-02-14 02:41" glusterd.log >>> [2019-02-14 02:36:49.585034] I [MSGID: 106499] >>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: >>> Received status volume req for volume non_prod_b >>> [2019-02-14 02:36:49.597788] E [MSGID: 101191] >>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch >>> handler >>> The message "E [MSGID: 101191] >>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch >>> handler" repeated 2 times between [2019-02-14 02:36:49.597788] and >>> [2019-02-14 02:36:49.900505] >>> [2019-02-14 02:36:53.437539] I [MSGID: 106499] >>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: >>> Received status volume req for volume non_prod_a >>> [2019-02-14 02:36:53.452816] E [MSGID: 101191] >>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch >>> handler >>> [2019-02-14 02:36:53.864153] I [MSGID: 106499] >>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: >>> Received status volume req for volume non_prod_a >>> [2019-02-14 02:36:53.875835] E [MSGID: 101191] >>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch >>> handler >>> [2019-02-14 02:36:30.958649] I [MSGID: 106499] >>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: >>> Received status volume req for volume engine >>> [2019-02-14 02:36:35.322129] I [MSGID: 106499] >>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: >>> Received status volume req for volume prod_b >>> [2019-02-14 02:36:39.639645] I [MSGID: 106499] >>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: >>> Received status volume req for volume isos >>> [2019-02-14 02:36:45.301275] I [MSGID: 106499] >>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: >>> Received status volume req for volume prod_a >>> The message "E [MSGID: 101191] >>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch >>> handler" repeated 2 times between [2019-02-14 02:36:53.875835] and >>> [2019-02-14 02:36:54.180780] >>> [2019-02-14 02:37:59.193409] E [MSGID: 101191] >>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch >>> handler >>> [2019-02-14 02:38:44.065560] I [MSGID: 106499] >>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: >>> Received status volume req for volume engine >>> [2019-02-14 02:38:44.072680] I [MSGID: 106499] >>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: >>> Received status volume req for volume isos >>> [2019-02-14 02:38:44.077841] I [MSGID: 106499] >>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: >>> Received status volume req for volume non_prod_a >>> [2019-02-14 02:38:44.082798] I [MSGID: 106499] >>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: >>> Received status volume req for volume non_prod_b >>> [2019-02-14 02:38:44.088237] I [MSGID: 106499] >>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: >>> Received status volume req for volume prod_a >>> [2019-02-14 02:38:44.093518] I [MSGID: 106499] >>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: >>> Received status volume req for volume prod_b >>> The message "E [MSGID: 101191] >>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch >>> handler" repeated 2 times between [2019-02-14 02:37:59.193409] and >>> [2019-02-14 02:38:44.100494] >>> [2019-02-14 02:41:58.649683] E [MSGID: 101191] >>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch >>> handler >>> The message "E [MSGID: 101191] >>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch >>> handler" repeated 6 times between [2019-02-14 02:41:58.649683] and >>> [2019-02-14 02:43:00.286999] >>> [2019-02-14 02:43:46.366743] I [MSGID: 106499] >>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: >>> Received status volume req for volume engine >>> [2019-02-14 02:43:46.373587] I [MSGID: 106499] >>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: >>> Received status volume req for volume isos >>> [2019-02-14 02:43:46.378997] I [MSGID: 106499] >>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: >>> Received status volume req for volume non_prod_a >>> [2019-02-14 02:43:46.384324] I [MSGID: 106499] >>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: >>> Received status volume req for volume non_prod_b >>> [2019-02-14 02:43:46.390310] I [MSGID: 106499] >>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: >>> Received status volume req for volume prod_a >>> [2019-02-14 02:43:46.397031] I [MSGID: 106499] >>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: >>> Received status volume req for volume prod_b >>> [2019-02-14 02:43:46.404083] E [MSGID: 101191] >>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch >>> handler >>> [2019-02-14 02:45:47.302884] I [MSGID: 106499] >>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: >>> Received status volume req for volume engine >>> [2019-02-14 02:45:47.309697] I [MSGID: 106499] >>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: >>> Received status volume req for volume isos >>> [2019-02-14 02:45:47.315149] I [MSGID: 106499] >>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: >>> Received status volume req for volume non_prod_a >>> [2019-02-14 02:45:47.320806] I [MSGID: 106499] >>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: >>> Received status volume req for volume non_prod_b >>> [2019-02-14 02:45:47.326865] I [MSGID: 106499] >>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: >>> Received status volume req for volume prod_a >>> [2019-02-14 02:45:47.332192] I [MSGID: 106499] >>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: >>> Received status volume req for volume prod_b >>> [2019-02-14 02:45:47.338991] E [MSGID: 101191] >>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch >>> handler >>> [2019-02-14 02:46:47.789575] I [MSGID: 106499] >>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: >>> Received status volume req for volume non_prod_b >>> [2019-02-14 02:46:47.795276] I [MSGID: 106499] >>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: >>> Received status volume req for volume prod_a >>> [2019-02-14 02:46:47.800584] I [MSGID: 106499] >>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: >>> Received status volume req for volume prod_b >>> [2019-02-14 02:46:47.770601] I [MSGID: 106499] >>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: >>> Received status volume req for volume engine >>> [2019-02-14 02:46:47.778161] I [MSGID: 106499] >>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: >>> Received status volume req for volume isos >>> [2019-02-14 02:46:47.784020] I [MSGID: 106499] >>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: >>> Received status volume req for volume non_prod_a >>> >>> engine.log >>> >>> # grep -B20 -A20 "2019-02-14 02:41:48" engine.log-20190214 >>> 2019-02-14 02:41:43,495-04 INFO >>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] >>> (DefaultQuartzScheduler1) [5ff5b093] START, >>> GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host1, >>> VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}), >>> log id: 172c9ee8 >>> 2019-02-14 02:41:43,609-04 INFO >>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] >>> (DefaultQuartzScheduler1) [5ff5b093] FINISH, >>> GetGlusterLocalLogicalVolumeListVDSCommand, return: >>> [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@479fcb69, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@6443e68f, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2b4cf035, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5864f06a, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@6119ac8c, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1a9549be, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5614cf81, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@290c9289, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5dd26e8, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@35355754, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@452deeb4, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@8f8b442, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@647e29d3, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7bee4dff, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@511c4478, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1c0bb0bd, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@92e325e, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@260731, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@33aaacc9, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@72657c59, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@aa10c89], >>> log id: 172c9ee8 >>> 2019-02-14 02:41:43,610-04 INFO >>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] >>> (DefaultQuartzScheduler1) [5ff5b093] START, >>> GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host1, >>> VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}), >>> log id: 3a0e9d63 >>> 2019-02-14 02:41:43,703-04 INFO >>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] >>> (DefaultQuartzScheduler1) [5ff5b093] FINISH, >>> GetGlusterLocalPhysicalVolumeListVDSCommand, return: >>> [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@5ca4a20f, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@57a8a76, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@7bd1b14], >>> log id: 3a0e9d63 >>> 2019-02-14 02:41:43,704-04 INFO >>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] >>> (DefaultQuartzScheduler1) [5ff5b093] START, >>> GetGlusterVDOVolumeListVDSCommand(HostName = Host1, >>> VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}), >>> log id: 49966b05 >>> 2019-02-14 02:41:44,213-04 INFO >>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] >>> (DefaultQuartzScheduler1) [5ff5b093] FINISH, >>> GetGlusterVDOVolumeListVDSCommand, return: [], log id: 49966b05 >>> 2019-02-14 02:41:44,214-04 INFO >>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] >>> (DefaultQuartzScheduler1) [5ff5b093] START, >>> GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host2, >>> VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}), >>> log id: 30db0ce2 >>> 2019-02-14 02:41:44,311-04 INFO >>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] >>> (DefaultQuartzScheduler1) [5ff5b093] FINISH, >>> GetGlusterLocalLogicalVolumeListVDSCommand, return: >>> [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@61a309b5, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@ea9cb2e, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@749d57bd, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1c49f9d0, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@655eb54d, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@256ee273, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3bd079dc, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@6804900f, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@78e0a49f, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2acfbc8a, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@12e92e96, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5ea1502c, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2398c33b, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7464102e, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2f221daa, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7b561852, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1eb29d18, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@4a030b80, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@75739027, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3eac8253, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@34fc82c3], >>> log id: 30db0ce2 >>> 2019-02-14 02:41:44,312-04 INFO >>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] >>> (DefaultQuartzScheduler1) [5ff5b093] START, >>> GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host2, >>> VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}), >>> log id: 6671d0d7 >>> 2019-02-14 02:41:44,329-04 INFO >>> [org.ovirt.engine.core.bll.lock.InMemoryLockManager] >>> (DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock >>> 'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]', >>> sharedLocks=''}' >>> 2019-02-14 02:41:44,345-04 INFO >>> [org.ovirt.engine.core.bll.lock.InMemoryLockManager] >>> (DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock >>> 'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]', >>> sharedLocks=''}' >>> 2019-02-14 02:41:44,374-04 INFO >>> [org.ovirt.engine.core.bll.lock.InMemoryLockManager] >>> (DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock >>> 'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]', >>> sharedLocks=''}' >>> 2019-02-14 02:41:44,405-04 INFO >>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] >>> (DefaultQuartzScheduler1) [5ff5b093] FINISH, >>> GetGlusterLocalPhysicalVolumeListVDSCommand, return: >>> [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@f6a9696, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@558e3332, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@5b449da], >>> log id: 6671d0d7 >>> 2019-02-14 02:41:44,406-04 INFO >>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] >>> (DefaultQuartzScheduler1) [5ff5b093] START, >>> GetGlusterVDOVolumeListVDSCommand(HostName = Host2, >>> VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}), >>> log id: 6d2bc6d3 >>> 2019-02-14 02:41:44,908-04 INFO >>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] >>> (DefaultQuartzScheduler1) [5ff5b093] FINISH, >>> GetGlusterVDOVolumeListVDSCommand, return: [], log id: 6d2bc6d3 >>> 2019-02-14 02:41:44,909-04 INFO >>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand] >>> (DefaultQuartzScheduler1) [5ff5b093] START, >>> GetGlusterVolumeAdvancedDetailsVDSCommand(HostName = Host0, >>> GlusterVolumeAdvancedDetailsVDSParameters:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5', >>> volumeName='non_prod_b'}), log id: 36ae23c6 >>> 2019-02-14 02:41:47,336-04 INFO >>> [org.ovirt.engine.core.bll.lock.InMemoryLockManager] >>> (DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock >>> 'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]', >>> sharedLocks=''}' >>> 2019-02-14 02:41:47,351-04 INFO >>> [org.ovirt.engine.core.bll.lock.InMemoryLockManager] >>> (DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock >>> 'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]', >>> sharedLocks=''}' >>> 2019-02-14 02:41:47,379-04 INFO >>> [org.ovirt.engine.core.bll.lock.InMemoryLockManager] >>> (DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock >>> 'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]', >>> sharedLocks=''}' >>> 2019-02-14 02:41:47,979-04 INFO >>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand] >>> (DefaultQuartzScheduler1) [5ff5b093] FINISH, >>> GetGlusterVolumeAdvancedDetailsVDSCommand, return: >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeAdvancedDetails@7a4a787b, >>> log id: 36ae23c6 >>> 2019-02-14 02:41:48,018-04 WARN >>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>> (DefaultQuartzScheduler1) [5ff5b093] EVENT_ID: >>> GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick >>> host2.replaced.domain.com:/gluster_bricks/non_prod_b/non_prod_b of >>> volume non_prod_b of cluster Default from UP to DOWN via cli. >>> 2019-02-14 02:41:48,046-04 WARN >>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>> (DefaultQuartzScheduler1) [5ff5b093] EVENT_ID: >>> GLUSTER_BRICK_STATUS_DOWN(4,151), Status of brick >>> host2.replaced.domain.com:/gluster_bricks/non_prod_b/non_prod_b >>> of volume non_prod_b on cluster Default is down. >>> 2019-02-14 02:41:48,139-04 INFO >>> [org.ovirt.engine.core.bll.lock.InMemoryLockManager] >>> (DefaultQuartzScheduler1) [5ff5b093] Failed to acquire lock and wait lock >>> 'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]', >>> sharedLocks=''}' >>> 2019-02-14 02:41:48,140-04 INFO >>> [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] >>> (DefaultQuartzScheduler3) [7b9bd2d] START, >>> GlusterServersListVDSCommand(HostName = Host0, >>> VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}), >>> log id: e1fb23 >>> 2019-02-14 02:41:48,911-04 INFO >>> [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] >>> (DefaultQuartzScheduler3) [7b9bd2d] FINISH, GlusterServersListVDSCommand, >>> return: [10.12.0.220/24:CONNECTED, host1.replaced.domain.com:CONNECTED, >>> host2.replaced.domain.com:CONNECTED], log id: e1fb23 >>> 2019-02-14 02:41:48,930-04 INFO >>> [org.ovirt.engine.core.bll.lock.InMemoryLockManager] >>> (DefaultQuartzScheduler1) [5ff5b093] Failed to acquire lock and wait lock >>> 'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]', >>> sharedLocks=''}' >>> 2019-02-14 02:41:48,931-04 INFO >>> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] >>> (DefaultQuartzScheduler3) [7b9bd2d] START, >>> GlusterVolumesListVDSCommand(HostName = Host0, >>> GlusterVolumesListVDSParameters:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}), >>> log id: 68f1aecc >>> 2019-02-14 02:41:49,366-04 INFO >>> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] >>> (DefaultQuartzScheduler3) [7b9bd2d] FINISH, GlusterVolumesListVDSCommand, >>> return: >>> {6c05dfc6-4dc0-41e3-a12f-55b4767f1d35=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@1952a85, >>> 3f8f6a0f-aed4-48e3-9129-18a2a3f64eef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@2f6688ae, >>> 71ff56d9-79b8-445d-b637-72ffc974f109=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@730210fb, >>> 752a9438-cd11-426c-b384-bc3c5f86ed07=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@c3be510c, >>> c3e7447e-8514-4e4a-9ff5-a648fe6aa537=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@450befac, >>> 79e8e93c-57c8-4541-a360-726cec3790cf=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@1926e392}, >>> log id: 68f1aecc >>> 2019-02-14 02:41:49,489-04 INFO >>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] >>> (DefaultQuartzScheduler1) [5ff5b093] START, >>> GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host0, >>> VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}), >>> log id: 38debe74 >>> 2019-02-14 02:41:49,581-04 INFO >>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] >>> (DefaultQuartzScheduler1) [5ff5b093] FINISH, >>> GetGlusterLocalLogicalVolumeListVDSCommand, return: >>> [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5e5a7925, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2cdf5c9e, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@443cb62, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@49a3e880, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@443d23c0, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1250bc75, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@8d27d86, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5e6363f4, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@73ed78db, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@64c9d1c7, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7fecbe95, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3a551e5f, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2266926e, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@88b380c, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1209279e, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3c6466, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@16df63ed, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@47456262, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1c2b88c3, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7f57c074, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@12fa0478], >>> log id: 38debe74 >>> 2019-02-14 02:41:49,582-04 INFO >>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] >>> (DefaultQuartzScheduler1) [5ff5b093] START, >>> GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host0, >>> VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}), >>> log id: 7ec02237 >>> 2019-02-14 02:41:49,660-04 INFO >>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] >>> (DefaultQuartzScheduler1) [5ff5b093] FINISH, >>> GetGlusterLocalPhysicalVolumeListVDSCommand, return: >>> [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@3eedd0bc, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@7f78e375, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@3d63e126], >>> log id: 7ec02237 >>> 2019-02-14 02:41:49,661-04 INFO >>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] >>> (DefaultQuartzScheduler1) [5ff5b093] START, >>> GetGlusterVDOVolumeListVDSCommand(HostName = Host0, >>> VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}), >>> log id: 42cdad27 >>> 2019-02-14 02:41:50,142-04 INFO >>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] >>> (DefaultQuartzScheduler1) [5ff5b093] FINISH, >>> GetGlusterVDOVolumeListVDSCommand, return: [], log id: 42cdad27 >>> 2019-02-14 02:41:50,143-04 INFO >>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] >>> (DefaultQuartzScheduler1) [5ff5b093] START, >>> GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host1, >>> VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}), >>> log id: 12f5fdf2 >>> 2019-02-14 02:41:50,248-04 INFO >>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] >>> (DefaultQuartzScheduler1) [5ff5b093] FINISH, >>> GetGlusterLocalLogicalVolumeListVDSCommand, return: >>> [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2aaed792, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@8e66930, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@276d599e, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1aca2aec, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@46846c60, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7d103269, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@30fc25fc, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7baae445, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1ea8603c, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@62578afa, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@33d58089, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1f71d27a, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@4205e828, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1c5bbac8, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@395a002, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@12664008, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7f4faec4, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3e03d61f, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1038e46d, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@307e8062, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@32453127], >>> log id: 12f5fdf2 >>> 2019-02-14 02:41:50,249-04 INFO >>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] >>> (DefaultQuartzScheduler1) [5ff5b093] START, >>> GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host1, >>> VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}), >>> log id: 1256aa5e >>> 2019-02-14 02:41:50,338-04 INFO >>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] >>> (DefaultQuartzScheduler1) [5ff5b093] FINISH, >>> GetGlusterLocalPhysicalVolumeListVDSCommand, return: >>> [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@459a2ff5, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@123cab4, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@1af41fbe], >>> log id: 1256aa5e >>> 2019-02-14 02:41:50,339-04 INFO >>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] >>> (DefaultQuartzScheduler1) [5ff5b093] START, >>> GetGlusterVDOVolumeListVDSCommand(HostName = Host1, >>> VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}), >>> log id: 3dd752e4 >>> 2019-02-14 02:41:50,847-04 INFO >>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] >>> (DefaultQuartzScheduler1) [5ff5b093] FINISH, >>> GetGlusterVDOVolumeListVDSCommand, return: [], log id: 3dd752e4 >>> 2019-02-14 02:41:50,848-04 INFO >>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] >>> (DefaultQuartzScheduler1) [5ff5b093] START, >>> GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host2, >>> VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}), >>> log id: 29a6272c >>> 2019-02-14 02:41:50,954-04 INFO >>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] >>> (DefaultQuartzScheduler1) [5ff5b093] FINISH, >>> GetGlusterLocalLogicalVolumeListVDSCommand, return: >>> [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@364f3ec6, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@c7cce5e, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@b3bed47, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@13bc244b, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5cca81f4, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@36aeba0d, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@62ab384a, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1047d628, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@188a30f5, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5bb79f3b, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@60e5956f, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@4e3df9cd, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7796567, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@60d06cf4, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2cd2d36c, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@d80a4aa, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@411eaa20, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@22cac93b, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@18b927bd, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@101465f4, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@246f927c], >>> log id: 29a6272c >>> 2019-02-14 02:41:50,955-04 INFO >>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] >>> (DefaultQuartzScheduler1) [5ff5b093] START, >>> GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host2, >>> VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}), >>> log id: 501814db >>> 2019-02-14 02:41:51,044-04 INFO >>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] >>> (DefaultQuartzScheduler1) [5ff5b093] FINISH, >>> GetGlusterLocalPhysicalVolumeListVDSCommand, return: >>> [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@1cd55aa, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@32c5aba2, >>> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@6ae123f4], >>> log id: 501814db >>> 2019-02-14 02:41:51,045-04 INFO >>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] >>> (DefaultQuartzScheduler1) [5ff5b093] START, >>> GetGlusterVDOVolumeListVDSCommand(HostName = Host2, >>> VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}), >>> log id: 7acf4cbf >>> 2019-02-14 02:41:51,546-04 INFO >>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] >>> (DefaultQuartzScheduler1) [5ff5b093] FINISH, >>> GetGlusterVDOVolumeListVDSCommand, return: [], log id: 7acf4cbf >>> 2019-02-14 02:41:51,547-04 INFO >>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand] >>> (DefaultQuartzScheduler1) [5ff5b093] START, >>> GetGlusterVolumeAdvancedDetailsVDSCommand(HostName = Host0, >>> GlusterVolumeAdvancedDetailsVDSParameters:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5', >>> volumeName='non_prod_a'}), log id: 11c42649 >>> >>> On Thu, Feb 14, 2019 at 10:16 AM Sandro Bonazzola <sbona...@redhat.com> >>> wrote: >>> >>>> >>>> >>>> Il giorno gio 14 feb 2019 alle ore 07:54 Jayme <jay...@gmail.com> ha >>>> scritto: >>>> >>>>> I have a three node HCI gluster which was previously running 4.2 with >>>>> zero problems. I just upgraded it yesterday. I ran in to a few bugs >>>>> right >>>>> away with the upgrade process, but aside from that I also discovered other >>>>> users with severe GlusterFS problems since the upgrade to new GlusterFS >>>>> version. It is less than 24 hours since I upgrade my cluster and I just >>>>> got a notice that one of my GlusterFS bricks is offline. There does >>>>> appear >>>>> to be a very real and serious issue here with the latest updates. >>>>> >>>> >>>> tracking the issue on Gluster side on this bug: >>>> https://bugzilla.redhat.com/show_bug.cgi?id=1677160 >>>> If you can help Gluster community providing requested logs it would be >>>> great. >>>> >>>> >>>> >>>> >>>>> >>>>> >>>>> On Wed, Feb 13, 2019 at 7:26 PM <dsc...@umbctraining.com> wrote: >>>>> >>>>>> I'm abandoning my production ovirt cluster due to instability. I >>>>>> have a 7 host cluster running about 300 vms and have been for over a >>>>>> year. >>>>>> It has become unstable over the past three days. I have random hosts >>>>>> both, >>>>>> compute and storage disconnecting. AND many vms disconnecting and >>>>>> becoming >>>>>> unusable. >>>>>> >>>>>> 7 host are 4 compute hosts running Ovirt 4.2.8 and three glusterfs >>>>>> hosts running 3.12.5. I submitted a bugzilla bug and they immediately >>>>>> assigned it to the storage people but have not responded with any >>>>>> meaningful information. I have submitted several logs. >>>>>> >>>>>> I have found some discussion on problems with instability with >>>>>> gluster 3.12.5. I would be willing to upgrade my gluster to a more >>>>>> stable >>>>>> version if that's the culprit. I installed gluster using the ovirt gui >>>>>> and >>>>>> this is the version the ovirt gui installed. >>>>>> >>>>>> Is there an ovirt health monitor available? Where should I be >>>>>> looking to get a resolution the problems I'm facing. >>>>>> _______________________________________________ >>>>>> Users mailing list -- users@ovirt.org >>>>>> To unsubscribe send an email to users-le...@ovirt.org >>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>> oVirt Code of Conduct: >>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>> List Archives: >>>>>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/BL4M3JQA3IEXCQUY4IGQXOAALRUQ7TVB/ >>>>>> >>>>> _______________________________________________ >>>>> Users mailing list -- users@ovirt.org >>>>> To unsubscribe send an email to users-le...@ovirt.org >>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>> oVirt Code of Conduct: >>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>> List Archives: >>>>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/QULCBXHTKSCPKH4UV6GLMOLJE6J7M5UW/ >>>>> >>>> >>>> >>>> -- >>>> >>>> SANDRO BONAZZOLA >>>> >>>> MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV >>>> >>>> Red Hat EMEA <https://www.redhat.com/> >>>> >>>> sbona...@redhat.com >>>> <https://red.ht/sig> >>>> >>> _______________________________________________ >>> Users mailing list -- users@ovirt.org >>> To unsubscribe send an email to users-le...@ovirt.org >>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>> oVirt Code of Conduct: >>> https://www.ovirt.org/community/about/community-guidelines/ >>> List Archives: >>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/4YV6ERVJ4OYRJYTH4BWPJMUABNKYL45R/ >>> >> _______________________________________________ > Users mailing list -- users@ovirt.org > To unsubscribe send an email to users-le...@ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/U3GL3SYP6BWMYKVSGXAKXBXF5C6APGQF/ >
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/K3HJIOO23LS3653C65QULV74M3UNRLMM/
