Bart, Try to run 'onedb fsck' to check for any errors in DB [1].
[1] http://docs.opennebula.org/4.12/administration/references/onedb.html -- Thank you, Dmitri Chebotarov VCL Sys Eng, Engineering & Architectural Support, TSD - Ent Servers & Messaging 223 Aquia Building, Ffx, MSN: 1B5 Phone: (703) 993-6175 | Fax: (703) 993-3404 > On Mar 12, 2015, at 6:50 , Bart <b...@pleh.info> wrote: > > Hi Everyone, > > We're currently experiencing a weird issue on our OpenNebula management node. > > The node is setup with the following configuration: > > • CentOS 6.6 > • Two interfaces in an balance-alb bond. > • MySQL backend > • /var/lib/one mounted on glusterfs and shared with all nodes. > • OpenNebula 4.12 > • We're behind a proxy, so all proxy variables are set in the > environment for all users. > > > When working on the commandline everything works as a charm, you can list all > nodes/datastores, view the details, etc. All working smooth and fast! > > The problem we face however is working inside the sunstone interface. It > seems that listing resources (nodes and datastores) works normally but when > we click a datastore/node/whatever it takes ages (minutes) before it shows > the contents. > > The sunstone.error logs only give the following info (in debug mode): > > Errno::ETIMEDOUT - Connection timed out - connect(2): > /usr/lib/ruby/1.8/net/http.rb:560:in `initialize' > /usr/lib/ruby/1.8/net/http.rb:560:in `open' > /usr/lib/ruby/1.8/net/http.rb:560:in `connect' > /usr/lib/ruby/1.8/timeout.rb:53:in `timeout' > /usr/lib/ruby/1.8/timeout.rb:101:in `timeout' > /usr/lib/ruby/1.8/net/http.rb:560:in `connect' > /usr/lib/ruby/1.8/net/http.rb:553:in `do_start' > /usr/lib/ruby/1.8/net/http.rb:542:in `start' > /usr/lib/ruby/1.8/net/http.rb:1035:in `request' > /usr/lib/ruby/1.8/net/http.rb:772:in `get' > > /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/adapter/net_http.rb:80:in > `perform_request' > > /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/adapter/net_http.rb:40:in > `call' > > /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/adapter/net_http.rb:87:in > `with_net_http_connection' > > /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/adapter/net_http.rb:32:in > `call' > > /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/request/retry.rb:20:in > `call' > > /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/request/encode_json.rb:21:in > `call' > > /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/request/multipart.rb:14:in > `call' > > /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/request/upload.rb:16:in > `call' > > /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/request/etag_cache.rb:31:in > `call' > > /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/request/authorization.rb:38:in > `call' > /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/response.rb:8:in `call' > > /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/response/parse_iso_dates.rb:11:in > `call' > > /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/response/logger.rb:20:in > `call' > > /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/response/callback.rb:14:in > `call' > /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/response.rb:8:in `call' > > /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/response/raise_error.rb:9:in > `call' > /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/rack_builder.rb:139:in > `build_response' > /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/connection.rb:377:in > `run_request' > /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/connection.rb:140:in > `get' > > /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/actions.rb:104:in > `find!' > > /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/actions.rb:119:in > `find' > > /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/collection.rb:62:in > `send' > > /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/collection.rb:62:in > `find' > > /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/client.rb:56:in > `current_user' > /usr/lib/one/sunstone/routes/support.rb:66:in `zendesk_client' > /usr/lib/one/sunstone/routes/support.rb:121:in `GET /support/request' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:863:in `call' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:863:in `route' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:521:in > `instance_eval' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:521:in > `route_eval' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:500:in `route!' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:497:in `catch' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:497:in `route!' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:476:in `each' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:476:in `route!' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:601:in > `dispatch!' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:411:in `call!' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in > `instance_eval' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `invoke' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `catch' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `invoke' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:411:in `call!' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:399:in `call' > /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/commonlogger.rb:18:in `call' > /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/deflater.rb:13:in `call' > /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/session/abstract/id.rb:63:in > `context' > /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/session/abstract/id.rb:58:in > `call' > /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/showexceptions.rb:24:in > `call' > /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/methodoverride.rb:24:in > `call' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:979:in `call' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:1005:in > `synchronize' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:979:in `call' > /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/content_length.rb:13:in > `call' > /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/chunked.rb:15:in `call' > /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:84:in > `pre_process' > /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:82:in `catch' > /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:82:in > `pre_process' > /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:57:in `process' > /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:42:in > `receive_data' > /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in > `run_machine' > /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in > `run' > /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/backends/base.rb:61:in > `start' > /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/server.rb:159:in `start' > /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/handler/thin.rb:14:in `run' > /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:946:in `run!' > /usr/lib/one/sunstone/sunstone-server.rb:627 > > > > When I click a datastore in the interface, nothing will happen in the logs > (only access logs give some info). Then, when this error appears, only then > will we get information about the datastore. > > So everything works eventually, but there's a massive timeout causing this > problem. > > I've also noticed that I'm getting these connection time out errors without > using the interface, so it might be something in the background doing some > action?! If so, this action seems to block all other activities untill its > finally done. > > At first I thought this was IPv6 related, so we've disabled that, then I > doubted the bonding interface, but that wasnt it either. Then DNS, but even > that didnt budge. We even did traces on the OS but didn't find anything > suspicious. > > Before we ran version 4.10, so after upgrading we hoped it would solve > something, but that also didn't happen. > > So right now we're kinda stuck... Does anyone have an idea on how to > troubleshoot and solve this issue? What is causing this connection time out? > > > > -- > Bart G. > _______________________________________________ > Users mailing list > Users@lists.opennebula.org > http://lists.opennebula.org/listinfo.cgi/users-opennebula.org _______________________________________________ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
