Hi Kabelo, do you mean RabbitMQ and Cassandra are not available? I
don't see any Redis errors.
For RabbitMQ, you'll want to synchronize it with your app server
startup as well.
For Cassandra, I believe the startup command returns before Cassandra
is actually ready to start listening for requests. You could put a
sleep delay into the script or create a utility that will wait until
the Cassandra thrift port (9160) is open.
Hope that helps,
Branden
On Wed, Jan 8, 2014 at 1:29 PM, Kabelo Letsoalo <kab...@opencollab.co.za> wrote:
> Hi guys.
>
> Having trouble with the Hilary upstart
>
> For Hilary to start succcessfully cassandra and redis must be running. The
> upstart script on at [1] doesn't wait for cassandra and redis.
> My upstart fails to start Hilary successfully. upstart log :
>
> {"name":"oae-cassandra","hostname":"oae","pid":632,"level":50,"err":{"message":"","name":"HelenusUnavailableException","stack":"HelenusUnavailableException\n
> at onReturn
> (/home/dev/oae/Hilary/node_modules/helenus/lib/connection.js:404:23)\n at
> client._reqs.(anonymous function)
> (/home/dev/oae/Hilary/node_modules/helenus/node_modules/helenus-thrift/lib/thrift/connection.js:80:11)\n
> at Object.CassandraClient.recv_execute_cql_query
> (/home/dev/oae/Hilary/node_modules/helenus/lib/cassandra/Cassandra.js:8286:12)\n
> at
> /home/dev/oae/Hilary/node_modules/helenus/node_modules/helenus-thrift/lib/thrift/connection.js:83:37\n
> at Socket.<anonymous>
> (/home/dev/oae/Hilary/node_modules/helenus/node_modules/helenus-thrift/lib/thrift/transport.js:70:9)\n
> at Socket.EventEmitter.emit (events.js:95:17)\n at Socket.<anonymous>
> (_stream_readable.js:736:14)\n at Socket.EventEmitter.emit
> (events.js:92:17)\n at emitReadable_ (_stream_readable.js:408:10)\n at
> emitReadable (_stream_readable.js:404:5)"},"query":"SELECT * FROM Tenant
> USING CONSISTENCY QUORUM","parameters":null,"msg":"An error occurred
> executing a cassandra query.","time":"2014-01-08T19:40:13.428Z","v":0}
> {"name":"oae-modules","hostname":"oae","pid":632,"level":50,"err":{"code":500,"msg":"An
> error occurred executing a query"},"msg":"Error initializing module
> oae-tenants","time":"2014-01-08T19:40:13.428Z","v":0}
> {"name":"oae","hostname":"oae","pid":632,"level":30,"msg":"All modules are
> bootstrapped, initializing
> servers.","time":"2014-01-08T19:40:13.431Z","v":0}
> {"name":"system","hostname":"oae","pid":632,"level":50,"err":{"code":500,"msg":"An
> error occurred executing a query"},"msg":"Error initializing
> server.","time":"2014-01-08T19:40:13.442Z","v":0}
> {"name":"system","hostname":"oae","pid":632,"level":30,"msg":"Initialization
> all done ... Firing up tenants ...
> Enjoy!","time":"2014-01-08T19:40:13.452Z","v":0}
> {"name":"mq","hostname":"oae","pid":632,"level":50,"err":{"message":"connect
> ECONNREFUSED","name":"Error","stack":"Error: connect ECONNREFUSED\n at
> errnoException (net.js:884:11)\n at Object.afterConnect [as oncomplete]
> (net.js:875:19)","code":"ECONNREFUSED"},"msg":"Error in the RabbitMQ
> connection. Reconnecting.","time":"2014-01-08T19:40:13.691Z","v":0}
> {"name":"mq","hostname":"oae","pid":632,"level":50,"err":{"message":"connect
> ECONNREFUSED","name":"Error","stack":"Error: connect ECONNREFUSED\n at
> errnoException (net.js:884:11)\n at Object.afterConnect [as oncomplete]
> (net.js:875:19)","code":"ECONNREFUSED"},"msg":"Error in the RabbitMQ
> connection. Reconnecting.","time":"2014-01-08T19:40:14.693Z","v":0}
> {"name":"mq","hostname":"oae","pid":632,"level":50,"err":{"message":"connect
> ECONNREFUSED","name":"Error","stack":"Error: connect ECONNREFUSED\n at
> errnoException (net.js:884:11)\n at Object.afterConnect [as oncomplete]
> (net.js:875:19)","code":"ECONNREFUSED"},"msg":"Error in the RabbitMQ
> connection. Reconnecting.","time":"2014-01-08T19:40:15.694Z","v":0}
> {"name":"mq","hostname":"oae","pid":632,"level":50,"err":{"message":"connect
> ECONNREFUSED","name":"Error","stack":"Error: connect ECONNREFUSED\n at
> errnoException (net.js:884:11)\n at Object.afterConnect [as oncomplete]
> (net.js:875:19)","code":"ECONNREFUSED"},"msg":"Error in the RabbitMQ
> connection. Reconnecting.","time":"2014-01-08T19:40:16.695Z","v":0}
> {"name":"mq","hostname":"oae","pid":632,"level":50,"err":{"message":"connect
> ECONNREFUSED","name":"Error","stack":"Error: connect ECONNREFUSED\n at
> errnoException (net.js:884:11)\n at Object.afterConnect [as oncomplete]
> (net.js:875:19)","code":"ECONNREFUSED"},"msg":"Error in the RabbitMQ
> connection. Reconnecting.","time":"2014-01-08T19:40:17.695Z","v":0}
> {"name":"mq","hostname":"oae","pid":632,"level":50,"err":{"message":"connect
> ECONNREFUSED","name":"Error","stack":"Error: connect ECONNREFUSED\n at
> errnoException (net.js:884:11)\n at Object.afterConnect [as oncomplete]
> (net.js:875:19)","code":"ECONNREFUSED"},"msg":"Error in the RabbitMQ
> connection. Reconnecting.","time":"2014-01-08T19:40:18.696Z","v":0}
> {"name":"mq","hostname":"oae","pid":632,"level":50,"err":{"message":"connect
> ECONNREFUSED","name":"Error","stack":"Error: connect ECONNREFUSED\n at
> errnoException (net.js:884:11)\n at Object.afterConnect [as oncomplete]
> (net.js:875:19)","code":"ECONNREFUSED"},"msg":"Error in the RabbitMQ
> connection. Reconnecting.","time":"2014-01-08T19:40:19.697Z","v":0}
> {"name":"mq","hostname":"oae","pid":632,"level":50,"err":{"message":"connect
> ECONNREFUSED","name":"Error","stack":"Error: connect ECONNREFUSED\n at
> errnoException (net.js:884:11)\n at Object.afterConnect [as oncomplete]
> (net.js:875:19)","code":"ECONNREFUSED"},"msg":"Error in the RabbitMQ
> connection. Reconnecting.","time":"2014-01-08T19:40:20.699Z","v":0}
> {"name":"mq","hostname":"oae","pid":632,"level":50,"err":{"message":"connect
> ECONNREFUSED","name":"Error","stack":"Error: connect ECONNREFUSED\n at
> errnoException (net.js:884:11)\n at Object.afterConnect [as oncomplete]
> (net.js:875:19)","code":"ECONNREFUSED"},"msg":"Error in the RabbitMQ
> connection. Reconnecting.","time":"2014-01-08T19:40:21.702Z","v":0}
> {"name":"mq","hostname":"oae","pid":632,"level":30,"msg":"Connection to
> RabbitMQ established.","time":"2014-01-08T19:40:22.779Z","v":0}
> {"name":"oae-server","hostname":"oae","pid":632,"level":30,"path":"/api/me","msg":"Rejecting
> request during shutdown with 502
> error.","time":"2014-01-08T19:40:50.658Z","v":0}
> {"name":"oae-server","hostname":"oae","pid":632,"level":30,"path":"/api/me","msg":"Rejecting
> request during shutdown with 502
> error.","time":"2014-01-08T19:41:50.662Z","v":0}
> {"name":"oae-server","hostname":"oae","pid":632,"level":30,"path":"/api/me","msg":"Rejecting
> request during shutdown with 502
> error.","time":"2014-01-08T19:42:50.652Z","v":0}
> {"name":"oae-server","hostname":"oae","pid":632,"level":30,"path":"/api/me","msg":"Rejecting
> request during shutdown with 502
> error.","time":"2014-01-08T19:43:50.670Z","v":0}
> {"name":"oae-server","hostname":"oae","pid":632,"level":30,"path":"/api/me","msg":"Rejecting
> request during shutdown with 502
> error.","time":"2014-01-08T19:44:50.642Z","v":0}
> {"name":"oae-server","hostname":"oae","pid":632,"level":30,"path":"/api/me","msg":"Rejecting
> request during shutdown with 502
> error.","time":"2014-01-08T19:45:50.664Z","v":0}
> {"name":"oae-server","hostname":"oae","pid":632,"level":30,"path":"/api/ui/skin","msg":"Rejecting
> request during shutdown with 502
> error.","time":"2014-01-08T19:46:18.538Z","v":0}
> {"name":"oae-server","hostname":"oae","pid":632,"level":30,"path":"/api/me","msg":"Rejecting
> request during shutdown with 502
> error.","time":"2014-01-08T19:47:19.716Z","v":0}
>
>
>
> Cassandra, Redis and everything else starts up fine.
> Cassandra and Redis upstart scripts are named "upstart_oae_cassandra.conf"
> and "upstart_oae_redis.conf"
>
> Hilary upstart :
>
> # Upstart script for the Node.JS Hilary app sever.
> description "start OAE Hilary"
>
> start on started upstart_oae_cassandra and started upstart_oae_redis
> stop on shutdown
>
> # Restart the process if it dies with a signal
> # or exit code not given by the 'normal exit' stanza.
> # Give up if restart occurs 10 times in 90 seconds.
> respawn
> respawn limit 10 90
>
> # The command to start the node process.
> respawn
> respawn limit 5 60
>
> # The node app sometimes creates child processes.
> expect fork
>
> # Pass along some environment variables
> env NODE_ENV=production
>
> chdir /home/dev/oae/Hilary
> script
> exec /home/dev/oae/node-v0.10.10-linux-x64/bin/node app.js &
> end script
>
>
>
> [1]
> https://github.com/oaeproject/puppet-hilary/blob/master/modules/hilary/templates/upstart_hilary.conf.erb
>
> Any pointers?
>
> Thanks,
> Kabelo
>
> ________________________________
> See OpenCollab email disclaimer at
> http://www.opencollab.co.za/email-disclaimer
> _______________________________________________
> oae-dev mailing list
> oae-dev@collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/oae-dev
>
_______________________________________________
oae-dev mailing list
oae-dev@collab.sakaiproject.org
http://collab.sakaiproject.org/mailman/listinfo/oae-dev
