Digging more the Ambari agents are not registered with the driver. I can not find documentation on how to cause the agents (which *are *running on each of the hosts) register themselves with the driver. Tips appreciated.
2015-10-15 19:36 GMT-07:00 Stephen Boesch <[email protected]>: > I neglected to mention: the ambari console shows the attempted cluster > launch as "Logical REquest: Provision Cluster <name>: Not started" > > > 1 Background Operation Running > Operations > Start Time > Duration > Show: All (1)Pending (1)In Progress (0)Failed (0)Success (0)Aborted > (0)Timedout > (0) > Logical Request: Provision Cluster 'eazcluster' > <http://10.103.177.55:8080/#> > Not started > > 2015-10-15 19:33 GMT-07:00 Stephen Boesch <[email protected]>: > >> We have a blueprints defined cluster that has an accepted topology: >> >> ambari-server.log" 6839L, 998638C >> 16 Oct 2015 01:31:40,374 INFO [pool-2-thread-1] AmbariContext:375 - >> Config type hdfs-site is resolved in the cluster config. >> 16 Oct 2015 01:31:40,374 INFO [pool-2-thread-1] AmbariContext:375 - >> Config type hadoop-env is resolved in the cluster config. >> 16 Oct 2015 01:31:40,375 INFO [pool-2-thread-1] AmbariContext:375 - >> Config type hive-site is resolved in the cluster config. >> 16 Oct 2015 01:31:40,375 INFO [pool-2-thread-1] AmbariContext:375 - >> Config type yarn-site is resolved in the cluster config. >> 16 Oct 2015 01:31:40,375 INFO [pool-2-thread-1] AmbariContext:375 - >> Config type oozie-site is resolved in the cluster config. >> 16 Oct 2015 01:31:40,376 INFO [pool-2-thread-1] AmbariContext:375 - >> Config type webhcat-site is resolved in the cluster config. >> 16 Oct 2015 01:31:40,376 INFO [pool-2-thread-1] AmbariContext:375 - >> Config type cluster-env is resolved in the cluster config. >> 16 Oct 2015 01:31:40,376 INFO [pool-2-thread-1] AmbariContext:375 - >> Config type mapred-site is resolved in the cluster config. >> 16 Oct 2015 01:31:40,377 INFO [pool-2-thread-1] AmbariContext:375 - >> Config type core-site is resolved in the cluster config. >> 16 Oct 2015 01:31:40,377 INFO [pool-2-thread-1] AmbariContext:375 - >> Config type oozie-env is resolved in the cluster config. >> 16 Oct 2015 01:31:40,377 INFO [pool-2-thread-1] AmbariContext:381 - All >> required configuration types are in the* TOPOLOGY_RESOLVED state. >> Blueprint deployment can now continue.* >> 16 Oct 2015 01:31:40,377 INFO [pool-2-thread-1] TopologyManager:558 - >> TopologyManager.ConfigureClusterTask: Exiting >> >> >> We then have a few messages that seem healthy: >> >> 16 Oct 2015 01:32:00,945 INFO [Thread-25] >> AbstractPoolBackedDataSource:462 - Initializing c3p0 pool... >> 16 Oct 2015 01:32:00,985 INFO [Thread-25] JobStoreTX:861 - Freed 0 >> triggers from 'acquired' / 'blocked' state. >> 16 Oct 2015 01:32:00,992 INFO [Thread-25] JobStoreTX:871 - Recovering 0 >> jobs that were in-progress at the time of the last shut-down. >> 16 Oct 2015 01:32:00,992 INFO [Thread-25] JobStoreTX:884 - Recovery >> complete. >> 16 Oct 2015 01:32:00,998 INFO [Thread-25] QuartzScheduler:575 - >> Scheduler ExecutionScheduler_$_NON_CLUSTERED started. >> 16 Oct 2015 01:33:34,658 INFO [qtp-client-72] JvmInformation:446 - >> Detected JVM data model settings of: 64-Bit HotSpot JVM with Compressed >> OOPs and Concurrent Mark-and-Sweep GC >> 16 Oct 2015 01:33:34,928 INFO [qtp-client-72] >> TimelineMetricCacheProvider:73 - .. >> >> However we then have a series of LONG (multiple minutes) delays, in >> between which there are some "Looking for keyname <foo>" : >> >> >> 016.novalocal/10.103.177.55-9ac6f6e0-9d4a-447e-b3ee-53926d6e16bf >> 16 Oct 2015 01:33:35,088 INFO [qtp-client-76] PersistKeyValueService:82 >> - Looking for keyName CLUSTER_CURRENT_STATUS >> 16 Oct 2015 01:33:36,272 INFO [qtp-client-72] PersistKeyValueService:82 >> - Looking for keyName user-pref-admin-dashboard >> 16 Oct 2015 01:33:43,550 INFO [qtp-client-72] PersistKeyValueService:82 >> - Looking for keyName time-range-service-HDFS >> 16 Oct 2015 01:33:44,995 WARN [pool-7-thread-1] JMXPropertyProvider:202 >> - Unable to get JMX metrics. No host name for NAMENODE >> 16 Oct 2015 01:34:00,930 INFO [qtp-client-72] PersistKeyValueService:82 >> - Looking for keyName admin-settings-show-bg-admin >> 16 Oct 2015 01:34:01,010 INFO [qtp-client-76] PersistKeyValueService:82 >> - Looking for keyName hostPopup-pagination-displayLength-admin >> 16 Oct 2015 01:36:34,123 INFO [qtp-client-82] PersistKeyValueService:82 >> - Looking for keyName time-range-service-AMBARI_METRICS >> 16 Oct 2015 01:36:42,903 INFO [qtp-client-125] PersistKeyValueService:82 >> - Looking for keyName time-range-service-HDFS >> 16 Oct 2015 01:36:44,138 WARN [pool-7-thread-35] JMXPropertyProvider:202 >> - Unable to get JMX metrics. No host name for NAMENODE >> 16 Oct 2015 01:36:57,590 INFO [qtp-client-75] PersistKeyValueService:82 >> - Looking for keyName admin-settings-show-bg-admin >> 16 Oct 2015 01:48:10,901 INFO [qtp-client-192] PersistKeyValueService:82 >> - Looking for keyName admin-settings-show-bg-admin >> 16 Oct 2015 01:53:06,487 INFO [qtp-client-214] PersistKeyValueService:82 >> - Looking for keyName user-pref-admin-dashboard >> >> >> >> After that last message there was another 15 minute delay and then >> nothing (another sixty minutes has passed).. >> >> How to determine what is happening here and why? >> >> >> >
