I neglected to mention: the ambari console shows the attempted cluster launch as "Logical REquest: Provision Cluster <name>: Not started"
1 Background Operation Running Operations Start Time Duration Show: All (1)Pending (1)In Progress (0)Failed (0)Success (0)Aborted (0)Timedout (0) Logical Request: Provision Cluster 'eazcluster' <http://10.103.177.55:8080/#> Not started 2015-10-15 19:33 GMT-07:00 Stephen Boesch <[email protected]>: > We have a blueprints defined cluster that has an accepted topology: > > ambari-server.log" 6839L, 998638C > 16 Oct 2015 01:31:40,374 INFO [pool-2-thread-1] AmbariContext:375 - > Config type hdfs-site is resolved in the cluster config. > 16 Oct 2015 01:31:40,374 INFO [pool-2-thread-1] AmbariContext:375 - > Config type hadoop-env is resolved in the cluster config. > 16 Oct 2015 01:31:40,375 INFO [pool-2-thread-1] AmbariContext:375 - > Config type hive-site is resolved in the cluster config. > 16 Oct 2015 01:31:40,375 INFO [pool-2-thread-1] AmbariContext:375 - > Config type yarn-site is resolved in the cluster config. > 16 Oct 2015 01:31:40,375 INFO [pool-2-thread-1] AmbariContext:375 - > Config type oozie-site is resolved in the cluster config. > 16 Oct 2015 01:31:40,376 INFO [pool-2-thread-1] AmbariContext:375 - > Config type webhcat-site is resolved in the cluster config. > 16 Oct 2015 01:31:40,376 INFO [pool-2-thread-1] AmbariContext:375 - > Config type cluster-env is resolved in the cluster config. > 16 Oct 2015 01:31:40,376 INFO [pool-2-thread-1] AmbariContext:375 - > Config type mapred-site is resolved in the cluster config. > 16 Oct 2015 01:31:40,377 INFO [pool-2-thread-1] AmbariContext:375 - > Config type core-site is resolved in the cluster config. > 16 Oct 2015 01:31:40,377 INFO [pool-2-thread-1] AmbariContext:375 - > Config type oozie-env is resolved in the cluster config. > 16 Oct 2015 01:31:40,377 INFO [pool-2-thread-1] AmbariContext:381 - All > required configuration types are in the* TOPOLOGY_RESOLVED state. > Blueprint deployment can now continue.* > 16 Oct 2015 01:31:40,377 INFO [pool-2-thread-1] TopologyManager:558 - > TopologyManager.ConfigureClusterTask: Exiting > > > We then have a few messages that seem healthy: > > 16 Oct 2015 01:32:00,945 INFO [Thread-25] > AbstractPoolBackedDataSource:462 - Initializing c3p0 pool... > 16 Oct 2015 01:32:00,985 INFO [Thread-25] JobStoreTX:861 - Freed 0 > triggers from 'acquired' / 'blocked' state. > 16 Oct 2015 01:32:00,992 INFO [Thread-25] JobStoreTX:871 - Recovering 0 > jobs that were in-progress at the time of the last shut-down. > 16 Oct 2015 01:32:00,992 INFO [Thread-25] JobStoreTX:884 - Recovery > complete. > 16 Oct 2015 01:32:00,998 INFO [Thread-25] QuartzScheduler:575 - Scheduler > ExecutionScheduler_$_NON_CLUSTERED started. > 16 Oct 2015 01:33:34,658 INFO [qtp-client-72] JvmInformation:446 - > Detected JVM data model settings of: 64-Bit HotSpot JVM with Compressed > OOPs and Concurrent Mark-and-Sweep GC > 16 Oct 2015 01:33:34,928 INFO [qtp-client-72] > TimelineMetricCacheProvider:73 - .. > > However we then have a series of LONG (multiple minutes) delays, in > between which there are some "Looking for keyname <foo>" : > > > 016.novalocal/10.103.177.55-9ac6f6e0-9d4a-447e-b3ee-53926d6e16bf > 16 Oct 2015 01:33:35,088 INFO [qtp-client-76] PersistKeyValueService:82 - > Looking for keyName CLUSTER_CURRENT_STATUS > 16 Oct 2015 01:33:36,272 INFO [qtp-client-72] PersistKeyValueService:82 - > Looking for keyName user-pref-admin-dashboard > 16 Oct 2015 01:33:43,550 INFO [qtp-client-72] PersistKeyValueService:82 - > Looking for keyName time-range-service-HDFS > 16 Oct 2015 01:33:44,995 WARN [pool-7-thread-1] JMXPropertyProvider:202 - > Unable to get JMX metrics. No host name for NAMENODE > 16 Oct 2015 01:34:00,930 INFO [qtp-client-72] PersistKeyValueService:82 - > Looking for keyName admin-settings-show-bg-admin > 16 Oct 2015 01:34:01,010 INFO [qtp-client-76] PersistKeyValueService:82 - > Looking for keyName hostPopup-pagination-displayLength-admin > 16 Oct 2015 01:36:34,123 INFO [qtp-client-82] PersistKeyValueService:82 - > Looking for keyName time-range-service-AMBARI_METRICS > 16 Oct 2015 01:36:42,903 INFO [qtp-client-125] PersistKeyValueService:82 > - Looking for keyName time-range-service-HDFS > 16 Oct 2015 01:36:44,138 WARN [pool-7-thread-35] JMXPropertyProvider:202 > - Unable to get JMX metrics. No host name for NAMENODE > 16 Oct 2015 01:36:57,590 INFO [qtp-client-75] PersistKeyValueService:82 - > Looking for keyName admin-settings-show-bg-admin > 16 Oct 2015 01:48:10,901 INFO [qtp-client-192] PersistKeyValueService:82 > - Looking for keyName admin-settings-show-bg-admin > 16 Oct 2015 01:53:06,487 INFO [qtp-client-214] PersistKeyValueService:82 > - Looking for keyName user-pref-admin-dashboard > > > > After that last message there was another 15 minute delay and then nothing > (another sixty minutes has passed).. > > How to determine what is happening here and why? > > >
