Ok, I think I see what my problem is. First, my falcon-prism/conf/runtime.properties had:
prism.falcon.local.endpoint=http://localhost:16444 which pointed back at itself, not: prism.falcon.local.endpoint=http://localhost:15000 which points at falcon server. So I fixed that and restarted falcon-prism. Now I get a different error: POST http://localhost:16444/api/entities/submit/cluster returned a response status of 400 Bad Request ERROR: Bad Request;local/org.apache.falcon.FalconException::org.apache.falcon.FalconException: <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/> <title>Error 503 SERVICE_UNAVAILABLE</title> </head> <body> <h2>HTTP ERROR: 503</h2> <p>Problem accessing /secure/sync/submit/cluster. Reason: <pre> SERVICE_UNAVAILABLE</pre></p> <hr /><i><small>Powered by Jetty://</small></i> The prism.application.log now says it's trying to talk to falcon-server: 2017-01-05 10:12:53,952 DEBUG - [352083716@qtp-771418758-0 - 9149b35e-3554-4775-bbd3-5ce30761ead8:aaron_turner:POST//entities/submit/cluster] ~ Executing http://localhost:15000/secure/sync/submit/cluster?colo=local&; (HTTPChannel:96) 2017-01-05 10:12:53,953 INFO - [352083716@qtp-771418758-0 - 9149b35e-3554-4775-bbd3-5ce30761ead8:aaron_turner:POST//entities/submit/cluster] ~ Configuring client with /usr/lib/falcon-prism/conf/prism.keystore (SecureHTTPChannel:56) 2017-01-05 10:12:54,033 ERROR - [352083716@qtp-771418758-0 - 9149b35e-3554-4775-bbd3-5ce30761ead8:aaron_turner:POST//entities/submit/cluster] ~ Request failed: 503 (HTTPChannel:127) 2017-01-05 10:12:54,035 ERROR - [352083716@qtp-771418758-0 - 9149b35e-3554-4775-bbd3-5ce30761ead8:aaron_turner:POST//entities/submit/cluster] ~ Request failed (HTTPChannel:134) Trying in my browser to connect to tcp/15000 gives me the same 503 error. So I went looking in the falcon.application.log for what the problem is: 2017-01-05 10:09:05,551 INFO - [main:] ~ Initializing graph db (MetadataMappingService:143) 2017-01-05 10:09:05,610 INFO - [main:] ~ The HBase version 1.2.0-cdh5.8.2 is not explicitly supported by Titan. Loading Titan's compatibility layer for its most recent supported HBase version (1.1) (HBaseCompatLoader:57) <snip> 2017-01-05 10:09:05,832 WARN - [main:] ~ hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size (HeapMemorySizeUtil:55) 2017-01-05 10:09:06,084 WARN - [main:] ~ The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. (DomainSocketFactory:117) 2017-01-05 10:09:06,088 INFO - [main:] ~ Set a new configuration for the first time. (N:110) 2017-01-05 10:09:06,119 INFO - [main:] ~ mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id (deprecation:1174) 2017-01-05 10:09:06,120 INFO - [main:] ~ Scheduling statistics report every 2000 millisecs (u:110) 2017-01-05 10:09:06,120 INFO - [main:] ~ Initialized the distributed file system (ThrottledDistributedFileSystem:110) 2017-01-05 10:09:06,421 INFO - [main:] ~ hadoop.native.lib is deprecated. Instead, use io.native.lib.available (deprecation:1174) 2017-01-05 10:09:06,514 WARN - [main:] ~ hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size (HeapMemorySizeUtil:55) 2017-01-05 10:09:06,521 ERROR - [main:] ~ Failed to initialize service org.apache.falcon.metadata.MetadataMappingService (ServiceInitializer:49) java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [com.thinkaurelius.titan.core.TitanFactory]. at com.tinkerpop.blueprints.GraphFactory.open(GraphFactory.java:50) at org.apache.falcon.metadata.MetadataMappingService.initializeGraphDB(MetadataMappingService.java:146) at org.apache.falcon.metadata.MetadataMappingService.init(MetadataMappingService.java:113) at org.apache.falcon.service.ServiceInitializer.initialize(ServiceInitializer.java:47) at org.apache.falcon.listener.ContextStartupListener.contextInitialized(ContextStartupListener.java:56) at org.mortbay.jetty.handler.ContextHandler.startContext(ContextHandler.java:550) at org.mortbay.jetty.servlet.Context.startContext(Context.java:136) at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282) at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:519) at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130) at org.mortbay.jetty.Server.doStart(Server.java:224) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at org.apache.falcon.util.EmbeddedServer.start(EmbeddedServer.java:58) at org.apache.falcon.FalconServer.main(FalconServer.java:118) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.tinkerpop.blueprints.GraphFactory.open(GraphFactory.java:45) ... 15 more Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.hbase.HTableDescriptor.addFamily(Lorg/apache/hadoop/hbase/HColumnDescriptor;)Lorg/apache/hadoop/hbase/HTableDescriptor; at com.thinkaurelius.titan.diskstorage.hbase.HBaseCompat1_1.addColumnFamilyToTableDescriptor(HBaseCompat1_1.java:49) at com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.createTable(HBaseStoreManager.java:728) at com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.ensureTableExists(HBaseStoreManager.java:711) at com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.getLocalKeyPartition(HBaseStoreManager.java:511) at com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.getDeployment(HBaseStoreManager.java:352) at com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.getFeatures(HBaseStoreManager.java:395) at com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration.<init>(GraphDatabaseConfiguration.java:1276) at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:93) at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:73) ... 20 more 2017-01-05 10:09:06,524 ERROR - [main:] ~ Failed startup of context org.mortbay.jetty.webapp.WebAppContext@6caf7803{/,/usr/lib/falcon-server/server/webapp/falcon} (log:87) java.lang.RuntimeException: org.apache.falcon.FalconException: java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [com.thinkaurelius.titan.core.TitanFactory]. at org.apache.falcon.listener.ContextStartupListener.contextInitialized(ContextStartupListener.java:59) at org.mortbay.jetty.handler.ContextHandler.startContext(ContextHandler.java:550) at org.mortbay.jetty.servlet.Context.startContext(Context.java:136) at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282) at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:519) at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130) at org.mortbay.jetty.Server.doStart(Server.java:224) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at org.apache.falcon.util.EmbeddedServer.start(EmbeddedServer.java:58) at org.apache.falcon.FalconServer.main(FalconServer.java:118) Caused by: org.apache.falcon.FalconException: java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [com.thinkaurelius.titan.core.TitanFactory]. at org.apache.falcon.service.ServiceInitializer.initialize(ServiceInitializer.java:50) at org.apache.falcon.listener.ContextStartupListener.contextInitialized(ContextStartupListener.java:56) ... 11 more Caused by: java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [com.thinkaurelius.titan.core.TitanFactory]. at com.tinkerpop.blueprints.GraphFactory.open(GraphFactory.java:50) at org.apache.falcon.metadata.MetadataMappingService.initializeGraphDB(MetadataMappingService.java:146) at org.apache.falcon.metadata.MetadataMappingService.init(MetadataMappingService.java:113) at org.apache.falcon.service.ServiceInitializer.initialize(ServiceInitializer.java:47) ... 12 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.tinkerpop.blueprints.GraphFactory.open(GraphFactory.java:45) ... 15 more Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.hbase.HTableDescriptor.addFamily(Lorg/apache/hadoop/hbase/HColumnDescriptor;)Lorg/apache/hadoop/hbase/HTableDescriptor; at com.thinkaurelius.titan.diskstorage.hbase.HBaseCompat1_1.addColumnFamilyToTableDescriptor(HBaseCompat1_1.java:49) at com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.createTable(HBaseStoreManager.java:728) at com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.ensureTableExists(HBaseStoreManager.java:711) at com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.getLocalKeyPartition(HBaseStoreManager.java:511) at com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.getDeployment(HBaseStoreManager.java:352) at com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.getFeatures(HBaseStoreManager.java:395) at com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration.<init>(GraphDatabaseConfiguration.java:1276) at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:93) at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:73) ... 20 more 2017-01-05 10:09:06,525 INFO - [main:] ~ Started SocketConnector@0.0.0.0:15000 (log:67) 2017-01-05 10:14:04,983 INFO - [Thread-16:] ~ config.location is set, using: /usr/lib/falcon-server/conf/runtime.properties (ApplicationProperties:120) 2017-01-05 10:14:04,983 INFO - [Thread-16:] ~ Initializing org.apache.falcon.util.RuntimeProperties properties with domain falcon (ApplicationProperties:156) 2017-01-05 10:14:04,983 DEBUG - [Thread-16:] ~ feed.retry.policy=exp-backoff (ApplicationProperties:162) 2017-01-05 10:14:04,983 DEBUG - [Thread-16:] ~ feed.retry.frequency=minutes(5) (ApplicationProperties:162) 2017-01-05 10:14:04,983 DEBUG - [Thread-16:] ~ workflow.status.retry.count=30 (ApplicationProperties:162) 2017-01-05 10:14:04,983 DEBUG - [Thread-16:] ~ falcon.service.ProxyUserService.proxyuser.#USER#.hosts=* (ApplicationProperties:162) 2017-01-05 10:14:04,984 DEBUG - [Thread-16:] ~ feed.late.allowed=true (ApplicationProperties:162) 2017-01-05 10:14:04,984 DEBUG - [Thread-16:] ~ webservices.default.results.per.page=10 (ApplicationProperties:162) 2017-01-05 10:14:04,984 DEBUG - [Thread-16:] ~ feed.retry.onTimeout=false (ApplicationProperties:162) 2017-01-05 10:14:04,984 DEBUG - [Thread-16:] ~ falcon.jms.notification.enabled=true (ApplicationProperties:162) 2017-01-05 10:14:04,984 DEBUG - [Thread-16:] ~ falcon.retention.keep.instances.beyond.validity=true (ApplicationProperties:162) 2017-01-05 10:14:04,984 DEBUG - [Thread-16:] ~ feed.late.policy=exp-backoff (ApplicationProperties:162) 2017-01-05 10:14:04,984 DEBUG - [Thread-16:] ~ current.colo=local (ApplicationProperties:162) 2017-01-05 10:14:04,984 DEBUG - [Thread-16:] ~ domain=falcon (ApplicationProperties:162) 2017-01-05 10:14:04,984 DEBUG - [Thread-16:] ~ feed.late.frequency=hours(3) (ApplicationProperties:162) 2017-01-05 10:14:04,984 DEBUG - [Thread-16:] ~ falcon.skip.dryrun=false (ApplicationProperties:162) 2017-01-05 10:14:04,984 DEBUG - [Thread-16:] ~ feed.retry.attempts=3 (ApplicationProperties:162) 2017-01-05 10:14:04,984 DEBUG - [Thread-16:] ~ falcon.service.ProxyUserService.proxyuser.#USER#.groups=* (ApplicationProperties:162) 2017-01-05 10:14:04,984 INFO - [Thread-16:] ~ config.location is set, using: /usr/lib/falcon-server/conf/runtime.properties (ApplicationProperties:120) 2017-01-05 10:14:04,985 INFO - [Thread-16:] ~ Initializing org.apache.falcon.util.RuntimeProperties properties with domain falcon (ApplicationProperties:156) 2017-01-05 10:14:04,985 DEBUG - [Thread-16:] ~ feed.retry.policy=exp-backoff (ApplicationProperties:162) 2017-01-05 10:14:04,985 DEBUG - [Thread-16:] ~ feed.retry.frequency=minutes(5) (ApplicationProperties:162) 2017-01-05 10:14:04,985 DEBUG - [Thread-16:] ~ workflow.status.retry.count=30 (ApplicationProperties:162) 2017-01-05 10:14:04,985 DEBUG - [Thread-16:] ~ falcon.service.ProxyUserService.proxyuser.#USER#.hosts=* (ApplicationProperties:162) 2017-01-05 10:14:04,985 DEBUG - [Thread-16:] ~ feed.late.allowed=true (ApplicationProperties:162) 2017-01-05 10:14:04,985 DEBUG - [Thread-16:] ~ webservices.default.results.per.page=10 (ApplicationProperties:162) 2017-01-05 10:14:04,985 DEBUG - [Thread-16:] ~ feed.retry.onTimeout=false (ApplicationProperties:162) 2017-01-05 10:14:04,985 DEBUG - [Thread-16:] ~ falcon.jms.notification.enabled=true (ApplicationProperties:162) 2017-01-05 10:14:04,985 DEBUG - [Thread-16:] ~ falcon.retention.keep.instances.beyond.validity=true (ApplicationProperties:162) 2017-01-05 10:14:04,985 DEBUG - [Thread-16:] ~ feed.late.policy=exp-backoff (ApplicationProperties:162) 2017-01-05 10:14:04,985 DEBUG - [Thread-16:] ~ current.colo=local (ApplicationProperties:162) 2017-01-05 10:14:04,985 DEBUG - [Thread-16:] ~ domain=falcon (ApplicationProperties:162) 2017-01-05 10:14:04,985 DEBUG - [Thread-16:] ~ feed.late.frequency=hours(3) (ApplicationProperties:162) 2017-01-05 10:14:04,985 DEBUG - [Thread-16:] ~ falcon.skip.dryrun=false (ApplicationProperties:162) 2017-01-05 10:14:04,985 DEBUG - [Thread-16:] ~ feed.retry.attempts=3 (ApplicationProperties:162) 2017-01-05 10:14:04,985 DEBUG - [Thread-16:] ~ falcon.service.ProxyUserService.proxyuser.#USER#.groups=* (ApplicationProperties:162) So it appears my problem is that CDH5.8.2 is shipping a more recent version of Hbase then Titan supports? Not sure if there is a work around or should I switch to BerkeleyDB?
