I changed the connection_config.json to reference node1 instead or localhost. I also restarted the opentaxii service.
I was able to get further in the load but then yarn, hdfs, and hbase started dying randomly. I think taxii oversubscribes the VM. Thanks Carolyn On 10/4/16, 2:17 PM, "Carolyn Duby" <[email protected]> wrote: > > >From: Carolyn Duby <[email protected]<mailto:[email protected]>> >Date: Tuesday, October 4, 2016 at 2:15 PM >To: Anand Subramanian ><[email protected]<mailto:[email protected]>>, Nicholas >Allen <[email protected]<mailto:[email protected]>> >Subject: taxi service doesn't respond > > >I loaded up phishtank_com: > > >[root@node1 ~]# service opentaxii status > >Checking opentaxii... Running > >guest.phishtank_com 21779 > >guest.Abuse_ch 0 > > >[root@node1 ~]# cat connection_config.json > >{ > > "endpoint" : "http://localhost:9000/services/discovery"; > > ,"username" : "guest" > > ,"password" : "guest" > > ,"type" : "DISCOVER" > > ,"collection" : "guest.phishtank_com" > > ,"table" : "threatintel" > > ,"columnFamily" : "t" > > ,"allowedIndicatorTypes" : [ "domainname:FQDN", "address:IPV_4_ADDR" ] > >} > > >[root@node1 ~]# cat extractor.json > >{ > > "config": { > > "columns": { > > "ip": 0 > > }, > > "indicator_column": "ip", > > "type" : "malicious_ip", > > "separator" : "," > > }, > > "extractor" : "STIX" > >} > > >The Taxii load fails with the following error: > > >[root@node1 ~]# /usr/metron/0.2.0BETA/bin/threatintel_taxii_load.sh -b >"2016-08-01 00:00:00" -c ~/connection_config.json -e ~/extractor.json -p 10000 > >WARNING: Use "yarn jar" to launch YARN applications. > >16/10/04 18:06:44 INFO taxii.TaxiiHandler: Loading configuration: >TaxiiConnectionConfig{endpoint=http://localhost:9000/services/discovery, >port=443, proxy=null, username='guest', password='******', type=DISCOVER, >allowedIndicatorTypes=domainname:FQDN,address:IPV_4_ADDR, >collection='guest.phishtank_com', subscriptionId='null', beginTime=Mon Aug 01 >00:00:00 UTC 2016, table=threatintel:t} > >16/10/04 18:06:44 INFO taxii.TaxiiHandler: Initializing client.. > >16/10/04 18:06:44 INFO taxii.TaxiiHandler: Discovering endpoint > >16/10/04 18:06:47 INFO taxii.TaxiiHandler: Request made : >org.mitre.taxii.messages.xml11.DiscoveryRequest => >org.mitre.taxii.messages.xml11.DiscoveryResponse (expected >org.mitre.taxii.messages.xml11.DiscoveryResponse) > >16/10/04 18:06:47 INFO taxii.TaxiiHandler: Discovered endpoint as >http://localhost:9000/services/poll > >16/10/04 18:06:48 INFO taxii.TaxiiHandler: Configured, starting polling >http://localhost:9000/services/poll for guest.phishtank_com > >16/10/04 18:06:48 INFO taxii.TaxiiHandler: Polling...10/4/16 6:06 PM > >16/10/04 18:06:48 INFO taxii.TaxiiHandler: Begin Time: 2016-08-01T00:00:00Z > >16/10/04 18:07:11 ERROR taxii.TaxiiHandler: The target server failed to respond > >org.apache.metron.httpcore.dataload.NoHttpResponseException: The target server >failed to respond > >at(DefaultHttpResponseParser.java:143) > >at >org.apache.metron.httpcore.dataload.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) > >at >org.apache.metron.httpcore.dataload.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260) > >at >org.apache.metron.httpcore.dataload.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:161) > >at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >at >sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > >at >sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >at java.lang.reflect.Method.invoke(Method.java:497) > >at >org.apache.metron.httpcore.dataload.impl.conn.CPoolProxy.invoke(CPoolProxy.java:138) > >at com.sun.proxy.$Proxy33.receiveResponseHeader(Unknown Source) > >at >org.apache.metron.httpcore.dataload.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271) > >at >org.apache.metron.httpcore.dataload.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) > >at >org.apache.metron.httpcore.dataload.impl.execchain.MainClientExec.execute(MainClientExec.java:254) > >at >org.apache.metron.httpcore.dataload.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195) > >at >org.apache.metron.httpcore.dataload.impl.execchain.RetryExec.execute(RetryExec.java:85) > >at >org.apache.metron.httpcore.dataload.impl.execchain.RedirectExec.execute(RedirectExec.java:108) > >at >org.apache.metron.httpcore.dataload.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186) > >at >org.apache.metron.httpcore.dataload.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) > >at org.mitre.taxii.client.HttpClient.callTaxiiService(HttpClient.java:297) > >at >org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.call(TaxiiHandler.java:336) > >at >org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.call(TaxiiHandler.java:242) > >at >org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.run(TaxiiHandler.java:171) > >at java.util.TimerThread.mainLoop(Timer.java:555) > >at java.util.TimerThread.run(Timer.java:505) > >Exception in thread "Timer-0" java.lang.RuntimeException: Unable to make >request > >at >org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.run(TaxiiHandler.java:214) > >at java.util.TimerThread.mainLoop(Timer.java:555) > >at java.util.TimerThread.run(Timer.java:505) > >Caused by: org.apache.metron.httpcore.dataload.NoHttpResponseException: The >target server failed to respond > >at >org.apache.metron.httpcore.dataload.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143) > >at >org.apache.metron.httpcore.dataload.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) > >at >org.apache.metron.httpcore.dataload.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260) > >at >org.apache.metron.httpcore.dataload.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:161) > >at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >at >sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > >at >sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >at java.lang.reflect.Method.invoke(Method.java:497) > >at >org.apache.metron.httpcore.dataload.impl.conn.CPoolProxy.invoke(CPoolProxy.java:138) > >at com.sun.proxy.$Proxy33.receiveResponseHeader(Unknown Source) > >at >org.apache.metron.httpcore.dataload.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271) > >at >org.apache.metron.httpcore.dataload.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) > >at >org.apache.metron.httpcore.dataload.impl.execchain.MainClientExec.execute(MainClientExec.java:254) > >at >org.apache.metron.httpcore.dataload.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195) > >at >org.apache.metron.httpcore.dataload.impl.execchain.RetryExec.execute(RetryExec.java:85) > >at >org.apache.metron.httpcore.dataload.impl.execchain.RedirectExec.execute(RedirectExec.java:108) > >at >org.apache.metron.httpcore.dataload.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186) > >at >org.apache.metron.httpcore.dataload.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) > >at org.mitre.taxii.client.HttpClient.callTaxiiService(HttpClient.java:297) > >at >org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.call(TaxiiHandler.java:336) > >at >org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.call(TaxiiHandler.java:242) > >at >org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.run(TaxiiHandler.java:171) > >... 2 more >
