[
https://issues.apache.org/jira/browse/METRON-527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15622998#comment-15622998
]
ASF GitHub Bot commented on METRON-527:
---------------------------------------
Github user justinleet commented on the issue:
https://github.com/apache/incubator-metron/pull/332
The Taxii HttpClient closes connections in callTaxiiService as seen here:
https://github.com/TAXIIProject/java-taxii/blob/master/src/main/java/org/mitre/taxii/client/HttpClient.java#L335
Because TaxiiHandler attempted to reuse a closed connection, the error was
seen. The fix is just to ensure the connection is created fresh before each
call. Integration test is updated to just call the run() method twice without
caring about results to ensure that the connection doesn't explode.
In addition, this was tested in quick-dev using a slightly modified version
(to work with Vagrant) of
https://community.hortonworks.com/articles/59698/pushing-stixtaxii-feeds-from-opentaxii-server-into.html,
where the playbook was run with
`ansible-playbook
--private-key=./vagrant/quick-dev-platform/.vagrant/machines/node1/virtualbox/private_key
-i 192.168.66.121, playbooks/install-opentaxii.yml -e
ansible_python_interpreter=python -e ansible_user=vagrant -vvvv`
with the IP being quick-dev's IP and the install-opentaxii script modified
to reflect this.
After the change, polling continues until ended and hbase dumps grow as new
records are added.
> Connection pool shut down seen when running threatintel_taxii_load.sh script
> ----------------------------------------------------------------------------
>
> Key: METRON-527
> URL: https://issues.apache.org/jira/browse/METRON-527
> Project: Metron
> Issue Type: Bug
> Reporter: Justin Leet
> Assignee: Justin Leet
>
> Discovered by [~anandsubbu]:
> I have setup opentaxii server on a 12-node metron cluster. I was able to scan
> and pull records into the taxi server.
> {code}
> [root@metron-test2-8 ~]# service opentaxii status
> Checking opentaxii... Running
> guest.phishtank_com 5891
> guest.Abuse_ch 49
> guest.CyberCrime_Tracker 0
> guest.EmergingThreats_rules 0
> guest.Lehigh_edu 2025
> guest.MalwareDomainList_Hostlist 50
> guest.blutmagie_de_torExits 14897
> guest.dataForLast_7daysOnly 23210
> guest.dshield_BlockList 0
> {code}
> When I ran the threatintel_taxii_load.sh script, it is able to push some of
> the records into hbase 'threatintel' table, but fails with a "Connection pool
> shut down" error shortly after. This was observed for all the subscriptions
> that I attempted.
> Here is the command that was used:
> {code}
> [root@metron-test2-8 ~]# /usr/metron/0.2.0BETA/bin//threatintel_taxii_load.sh
> -b "2015-01-01 00:00:00" -c ~/connection_config.json -e ~/extractor.json -p
> 300000
> {code}
> The error message is pasted below.
> {code}
> 16/08/30 09:24:14 ERROR taxii.TaxiiHandler: Connection pool shut down
> java.lang.IllegalStateException: Connection pool shut down
> at
> org.apache.metron.httpcore.dataload.util.Asserts.check(Asserts.java:34)
> at
> org.apache.metron.httpcore.dataload.pool.AbstractConnPool.lease(AbstractConnPool.java:169)
> at
> org.apache.metron.httpcore.dataload.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:217)
> at
> org.apache.metron.httpcore.dataload.impl.execchain.MainClientExec.execute(MainClientExec.java:158)
> at
> org.apache.metron.httpcore.dataload.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)
> at
> org.apache.metron.httpcore.dataload.impl.execchain.RetryExec.execute(RetryExec.java:85)
> at
> org.apache.metron.httpcore.dataload.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
> at
> org.apache.metron.httpcore.dataload.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
> at
> org.apache.metron.httpcore.dataload.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> at
> org.mitre.taxii.client.HttpClient.callTaxiiService(HttpClient.java:297)
> at
> org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.call(TaxiiHandler.java:336)
> at
> org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.call(TaxiiHandler.java:242)
> at
> org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.run(TaxiiHandler.java:171)
> at java.util.TimerThread.mainLoop(Timer.java:555)
> at java.util.TimerThread.run(Timer.java:505)
> Exception in thread "Timer-0" java.lang.RuntimeException: Unable to make
> request
> at
> org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.run(TaxiiHandler.java:214)
> at java.util.TimerThread.mainLoop(Timer.java:555)
> at java.util.TimerThread.run(Timer.java:505)
> Caused by: java.lang.IllegalStateException: Connection pool shut down
> at
> org.apache.metron.httpcore.dataload.util.Asserts.check(Asserts.java:34)
> at
> org.apache.metron.httpcore.dataload.pool.AbstractConnPool.lease(AbstractConnPool.java:169)
> at
> org.apache.metron.httpcore.dataload.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:217)
> at
> org.apache.metron.httpcore.dataload.impl.execchain.MainClientExec.execute(MainClientExec.java:158)
> at
> org.apache.metron.httpcore.dataload.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)
> at
> org.apache.metron.httpcore.dataload.impl.execchain.RetryExec.execute(RetryExec.java:85)
> at
> org.apache.metron.httpcore.dataload.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
> at
> org.apache.metron.httpcore.dataload.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
> at
> org.apache.metron.httpcore.dataload.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> at
> org.mitre.taxii.client.HttpClient.callTaxiiService(HttpClient.java:297)
> at
> org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.call(TaxiiHandler.java:336)
> at
> org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.call(TaxiiHandler.java:242)
> at
> org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.run(TaxiiHandler.java:171)
> ... 2 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
