[jira] [Commented] (METRON-484) Opentaxi service does not show count for subscribed services
[
https://issues.apache.org/jira/browse/METRON-484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15650941#comment-15650941
]
Anand Subramanian commented on METRON-484:
--
This defect was verified off the latest bits and the service counts are now
coming up fine. Here's the sample output.
{code}
[root@hcsa-12 ~]# service opentaxii status
Checking opentaxii... Running
guest.phishtank_com184
guest.Abuse_ch 0
guest.CyberCrime_Tracker 0
guest.EmergingThreats_rules0
guest.Lehigh_edu 0
guest.MalwareDomainList_Hostlist 0
guest.blutmagie_de_torExits469
guest.dataForLast_7daysOnly653
guest.dshield_BlockList0
{code}
Here's the build info against which this was verified:
{code}
commit e1401d01ecc87e652931aaada4bc7500df984da2
Author: JonZeolla
Date: Fri Nov 4 09:16:37 2016 -0400
METRON-510 Update elasticsearch bro templates for *_body_len (JonZeolla via
dlyle65535) closes apache/incubator-metron#326
commit b390c6d16a43a5907a742ab3690cf29bb76ea80d
Author: dlyle65535
Date: Fri Nov 4 09:12:28 2016 -0400
METRON-463 Pull RPMs from Remote (configurable) repos closes
apache/incubator-metron#339
{code}
> Opentaxi service does not show count for subscribed services
> --
>
> Key: METRON-484
> URL: https://issues.apache.org/jira/browse/METRON-484
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.2.1BETA
> Environment: 12 node cluster running CentOS 7 on Openstack.
>Reporter: Anand Subramanian
>Assignee: Nick Allen
> Fix For: 0.2.2BETA
>
>
> *Steps Followed*
> a) Created an ansible playbook for 'opentaxii' role and installed on a
> pre-deployed 12-node cluster with the following playbook definition:
> {code}
> - hosts: metron
> become: true
> roles:
> - role: opentaxii
> {code}
> b) Followed the documentation at:
> https://github.com/apache/incubator-metron/tree/master/metron-deployment/roles/opentaxii#getting-started
> c) Ensure that the opentaxii service is installed on the 'metron' nodes viz.
> 1, 3, 8, 10, 11 and 12.
> Observed on all the nodes that the 'status' command output is as follows:
> {code}
> [root@metron-test-1 ~]# service opentaxii status
> Checking opentaxii... Running
> Services not defined
> {code}
> *Workaround*
> In order to get around this issue, edit the file at
> /usr/local/opentaxii/bin/collection-status.py and change it to look like the
> below.
> {code}
>
> from cabby import create_client
> base_uri = 'http://localhost:9000'
> try:
> # create a connection
> client =
> create_client(host='localhost',port=9000,discovery_path='/services/discovery')
> # iterate through each defined collection
> collections =
> client.get_collections(uri='%s/services/collection'%base_uri)
> for collection in collections:
> # how many records in each collection?
> count = client.get_content_count(collection_name=collection.name,
> uri='%s/services/poll'%base_uri)
> print "%-50s %-10d" % (collection.name, count.count)
> except:
> print "Services not defined"
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (METRON-484) Opentaxi service does not show count for subscribed services
[
https://issues.apache.org/jira/browse/METRON-484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15625648#comment-15625648
]
ASF GitHub Bot commented on METRON-484:
---
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/306
> Opentaxi service does not show count for subscribed services
> --
>
> Key: METRON-484
> URL: https://issues.apache.org/jira/browse/METRON-484
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.2.1BETA
> Environment: 12 node cluster running CentOS 7 on Openstack.
>Reporter: Anand Subramanian
>
> *Steps Followed*
> a) Created an ansible playbook for 'opentaxii' role and installed on a
> pre-deployed 12-node cluster with the following playbook definition:
> {code}
> - hosts: metron
> become: true
> roles:
> - role: opentaxii
> {code}
> b) Followed the documentation at:
> https://github.com/apache/incubator-metron/tree/master/metron-deployment/roles/opentaxii#getting-started
> c) Ensure that the opentaxii service is installed on the 'metron' nodes viz.
> 1, 3, 8, 10, 11 and 12.
> Observed on all the nodes that the 'status' command output is as follows:
> {code}
> [root@metron-test-1 ~]# service opentaxii status
> Checking opentaxii... Running
> Services not defined
> {code}
> *Workaround*
> In order to get around this issue, edit the file at
> /usr/local/opentaxii/bin/collection-status.py and change it to look like the
> below.
> {code}
>
> from cabby import create_client
> base_uri = 'http://localhost:9000'
> try:
> # create a connection
> client =
> create_client(host='localhost',port=9000,discovery_path='/services/discovery')
> # iterate through each defined collection
> collections =
> client.get_collections(uri='%s/services/collection'%base_uri)
> for collection in collections:
> # how many records in each collection?
> count = client.get_content_count(collection_name=collection.name,
> uri='%s/services/poll'%base_uri)
> print "%-50s %-10d" % (collection.name, count.count)
> except:
> print "Services not defined"
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (METRON-484) Opentaxi service does not show count for subscribed services
[
https://issues.apache.org/jira/browse/METRON-484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15622281#comment-15622281
]
ASF GitHub Bot commented on METRON-484:
---
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/306
This got no love, sorry @nickwallen +1 by inspection
> Opentaxi service does not show count for subscribed services
> --
>
> Key: METRON-484
> URL: https://issues.apache.org/jira/browse/METRON-484
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.2.1BETA
> Environment: 12 node cluster running CentOS 7 on Openstack.
>Reporter: Anand Subramanian
>
> *Steps Followed*
> a) Created an ansible playbook for 'opentaxii' role and installed on a
> pre-deployed 12-node cluster with the following playbook definition:
> {code}
> - hosts: metron
> become: true
> roles:
> - role: opentaxii
> {code}
> b) Followed the documentation at:
> https://github.com/apache/incubator-metron/tree/master/metron-deployment/roles/opentaxii#getting-started
> c) Ensure that the opentaxii service is installed on the 'metron' nodes viz.
> 1, 3, 8, 10, 11 and 12.
> Observed on all the nodes that the 'status' command output is as follows:
> {code}
> [root@metron-test-1 ~]# service opentaxii status
> Checking opentaxii... Running
> Services not defined
> {code}
> *Workaround*
> In order to get around this issue, edit the file at
> /usr/local/opentaxii/bin/collection-status.py and change it to look like the
> below.
> {code}
>
> from cabby import create_client
> base_uri = 'http://localhost:9000'
> try:
> # create a connection
> client =
> create_client(host='localhost',port=9000,discovery_path='/services/discovery')
> # iterate through each defined collection
> collections =
> client.get_collections(uri='%s/services/collection'%base_uri)
> for collection in collections:
> # how many records in each collection?
> count = client.get_content_count(collection_name=collection.name,
> uri='%s/services/poll'%base_uri)
> print "%-50s %-10d" % (collection.name, count.count)
> except:
> print "Services not defined"
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (METRON-484) Opentaxi service does not show count for subscribed services
[ https://issues.apache.org/jira/browse/METRON-484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15569026#comment-15569026 ] ASF GitHub Bot commented on METRON-484: --- GitHub user nickwallen opened a pull request: https://github.com/apache/incubator-metron/pull/306 METRON-484 Opentaxii service does not show count for subscribed services [METRON-484](https://issues.apache.org/jira/browse/METRON-484) This PR was a collaboration between myself and @nsinha17. There were recent changes in the Cabby Python module used as part of the deployed Opentaxii service. These changes broke the service script which prevented it from listing the number of threat indicators for each collection. ### Testing Deployed Opentaxii on Quick-Dev. Able to list counts by service and load threat data. See transcript below. ``` [root@node1 ~]# service opentaxii status Checking opentaxii... Running guest.phishtank_com0 guest.Abuse_ch 0 guest.CyberCrime_Tracker 0 guest.EmergingThreats_rules0 guest.Lehigh_edu 0 guest.MalwareDomainList_Hostlist 0 guest.blutmagie_de_torExits0 guest.dataForLast_7daysOnly0 guest.dshield_BlockList0 [root@node1 ~]# service opentaxii sync guest.phishtank_com + /usr/local/opentaxii/opentaxii-venv/bin/taxii-proxy --poll-path http://hailataxii.com/taxii-data --poll-collection guest.phishtank_com --inbox-path http://localhost:9000/services/inbox --inbox-collection guest.phishtank_com --binding urn:stix.mitre.org:xml:1.1.1 --begin 2016-10-12 --end 2016-10-13 2016-10-12 15:19:28,701 INFO: Sending Poll_Request to http://hailataxii.com/taxii-data 2016-10-12 15:19:28,704 INFO: Starting new HTTP connection (1): hailataxii.com 2016-10-12 15:19:28,942 INFO: Response received for Poll_Request from http://hailataxii.com/taxii-data 2016-10-12 15:19:29,046 INFO: Sending Inbox_Message to http://localhost:9000/services/inbox 2016-10-12 15:19:29,048 INFO: Starting new HTTP connection (1): localhost ... 2016-10-12 15:20:28,317 INFO: 1762 blocks polled 2016-10-12 15:20:28,317 INFO: Content block successfully pushed + set +x 2016-10-12 15:20:28,436 INFO: Polling using data binding: ALL 2016-10-12 15:20:28,441 INFO: Sending Discovery_Request to http://localhost:9000/services/discovery 2016-10-12 15:20:28,444 INFO: Starting new HTTP connection (1): localhost 2016-10-12 15:20:28,450 INFO: Response received for Discovery_Request from http://localhost:9000/services/discovery 2016-10-12 15:20:28,451 INFO: 3 services discovered 2016-10-12 15:20:28,452 INFO: Sending Poll_Request to http://localhost:9000/services/poll 2016-10-12 15:20:28,453 INFO: Starting new HTTP connection (1): localhost 2016-10-12 15:20:28,462 INFO: Response received for Poll_Request from http://localhost:9000/services/poll 2016-10-12 15:20:28,462 INFO: Content blocks count: 1762, is partial: False [root@node1 ~]# service opentaxii status Checking opentaxii... Running guest.phishtank_com1762 guest.Abuse_ch 0 guest.CyberCrime_Tracker 0 guest.EmergingThreats_rules0 guest.Lehigh_edu 0 guest.MalwareDomainList_Hostlist 0 guest.blutmagie_de_torExits0 guest.dataForLast_7daysOnly0 guest.dshield_BlockList0 ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/nickwallen/incubator-metron METRON-484 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/306.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #306 commit ee65589eea36be441cd19709524a4ba015b937a8 Author: nsinha17 Date: 2016-10-05T06:36:52Z METRON-484: Opentaxi service does not show count for subscribed services commit 3e0b05e1858c4173fb3078fc36c4499421ba8fe4 Author: nsinha17 Date: 2016-10-05T08:49:49Z METRON-484: Opentaxi service does not show count for subscribed services commit d538fcdc2b29111d9949cb6e1cd9a6c5c1372527 Author: Nick Allen Date: 2016-10-12T15:01:04Z METRON-484 Allow use of http, https, or ftp and set bind variable appropriately commit c982f68b6c02f3da997ce51a05bcc83aa0ad3f27 A
[jira] [Commented] (METRON-484) Opentaxi service does not show count for subscribed services
[
https://issues.apache.org/jira/browse/METRON-484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15546162#comment-15546162
]
Nick Allen commented on METRON-484:
---
This fix does indeed work for me. Yay! The Cabby API changed since the
original implementation which is the cause of this bug.
It seems like we should not have to specify uri="..." on each subsequent call
using the client. Unfortunately there still seems to be a bug in the Cabby
API. The host and port specified in the call to 'create_client' is not carried
through to each subsequent call using that client. This forces us to use
uri="...".
> Opentaxi service does not show count for subscribed services
> --
>
> Key: METRON-484
> URL: https://issues.apache.org/jira/browse/METRON-484
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.2.1BETA
> Environment: 12 node cluster running CentOS 7 on Openstack.
>Reporter: Anand Subramanian
>
> *Steps Followed*
> a) Created an ansible playbook for 'opentaxii' role and installed on a
> pre-deployed 12-node cluster with the following playbook definition:
> {code}
> - hosts: metron
> become: true
> roles:
> - role: opentaxii
> {code}
> b) Followed the documentation at:
> https://github.com/apache/incubator-metron/tree/master/metron-deployment/roles/opentaxii#getting-started
> c) Ensure that the opentaxii service is installed on the 'metron' nodes viz.
> 1, 3, 8, 10, 11 and 12.
> Observed on all the nodes that the 'status' command output is as follows:
> {code}
> [root@metron-test-1 ~]# service opentaxii status
> Checking opentaxii... Running
> Services not defined
> {code}
> *Workaround*
> In order to get around this issue, edit the file at
> /usr/local/opentaxii/bin/collection-status.py and change it to look like the
> below.
> {code}
>
> from cabby import create_client
> base_uri = 'http://localhost:9000'
> try:
> # create a connection
> client =
> create_client(host='localhost',port=9000,discovery_path='/services/discovery')
> # iterate through each defined collection
> collections =
> client.get_collections(uri='%s/services/collection'%base_uri)
> for collection in collections:
> # how many records in each collection?
> count = client.get_content_count(collection_name=collection.name,
> uri='%s/services/poll'%base_uri)
> print "%-50s %-10d" % (collection.name, count.count)
> except:
> print "Services not defined"
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
