[
https://issues.apache.org/jira/browse/METRON-958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16018076#comment-16018076
]
ASF GitHub Bot commented on METRON-958:
---------------------------------------
GitHub user mmiklavc reopened a pull request:
https://github.com/apache/metron/pull/593
METRON-958: PCAP Query job throws exception when no files returned by time
range query
https://issues.apache.org/jira/browse/METRON-958
When the PCAP topology is run against a Kafka topic with multiple
partitions, there will be at least one file per partition. The current MR job
only expects adding a left trailing file for a single partition, so we miss
files when performing the filter by timestamp on the filenames and it ends up
throwing an exception.
## Testing
### Part 1
To test this, run a pcap topology against a Kafka topic with multiple
partitions, preferably 3 or more. You'll also want to set kafka.pcap.maxTimeMS
to something like 5 minutes (300000). You'll need to install pycapa and run it
to create some data -
https://github.com/apache/metron/tree/master/metron-sensors/pycapa. Let it run
for 4-5 minutes. To keep it simple, stop it shy of 5 minutes so there's only 1
large-ish file per partition.
Run pcap inspector on one of the files and make note of the starting and
ending time. Pick a time in the middle and run a pcap query using that time,
down to the minute, as a filter. e.g.
`${METRON_HOME}/bin/pcap_query.sh query -df "yyyyMMddHHmm" -st 201705162000
--query ""`
### Part 2
Run a pcap query with a time outside of the range of times included in the
pcap data and verify an exception is not thrown.
https://cwiki.apache.org/confluence/display/METRON/PCAP+CLI+Tool
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.
Please refer also to our [Build Verification
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been
executed in the root incubating-metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to
verify your changes?
- [ ] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered by building and verifying the site-book? If not then run
the following commands and the verify changes via
`site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up
for your personal repository such that your branches are built there before
submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mmiklavc/metron METRON-958
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/593.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #593
----
commit 30ae170833a3f400659a914ebc260f243f108125
Author: Michael Miklavcic <[email protected]>
Date: 2017-05-17T22:20:09Z
METRON-958: PCAP Query job throws exception when no files returned by time
range query
commit 7ac3c250bfdd0da811b3b8b5fe682373ba791811
Author: Michael Miklavcic <[email protected]>
Date: 2017-05-17T23:09:57Z
rat
commit 06ab1367b97c61d1be3c8acaa36a2b299d427147
Author: Michael Miklavcic <[email protected]>
Date: 2017-05-18T19:09:29Z
Fix exception on no results returned
commit cde214ca6196eeb9326aa65f7ab2695c0b53c0ba
Author: Michael Miklavcic <[email protected]>
Date: 2017-05-19T01:17:14Z
Still an outstanding bug with grabbing left trailing edge
commit 3f4ad2ebc703c26f899087f3cd0f06bd89a0edc5
Author: Michael Miklavcic <[email protected]>
Date: 2017-05-19T21:10:19Z
Fix handling special chars in the Kafka partition
----
> PCAP Query job throws exception when no files returned by time range query
> --------------------------------------------------------------------------
>
> Key: METRON-958
> URL: https://issues.apache.org/jira/browse/METRON-958
> Project: Metron
> Issue Type: Bug
> Reporter: Michael Miklavcic
> Assignee: Michael Miklavcic
>
> {code}
> bin/pcap_query.sh query -df "yyyyMMddHHmm" -st 201705162000 --query ""
> Exception in thread "main" java.lang.IllegalArgumentException: Can not create
> a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:126)
> at org.apache.hadoop.fs.Path.<init>(Path.java:134)
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPaths(FileInputFormat.java:485)
> at org.apache.metron.pcap.mr.PcapJob.createJob(PcapJob.java:285)
> at org.apache.metron.pcap.mr.PcapJob.query(PcapJob.java:237)
> at org.apache.metron.pcap.query.PcapCli.run(PcapCli.java:138)
> at org.apache.metron.pcap.query.PcapCli.main(PcapCli.java:53)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
> {code}
> For this particular data set, 201705161900 worked fine.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
