What order did you add the hadoop or yarn classpath? The "shaded" package
stands out to me in this name "org.apache.hadoop.hbase.*shaded*
.org.codehaus.jackson.jaxrs.JacksonJaxbJsonProvider." Maybe try adding
those packages earlier on the classpath.
I think that find command needs a "jar tvf",
To explore the idea of merging metron-api into metron-rest and running pcap
queries inside our REST application, I created a simple test here:
https://github.com/merrimanr/incubator-metron/tree/pcap-rest-test. A
summary of what's included:
- Added pcap as a dependency in the metron-rest
>From my perspective PCAP is primarily used as a follow-on to an alert or
meta-alert - people very rarely use PCAP for initial hunting. I know this
has been brought up by Otto, Mike, and Ryan across the two related threads
and I think it's all spot on. Going from an alert or meta-alert to
I can imagine a new generic service(s) capability whose job ( pun intended
) is to
abstract the submittal, tracking, and storage of results to yarn.
It would be extended with storage providers, queue provider, possibly some
set of policies or rather strategies.
The pcap ‘report’ would be a
RE: Tracking v. users
The submittal and tracking can associate the submitter with the yarn job
and track that,
regardless of the yarn credentials.
IE> if all submittals and monitoring are by the same yarn user ( Metron )
from a single or
co-operative set of services, that service can maintain
That sounds fine - I'd imagine we'd be looking to hit the classpath related
problems asap when merging the modules.
For the module, we just have a pom that supplies external dependencies.
Rather than every metron module depending on Guava or Jackson directly, or
via transitive dependencies, we
Otto, your use case makes sense to me. We'll have to think about how to
manage the user to job relationships. I'm assuming YARN jobs will be
submitted as the metron service user so YARN won't keep track of this for
us. Is that assumption correct? Do you have any ideas for doing that?
Mike, I
