[GitHub] incubator-metron issue #490: DO NOT MERGE METRON-797: Pass security.protocol...

[james-sirota]

Github user james-sirota commented on the issue:

https://github.com/apache/incubator-metron/pull/490
  
I reviewed this with 793.  I was able to validate this on a single-node 
Suse-based Hadoop cluster once with Kerberos disabled and second time with 
Kerberos enabled.  That's a good first step.  I will try to run this up in AWS 
tomorrow on a larger cluster to see if it still works.  But so far so good. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #486: METRON-793: Migrate to storm-kafka-client kafka...

[james-sirota]

Github user james-sirota commented on the issue:

https://github.com/apache/incubator-metron/pull/486
  
I was able to validate this on a single-node Suse-based Hadoop cluster once 
with Kerberos disabled and second time with Kerberos enabled.  That's a good 
first step.  I will try to run this up in AWS tomorrow on a larger cluster to 
see if it still works.  But so far so good. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Kerberos Support

[Casey Stella]

It occurs to me that I did not cite the work that inspired this
appropriately and I should've.  I do apologize for that; it was a
significant enough departure from the original investigatory branch that it
didn't occur to me.  I've adjusted the JIRA and the PR description as
follows to ensure proper credit for the inspiration:


This work was inspired by a portion of the investigatory work done at
https://github.com/dlyle65535/incubator-metron/tree/kerb-testing?files=1 by:
* @dlyle65535
* @merrimanr
* @mmiklavc

This carves out a specific piece of that functionality with the following
differences:
* The mpack work is not included, but the properties are set up to enable
it as a follow-on
* It presumes METRON-793, so it uses storm-kafka-client rather than
storm-kafka
* It adds a flag when starting the parsers to pass the security protocol
and sets up the writers and the spout automatically rather than relying on
the set of extra kafka configs (though both approaches would work here).


Sorry about that!

Casey


On Sat, Mar 25, 2017 at 12:14 PM, Casey Stella  wrote:

> Actually, METRON-797 (https://github.com/apache/incubator-metron/pull/490)
> is inspired by that work, Dave.
> Specifically here are the differences:
>
>- The mpack work is not included
>- It's based on 793, so it uses storm-kafka-client rather than
>storm-kafka
>- It adds a flag when starting the parsers to pass the security
>protocol and sets up the writers and the spout automatically
>- The core extension points (flux properties) are set up like in that
>PR, to make the follow-on work easier
>
>
> What's left is to pull the great work on the mpack out of there and get it
> working as a follow-on PR, which I created METRON-799 (
> https://issues.apache.org/jira/browse/METRON-799) to accomplish.
>
> Casey
>
>
> On Sat, Mar 25, 2017 at 10:11 AM, David Lyle  wrote:
>
>> Sounds good to me. A few of us did some initial exploration on that topic
>> a
>> week or so back. The branch is here:
>> https://github.com/dlyle65535/incubator-metron/tree/kerb-testing?files=1
>>
>> It contains a prototype quality version of everything you've identified
>> except METRON-793 and the probes.
>>
>> If it's a good starting place, it'd probably be short work to get it to a
>> PR-able form.
>>
>> Thoughts?
>>
>> -D...
>>
>> On Fri, Mar 24, 2017 at 23:03 Casey Stella  wrote:
>>
>> > Hi All,
>> >
>> > I'd like to talk and start to formulate a plan around supporting running
>> > Metron on a kerberized cluster.  This is a big bundle of work and seems
>> > dauntingly nebulous, so I wanted to have a chat and get a firm
>> direction.
>> > When initially contemplating the issue, it's apparent that there are a
>> few
>> > snags that need to be thought through:
>> >
>> >- The kafka spout (storm-kafka) from apache does not support
>> interacting
>> >with kerberized kafka
>> >- We need to ensure the kafka writers are passing along the security
>> >protocol
>> >- We need to ensure that the credentials are auto-renewed
>> >
>> > Since I knew this was necessary, I wanted to get over this initial
>> barrier
>> > and submitted a couple of PRs associated with the following JIRAs that
>> are
>> > in review now:
>> >
>> >- METRON-793: Migrate to storm-kafka-client, a kafka spout which
>> >supports kerberized kafka
>> >- METRON-797: Pass along security protocols and set up autorenewal
>> >
>> > Between these, we get what I believe is the enrichment, profiler and
>> parser
>> > topologies and interactions with hbase and hdfs from them working along
>> > with the MR jobs.  We still lack a few things:
>> >
>> >- A document describing the process of kerberizing a vagrant
>> environment
>> >to enable testing
>> >- Kerberos support for the librdkafka-based sensors (bro and
>> fastcapa)
>> >- Kerberos support for the sensors that use the console-producer
>> (snort
>> >and yaf)
>> >   - This should be as easy as ensuring to pass the right params to
>> the
>> >   console producer, but still, we need to adjust the sensor stubs
>> > to make the
>> >   right call.
>> >- Kerberos support for the mpack
>> >- Kerberos support for the REST API
>> >- Kerberos support for the pycapa
>> >
>> > I'm tracking these as JIRAs with the label of "kerberos" (see
>> >
>> > https://issues.apache.org/jira/browse/METRON-802?jql=labels%
>> 20%3D%20kerberos%20AND%20project%20%3D%20Metron
>> > )
>> >
>> > Please chime in if I missed something; I'll add it to the list.
>> >
>> > Best,
>> >
>> > Casey
>> >
>>
>
>

[GitHub] incubator-metron issue #488: METRON-796: Mpack uses wrong group for owning H...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/incubator-metron/pull/488
  
https://issues.apache.org/jira/browse/METRON-349 is updated to be more 
complete and reflect the current state of things.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #488: METRON-796: Mpack uses wrong group for owning H...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/incubator-metron/pull/488
  
@dlyle65535 Failure mode is that HDFS writes from Storm fail.  The 
directories are owned by metron:metron with 775.  Storm isn't in the metron 
group, so it fails to write.  The perms exception is thrown in the bolt and no 
output file is created.  Writes to ES work as expected.  Validating this is 
pretty easy, just run up full dev and see if the perms error shows up and if 
HDFS gets any files.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---