Re: Issues with Quick Dev Installation

2016-10-11 Thread Nick Allen
Try with "mysql" instead of "mysqld". I am probably remembering the service name incorrectly. On Oct 11, 2016 9:59 PM, "Rita McKissick" wrote: > Hi Nick, > > Aah, I had forgotten about this. Can you tell where I run "service mysqld > status" and "service mysqld > start” from. I keep getting mysq

Re: Issues with Quick Dev Installation

2016-10-11 Thread Rita McKissick
Hi Nick, Aah, I had forgotten about this. Can you tell where I run "service mysqld status" and "service mysqld start” from. I keep getting mysqld: unrecognized service and I’m pretty sure I’m running the command from the wrong location. Thanks, Rita Rita McKissick ! Sr. Technical Writer

Re: Issues with Quick Dev Installation

2016-10-11 Thread Nick Allen
Does it really need to sudo? Can we do something like "become: false" so it doesn't try to sudo? On Oct 11, 2016 9:33 PM, "Kyle Richardson" wrote: > Ok, I think I have the fix for this. With the new local_action logic, > ansible is checking the jar path on the host with sudo but doesn't have the

[GitHub] incubator-metron issue #188: METRON-227 (Time based flushing)

2016-10-11 Thread mattf-horton
Github user mattf-horton commented on the issue: https://github.com/apache/incubator-metron/pull/188 Michael, I'm in the process of reviewing this, and also pinging Ajay Yadav, in the context of METRON-322, to confirm he is not still active on this issue. Assuming he isn't, I will ch

Re: Issues with Quick Dev Installation

2016-10-11 Thread Kyle Richardson
Ok, I think I have the fix for this. With the new local_action logic, ansible is checking the jar path on the host with sudo but doesn't have the password. Just need to add the line below to the Vagrantfile. ansible.ask_sudo_pass = true I will test tomorrow and, if successful, open a new PR for t

Re: Issues with Quick Dev Installation

2016-10-11 Thread Kyle Richardson
This error could be related to my PR that was merged today (METRON-492 ). I tested this successfully in a single node vm deployment, but not with vagrant. Perhaps there is something about vagrant that doesn't like the ansible local_action logic?

[GitHub] incubator-metron pull request #276: METRON-363 Fix Cisco ASA Parser

2016-10-11 Thread kylerichardson
GitHub user kylerichardson reopened a pull request: https://github.com/apache/incubator-metron/pull/276 METRON-363 Fix Cisco ASA Parser I've rewritten the ASA parser which can be extended, as needed, to new ASA message types by editing the bundled asa patterns file and the static ma

[GitHub] incubator-metron pull request #276: METRON-363 Fix Cisco ASA Parser

2016-10-11 Thread kylerichardson
Github user kylerichardson closed the pull request at: https://github.com/apache/incubator-metron/pull/276 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if t

[GitHub] incubator-metron pull request #303: METRON-424 ability to validate ip addres...

2016-10-11 Thread ottobackwards
GitHub user ottobackwards reopened a pull request: https://github.com/apache/incubator-metron/pull/303 METRON-424 ability to validate ip addresses against both IPV4 and IPV6 Allow the specification of multiple validation types for IP ` { "fieldValidations" : [

[GitHub] incubator-metron pull request #303: METRON-424 ability to validate ip addres...

2016-10-11 Thread ottobackwards
Github user ottobackwards closed the pull request at: https://github.com/apache/incubator-metron/pull/303 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if th

Re: Issues with Quick Dev Installation

2016-10-11 Thread Nick Allen
To your second point about no data showing up, I have noticed that MySQL is sometimes not started which prevents the enrichment topic from moving data. I have not investigated but it might be due to memory constraints on the virtual node. Check if MySQL is running with "service mysqld status" and

Re: Issues with Quick Dev Installation

2016-10-11 Thread Nick Allen
Hi Rita - Yes, I was seeing this same issue today. I haven't looked into the cause yet, but you can comment out that check and proceed with the install. Doing this from memory, but I believe it is in metron-deployment/roles/metron_common/tasks/main.yml. Comment out or delete any task that refer

Issues with Quick Dev Installation

2016-10-11 Thread Rita McKissick
*** Resending because this message seemed to disappear into the ether and I didn’t receive a copy of it. I’m having difficulties with the latest Quick Development Platform installation. During deployment I received the following error message: -- TASK [metron_common

Issues with Quick Dev installation

2016-10-11 Thread Rita McKissick
I’m having difficulties with the latest Quick-Dev installation. During deployment I received the following error message: -- TASK [metron_common : Check for Metron jar path] *** fatal: [node1 -> localhost]: FAILED! => {"changed": false, "fa

[GitHub] incubator-metron issue #276: METRON-363 Fix Cisco ASA Parser

2016-10-11 Thread nickwallen
Github user nickwallen commented on the issue: https://github.com/apache/incubator-metron/pull/276 I would close and re-open. As our test suite has expanded and is more demanding, at certain times Travis will fail the build when there is not really a problem. We need to figure out ho

[GitHub] incubator-metron pull request #305: METRON-403 Bro Elasticsearch index item ...

2016-10-11 Thread nickwallen
GitHub user nickwallen opened a pull request: https://github.com/apache/incubator-metron/pull/305 METRON-403 Bro Elasticsearch index item fails when DNS response includes CNAME ### Problem When the Bro DNS logs contain FQDNs in the 'answers' field, writing to the Elasticse

Re: HDFS Compression

2016-10-11 Thread Matt Foley
Some of the things that are desirable to do with stored data (including those mentioned by others below): - Use it to train ML models o This implies that the format of records stored in HDFS and the format of records streamed to a “Threat Intel” topology should be readily transformable into each

[GitHub] incubator-metron issue #276: METRON-363 Fix Cisco ASA Parser

2016-10-11 Thread kylerichardson
Github user kylerichardson commented on the issue: https://github.com/apache/incubator-metron/pull/276 Not entirely sure why the CI build failed. The error was: ``` testExample1(org.apache.metron.profiler.integration.ProfilerIntegrationTest) Time elapsed: 35.546 sec

[GitHub] incubator-metron issue #304: METRON-496: Field transformations are applied a...

2016-10-11 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/304 I'd prefer to keep the functionality for the CSVParser the way it is in this JIRA. I just wanted to make it so that the CSVParser's `parse` method wouldn't actively stomp on the data pass

[GitHub] incubator-metron issue #304: METRON-496: Field transformations are applied a...

2016-10-11 Thread ottobackwards
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/304 So, for the JSONMapParser - we could remove it though? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

Re: [DISCUSS] Upgrading to Storm 1.0.x

2016-10-11 Thread Casey Stella
I think this would be good (+1 for me), but we should be quite careful that the testing for this PR is...exhaustive as it'll touch so much. At the very least, we'll need to ensure that: - Ansible full-dev and quick-dev continue to work - Ambari install continues to function - Data flows t

[GitHub] incubator-metron issue #304: METRON-496: Field transformations are applied a...

2016-10-11 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/304 Haha, definitely @ottobackwards comment! Thanks for the feedback. :) * I found a potential situation with the CSVParser wrt timestamps that would prevent users from retaining the

Re: [GitHub] incubator-metron pull request #304: METRON-496: Field transformations are ap...

2016-10-11 Thread Carolyn Duby
Have we considered having precondition validation and postcondition validation. Preconditions could check that the fields are in good condition to be transformed. Postconditions could find errors in events after field transformations. Thanks Carolyn On 10/11/16, 1:52 PM, "cestella" wrot

[GitHub] incubator-metron issue #304: METRON-496: Field transformations are applied a...

2016-10-11 Thread ottobackwards
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/304 two things, if I may comment * did you mean to bring in the CSV parser change? * isn't the benefit of this that you can _remove_ the timestamp injection from the generic parser

[GitHub] incubator-metron pull request #303: METRON-424 ability to validate ip addres...

2016-10-11 Thread ottobackwards
GitHub user ottobackwards reopened a pull request: https://github.com/apache/incubator-metron/pull/303 METRON-424 ability to validate ip addresses against both IPV4 and IPV6 Allow the specification of multiple validation types for IP ` { "fieldValidations" : [

[GitHub] incubator-metron pull request #303: METRON-424 ability to validate ip addres...

2016-10-11 Thread ottobackwards
Github user ottobackwards closed the pull request at: https://github.com/apache/incubator-metron/pull/303 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if th

[ANNOUNCE] Metron community demo (Oct.14, 2016, 11AM PST)

2016-10-11 Thread James Sirota
Synopsis of known demos: - Tutorial on extending Stellar functions (helpful for new and aspiring committers) - Using the profiler in enrichment topology - Various bug fixes - Address the incubator build comments Please respond to this email if you have something to show and want to get in

[GitHub] incubator-metron pull request #297: METRON-488: Snort should use a proper CS...

2016-10-11 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/incubator-metron/pull/297 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the featu

[GitHub] incubator-metron issue #303: METRON-424 ability to validate ip addresses aga...

2016-10-11 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/303 So, the problem with the tests is this: ``` test(org.apache.metron.parsers.integration.YafIntegrationTest) Time elapsed: 58.442 sec <<< ERROR! java.lang.RuntimeException: Too

[GitHub] incubator-metron pull request #291: METRON-482 Add logging to GrokParser to ...

2016-10-11 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/incubator-metron/pull/291 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the featu

[GitHub] incubator-metron pull request #304: METRON-496: Field transformations are ap...

2016-10-11 Thread cestella
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/304 METRON-496: Field transformations are applied after parser validation Field transformations are applied after validation, which means that the validation cannot be affected by the transfo

Re: [DISCUSS] Opinionated Data Flows

2016-10-11 Thread Nick Allen
oops, typo: A writer should not o̶c̶c̶u̶r̶ *care* in which topology or for what sensor type it is being used. On Tue, Oct 11, 2016 at 1:46 PM, Nick Allen wrote: > >> I disagree with the idea that Metron should not be responsible for >> defining >> data flows and I think that conflicts with

Re: [DISCUSS] Opinionated Data Flows

2016-10-11 Thread Nick Allen
> > > I disagree with the idea that Metron should not be responsible for defining > data flows and I think that conflicts with the idea of abstracting out the > CEP component (Storm, Flink, etc). When I say that a user should be able to define the data flow, I don't mean that in terms of the unde

Re: HDFS Compression

2016-10-11 Thread Otto Fowler
And also support the extensibility offered by STELLAR and enrichments, such that adding new fields using either will not mean having to write supporting java code etc. Or from a higher level : The flexibility for configuration based enrichment and modification of the data through ingest should not

Re: HDFS Compression

2016-10-11 Thread Carolyn Duby
The format should be compatible/optimal with spark and Zeppelin. Perhaps other interactive BI tools like Tableau. Thanks Carolyn On 10/11/16, 1:06 PM, "Nick Allen" wrote: >Right. The original idea is to do batch analytics. Kind of difficult to >work with data sitting in an ES index. But

[GitHub] incubator-metron issue #303: METRON-424 ability to validate ip addresses aga...

2016-10-11 Thread ottobackwards
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/303 OK, good then. I don't have to change anything. I have no idea why the travis build is failing - it doesn't look todo with my changes ( btw there are always so many errors it is h

Re: HDFS Compression

2016-10-11 Thread Nick Allen
Right. The original idea is to do batch analytics. Kind of difficult to work with data sitting in an ES index. But if we get a better understanding of the type of batch analytics, it might get us closer to the target. On Tue, Oct 11, 2016 at 1:03 PM, zeo...@gmail.com wrote: > I'm somewhat igno

[GitHub] incubator-metron issue #303: METRON-424 ability to validate ip addresses aga...

2016-10-11 Thread nickwallen
Github user nickwallen commented on the issue: https://github.com/apache/incubator-metron/pull/303 Jon described it well. I'd prefer "explicit and obvious." --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

Re: HDFS Compression

2016-10-11 Thread zeo...@gmail.com
I'm somewhat ignorant here, never having used the MaaS stuff yet, but isn't that the dataset that the models would run against? I understand there could be additional use cases, I just wanted to be clear. Jon On Tue, Oct 11, 2016 at 1:01 PM Nick Allen wrote: > I don't think we put much thought

[GitHub] incubator-metron issue #303: METRON-424 ability to validate ip addresses aga...

2016-10-11 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/303 I tend to agree, I prefer `["IPV4", "IPV6"] where we just do an `or` between the two. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHu

Re: HDFS Compression

2016-10-11 Thread Nick Allen
I don't think we put much thought into how exactly the data should be landed in HDFS and for what use cases. It just has not been a priority. That being said, this might be a good time to gather everyone's thoughts on how they would use that kind of data and for what purposes. On Tue, Oct 11,

[GitHub] incubator-metron issue #303: METRON-424 ability to validate ip addresses aga...

2016-10-11 Thread JonZeolla
Github user JonZeolla commented on the issue: https://github.com/apache/incubator-metron/pull/303 I would personally vote for `"type" : ["IPV4","IPV6"]`. I usually push for explicit and obvious so something like `retrieve_json.py | grep IPV4` would work. --- If your project is set

Re: HDFS Compression

2016-10-11 Thread Owen O'Malley
Be careful of using compressed JSON, since it isn't splittable. JSON is also very slow for reading. .. Owen On Tue, Oct 11, 2016 at 4:31 AM, Casey Stella wrote: > I'd also tack on to this that the configuration for the hdfs writer should > be moved to zookeeper rather than done in flux, IMO > O

[GitHub] incubator-metron issue #220: Metron-342 Fix permissions of {{ metron_directo...

2016-10-11 Thread 2xyo
Github user 2xyo commented on the issue: https://github.com/apache/incubator-metron/pull/220 @mmiklavc ok, let me check that this week. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this fe

[GitHub] incubator-metron issue #291: METRON-482 Add logging to GrokParser to indicat...

2016-10-11 Thread dlyle65535
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/291 Thanks! +1 as well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] incubator-metron issue #291: METRON-482 Add logging to GrokParser to indicat...

2016-10-11 Thread mmiklavc
Github user mmiklavc commented on the issue: https://github.com/apache/incubator-metron/pull/291 +1 by inspection with @justinleet's confirmation of quick dev testing --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If yo

[GitHub] incubator-metron issue #291: METRON-482 Add logging to GrokParser to indicat...

2016-10-11 Thread justinleet
Github user justinleet commented on the issue: https://github.com/apache/incubator-metron/pull/291 Yeah, ran it up in quick dev and it kicks out exactly what you'd expect when DEBUG is on. --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] incubator-metron issue #297: METRON-488: Snort should use a proper CSV imple...

2016-10-11 Thread mmiklavc
Github user mmiklavc commented on the issue: https://github.com/apache/incubator-metron/pull/297 +1 by inspection --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wish

[GitHub] incubator-metron pull request #286: METRON-326 Error Handling in Elasticsear...

2016-10-11 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/incubator-metron/pull/286 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the featu

[GitHub] incubator-metron issue #188: METRON-227 (Time based flushing)

2016-10-11 Thread mmiklavc
Github user mmiklavc commented on the issue: https://github.com/apache/incubator-metron/pull/188 Any new comments or actions for this PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] incubator-metron pull request #303: METRON-424 ability to validate ip addres...

2016-10-11 Thread ottobackwards
GitHub user ottobackwards opened a pull request: https://github.com/apache/incubator-metron/pull/303 METRON-424 ability to validate ip addresses against both IPV4 and IPV6 Allow the specification of multiple validation types for IP ` { "fieldValidations" : [

[GitHub] incubator-metron issue #220: Metron-342 Fix permissions of {{ metron_directo...

2016-10-11 Thread mmiklavc
Github user mmiklavc commented on the issue: https://github.com/apache/incubator-metron/pull/220 @2xyo This PR has been here a while - are you still seeing this on current master? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub a

[GitHub] incubator-metron pull request #296: METRON-439: Stellar : IS_EMPTY(host) thr...

2016-10-11 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/incubator-metron/pull/296 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the featu

[GitHub] incubator-metron pull request #293: METRON-473 Add LENGTH() To Stellar

2016-10-11 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/incubator-metron/pull/293 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the featu

[GitHub] incubator-metron pull request #295: METRON-371: Changing logging level to IN...

2016-10-11 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/incubator-metron/pull/295 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the featu

[GitHub] incubator-metron pull request #290: METRON-421 Make Stellar Profiler Client ...

2016-10-11 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/incubator-metron/pull/290 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the featu

[GitHub] incubator-metron pull request #298: METRON-432: Fix pcap field resolver to r...

2016-10-11 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/incubator-metron/pull/298 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the featu

[GitHub] incubator-metron issue #298: METRON-432: Fix pcap field resolver to return o...

2016-10-11 Thread mmiklavc
Github user mmiklavc commented on the issue: https://github.com/apache/incubator-metron/pull/298 FYI, verified this properly queries and filters based on numeric types in quick-dev --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

Re: [DISCUSS] Upgrading to Storm 1.0.x

2016-10-11 Thread Carolyn Duby
Storm 1.0 will allow config of kafka with SSL. Should I comment on METRON-495 or maybe it is better as a separate lira? Thanks Carolyn On 10/11/16, 10:07 AM, "David Lyle" wrote: >I'm +1 on this. > >On Tue, Oct 11, 2016 at 9:54 AM, Justin Leet wrote: > >> Hi all, >> >> I wanted to start a t

[GitHub] incubator-metron pull request #302: METRON-492 Run metron_common build check...

2016-10-11 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/incubator-metron/pull/302 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the featu

Re: [DISCUSS] Upgrading to Storm 1.0.x

2016-10-11 Thread David Lyle
I'm +1 on this. On Tue, Oct 11, 2016 at 9:54 AM, Justin Leet wrote: > Hi all, > > I wanted to start a thread around upgrading our Storm version from 0.10.x > to 1.0.x. I created a Jira at > https://issues.apache.org/jira/browse/METRON-495 to mirror this discussion > and the results and opinions

[GitHub] incubator-metron issue #296: METRON-439: Stellar : IS_EMPTY(host) throws exc...

2016-10-11 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/296 +1 by inspection --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wish

[GitHub] incubator-metron issue #293: METRON-473 Add LENGTH() To Stellar

2016-10-11 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/293 +1 by inspection. Great work, otto. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feat

[GitHub] incubator-metron issue #291: METRON-482 Add logging to GrokParser to indicat...

2016-10-11 Thread dlyle65535
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/291 @justinleet - did you run this up in Vagrant or elsewhere? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] incubator-metron issue #298: METRON-432: Fix pcap field resolver to return o...

2016-10-11 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/298 +1 by inspection --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wish

[DISCUSS] Upgrading to Storm 1.0.x

2016-10-11 Thread Justin Leet
Hi all, I wanted to start a thread around upgrading our Storm version from 0.10.x to 1.0.x. I created a Jira at https://issues.apache.org/jira/browse/METRON-495 to mirror this discussion and the results and opinions. As listed at https://storm.apache.org/2016/04/12/storm100-released.html, there'

[GitHub] incubator-metron issue #290: METRON-421 Make Stellar Profiler Client API Acc...

2016-10-11 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/290 +1 on inspection --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wish

[GitHub] incubator-metron issue #286: METRON-326 Error Handling in ElasticsearchWrite...

2016-10-11 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/286 +1 by inspection --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wish

[GitHub] incubator-metron issue #297: METRON-488: Snort should use a proper CSV imple...

2016-10-11 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/297 @james-sirota it depends on the parser. It applies to the CSVParser, for sure, but this isn't a generic fix (not sure how you'd go about making a generic fix here, in fact). --- If your

[GitHub] incubator-metron issue #302: METRON-492 Run metron_common build check as loc...

2016-10-11 Thread dlyle65535
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/302 +1, this is great. It'll be good to have this working. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project doe

Re: HDFS Compression

2016-10-11 Thread Casey Stella
I'd also tack on to this that the configuration for the hdfs writer should be moved to zookeeper rather than done in flux, IMO On Tue, Oct 11, 2016 at 07:20 Otto Fowler wrote: > The storage format and retrieval from that format should be configurable, > that is a ‘boundary’ for Metron so to speak

Re: HDFS Compression

2016-10-11 Thread Otto Fowler
The storage format and retrieval from that format should be configurable, that is a ‘boundary’ for Metron so to speak. On October 10, 2016 at 16:15:12, zeo...@gmail.com (zeo...@gmail.com) wrote: Is there a specific reason why the JSON files stored in HDFS are not compressed? I looked for some rel