Re: [DISCUSS] Build Log Usability

[David Lyle]

I'd prefer not to see a file output by default, it's non-standard and
people won't know to look there without additional documentation. If we
default to ERROR logging and suppress ERROR output in the case of
"expected" errors, we should be able to keep the test output under 4MB.
Then, if additional logging is desired, one could enable a different log
level and carry on.

For the out of order components, we can capture the output and suppress it.

-D...


On Thu, Oct 27, 2016 at 8:33 PM, Casey Stella  wrote:

> I can get behind logging to a file, but can we see the file as an output of
> Travis? Sometimes the problem is that it is failing in Travis but not
> locally or it is a situation that you want to give guidance on a PR but
> don't want to pull a branch and rerun tests. If we log to a file, I'd like
> to make sure we can get that file as part of the Travis build.
>
> On Fri, Oct 28, 2016 at 00:42 Otto Fowler  wrote:
>
> > I think we should have all the tests log to file verbosely and to console
> > reduced.  We can gather verbose logs on fail and have them part of the
> > clean etc.  Error messages from integration components that do not result
> > in test failures will still be a problem otherwise.  I do not think
> > personally that the Travis visible logs need to show more than what
> failed.
> >
> > On October 27, 2016 at 17:05:28, Ryan Merriman (merrim...@gmail.com)
> > wrote:
> >
> > > I definitely agree with Otto that we need to make our build logs less
> > > verbose. I think the first step is to get control of the log levels
> > across
> > > various components. Are there specific examples of the types of
> messages
> > > we want to suppress? For instance:
> > >
> > > org.apache.metron.profiler.integration.ProfilerIntegrationTest -
> > zookeeper
> > > and curator log levels are set to INFO
> > > org.apache.metron.enrichment.bolt.ThreatIntelJoinBoltTest - expected
> > > exceptions are printed out when they should be suppressed (to Dave's
> > > point)
> > > ...
> > >
> > > Maybe building a list of examples will be easier after we take a pass
> at
> > > setting all the log levels to ERROR. Ideally we only see the test
> results
> > > and an error if something really did fail. Does that sound like a
> > > reasonable approach?
> > >
> > > Ryan
> > >
> > > On Wed, Oct 19, 2016 at 7:44 AM, David Lyle 
> > wrote:
> > >
> > > Hi Otto,
> > >
> > > Good point. One of the practices I'd like to see is to suppress any
> > > exception/error output in tests when it's expected.
> > >
> > > Thanks!
> > >
> > > -D...
> > >
> > >
> > > On Wed, Oct 19, 2016 at 8:18 AM, Otto Fowler 
> > > wrote:
> > >
> > > The Metron build logs contain a great deal of information, including a
> > > large number of warnings / exceptions that do not result in test
> failures
> > > but still fill the log. This makes troubleshooting test/build issues
> > > difficult. For example - the travis ci log for PR #276’s recent failure
> > > doesn’t even have the final errors because the log is too long.
> > >
> > > What are some things that can be done to make the situation better?
> > >
> > >
> > >
> > >
> > >
> >
>

Re: [DISCUSS] Build Log Usability

[Casey Stella]

I can get behind logging to a file, but can we see the file as an output of
Travis? Sometimes the problem is that it is failing in Travis but not
locally or it is a situation that you want to give guidance on a PR but
don't want to pull a branch and rerun tests. If we log to a file, I'd like
to make sure we can get that file as part of the Travis build.

On Fri, Oct 28, 2016 at 00:42 Otto Fowler  wrote:

> I think we should have all the tests log to file verbosely and to console
> reduced.  We can gather verbose logs on fail and have them part of the
> clean etc.  Error messages from integration components that do not result
> in test failures will still be a problem otherwise.  I do not think
> personally that the Travis visible logs need to show more than what failed.
>
> On October 27, 2016 at 17:05:28, Ryan Merriman (merrim...@gmail.com)
> wrote:
>
> > I definitely agree with Otto that we need to make our build logs less
> > verbose. I think the first step is to get control of the log levels
> across
> > various components. Are there specific examples of the types of messages
> > we want to suppress? For instance:
> >
> > org.apache.metron.profiler.integration.ProfilerIntegrationTest -
> zookeeper
> > and curator log levels are set to INFO
> > org.apache.metron.enrichment.bolt.ThreatIntelJoinBoltTest - expected
> > exceptions are printed out when they should be suppressed (to Dave's
> > point)
> > ...
> >
> > Maybe building a list of examples will be easier after we take a pass at
> > setting all the log levels to ERROR. Ideally we only see the test results
> > and an error if something really did fail. Does that sound like a
> > reasonable approach?
> >
> > Ryan
> >
> > On Wed, Oct 19, 2016 at 7:44 AM, David Lyle 
> wrote:
> >
> > Hi Otto,
> >
> > Good point. One of the practices I'd like to see is to suppress any
> > exception/error output in tests when it's expected.
> >
> > Thanks!
> >
> > -D...
> >
> >
> > On Wed, Oct 19, 2016 at 8:18 AM, Otto Fowler 
> > wrote:
> >
> > The Metron build logs contain a great deal of information, including a
> > large number of warnings / exceptions that do not result in test failures
> > but still fill the log. This makes troubleshooting test/build issues
> > difficult. For example - the travis ci log for PR #276’s recent failure
> > doesn’t even have the final errors because the log is too long.
> >
> > What are some things that can be done to make the situation better?
> >
> >
> >
> >
> >
>

Re: [DISCUSS] Build Log Usability

[Ryan Merriman]

I definitely agree with Otto that we need to make our build logs less
verbose.  I think the first step is to get control of the log levels across
various components.  Are there specific examples of the types of messages
we want to suppress?  For instance:

org.apache.metron.profiler.integration.ProfilerIntegrationTest - zookeeper
and curator log levels are set to INFO
org.apache.metron.enrichment.bolt.ThreatIntelJoinBoltTest - expected
exceptions are printed out when they should be suppressed (to Dave's point)
...

Maybe building a list of examples will be easier after we take a pass at
setting all the log levels to ERROR.  Ideally we only see the test results
and an error if something really did fail.  Does that sound like a
reasonable approach?

Ryan

On Wed, Oct 19, 2016 at 7:44 AM, David Lyle  wrote:

> Hi Otto,
>
> Good point. One of the practices I'd like to see is to suppress any
> exception/error output in tests when it's expected.
>
> Thanks!
>
> -D...
>
>
> On Wed, Oct 19, 2016 at 8:18 AM, Otto Fowler 
> wrote:
>
> > The Metron build logs contain a great deal of information, including a
> > large number of warnings / exceptions that do not result in test failures
> > but still fill the log.  This makes troubleshooting test/build issues
> > difficult.  For example - the travis ci log for PR #276’s recent failure
> > doesn’t even have the final errors because the log is too long.
> >
> > What are some things that can be done to make the situation better?
> >
> >
> >
>

Re: [DISCUSS] Time-base flushing of all Writer queues

[Matt Foley]

Hi all,

For those who are interested, answers to the below questions have been posted 
in the jira, METRON-322.

 

Second, I am splitting the work into (1) implementing batchTimeout parameter 
for all queues that have batchSize parameter; (2) modifying each Writer to make 
use of the batchTimeout parameter.  The design for the first step is in 
https://issues.apache.org/jira/browse/METRON-516 .  Please have a look at it 
and comment if you will.

 

Thanks,

--Matt

 

From: Matt Foley 
Date: Thursday, October 20, 2016 at 3:06 PM
To: 
Subject: Re: [DISCUSS] Time-base flushing of all Writer queues

 

Design Questions for METRON-322

 

First, the following questions are open, and if you’re interested please give 
me your input:

You can respond here or in the Jira.

 

1.  Motivation: All bolts that do internal queuing or batching, should be 
able to flush on timeout, because they will all suffer from the problems cited. 
 
Question: Are there any queuing/batching bolts that do NOT use 
BulkMessageWriterBolt class?  
(Reason for asking: I would like to use BulkMessageWriterBolt class as my 
implementation bottleneck.  The code implies that it is used by all the 
Indexing bolts, and the HBase-like Enrichment bolts, but it’s hard to be sure.)
Note this does not refer to Storm’s queuing under-the-hood between bolts, only 
to Metron-specific batching inside of Metron-implemented bolts.

 

2.  Motivation: All topologies that have a positive value of 
“topology.message.timeout.secs” should also have a positive value of 
“topology.tick.tuple.freq.secs” for all queuing/batching bolts, to avoid 
spurious message failures due to timeout.
Question: Does any topology have TWO OR MORE qeueing bolts in series, in a 
single topo?  
(Reason for asking: If not, the default value of 
“topology.tick.tuple.freq.secs” should just be 1/2 of 
“topology.message.timeout.secs”, but if two such queues in series, then the 
default value of “topology.tick.tuple.freq.secs” should be 1/4 of 
“topology.message.timeout.secs”.) 

 

Design Proposal

 

Only Bolt objects can receive Tick Tuples.  They receive them from the 
SYSTEM_COMPONENT source component, via the SYSTEM_TICK_STREAM stream.  They are 
not structured like Metron message tuples.  “topology.tick.tuple.freq.secs” is 
a topology-level configuration parameter, but is specified by each bolt that 
wishes to receive periodic Tick Tuples.  If multiple bolts in the same topology 
specify different intervals, it appears that Storm uses the shortest interval 
for all recipients.

 

The BulkMessageWriterBolt appears to be the root class of all queuing/batching 
Bolts in Metron.  When BulkMessageWriterBolt receives a message tuple, it 
passes (sensorType, tuple, message, BulkMessageWriter, configuration) to the 
write() method of BulkWriterComponent, which owns the queue, and makes the 
decision whether to enqueue the message, or if the queue is full, to flush the 
queue by passing it to the BulkMessageWriter.  BulkWriterComponent actually 
owns a set of named queues, one for each allowed value of sensorType.  The 
underlying BulkMessageWriter may then write the different telemetries 
differently, per queue.

 

I propose to receive the Tick Tuple in BulkMessageWriterBolt, and not pass it 
to BulkWriterComponent::write().  Instead, will add a new method 
BulkWriterComponent::flushOlderThanSec(int interval, BulkMessageWriter, 
configuration).  This method will cause BulkWriterComponent to iterate over the 
set of queues, and flush each if needed.  As recommended here, I will not 
necessarily flush every queue, but only those that have not been flushed (due 
to batchSize or previous timeout-based flush) for at least interval seconds.

 

I also propose to be aware of changes in the configuration of 
“topology.tick.tuple.freq.secs”, and incorporate the new value in the next 
instance of a received Tick Tuple.

 

That’s about it.  The open questions again are:

1.  Are there any queuing/batching bolts that do NOT use 
BulkMessageWriterBolt class?

2.  Does any topology have TWO OR MORE qeueing bolts in series, in a single 
topo?

 

Thanks,

--Matt

 

From: Matt Foley 
Date: Friday, October 14, 2016 at 6:32 PM
To: 
Subject: [DISCUSS] Time-base flushing of all Writer queues

 

Hi all,

 

METRON-227 “Add Time-Based Flushing to Writer Bolt”, and 

METRON-322 “Global Batching & flushing”

have been dormant since July, but contain some very valuable ideas.  The basic 
idea is that Metron’s Writer queues in general will flush on queue size, but 
not on time.  As a result, low-traffic or bursty channels can languish 
unprocessed, and therefore un-ack’ed, which results in Storm automatically 
recycling the messages after a certain timeout (topology.message.timeout.secs), 
or if too many total pending messages accumulate in a topology 
(topology.max.spout.pending).  This 

[GitHub] incubator-metron pull request #324: METRON-515: Stellar IS_EMPTY() function ...

[merrimanr]

GitHub user merrimanr reopened a pull request:

https://github.com/apache/incubator-metron/pull/324

METRON-515: Stellar IS_EMPTY() function does not work as expected

The "IS_EMPTY" Stellar function is always returning true if the field value 
is not a String type.  This PR changes the function to only return true when a 
value is missing, an empty string, or has a null value.  

I also modified the unit test to include a test for an Integer type and 
removed the test that asserts true when the value is a newly created object.  
In my opinion that is not a valid test case.  

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/merrimanr/incubator-metron METRON-515

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/324.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #324


commit 3847ad828c4069cecba70667fd3662f4f5507e89
Author: rmerriman 
Date:   2016-10-25T18:39:59Z

Now handles other types besides String

commit a9f6538f48bf43cd1f725258ae619b056c9ef17c
Author: rmerriman 
Date:   2016-10-26T14:35:42Z

Updated docs to reflect change




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #324: METRON-515: Stellar IS_EMPTY() function ...

[merrimanr]

Github user merrimanr closed the pull request at:

https://github.com/apache/incubator-metron/pull/324


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #324: METRON-515: Stellar IS_EMPTY() function does no...

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/incubator-metron/pull/324
  
+1, pending Travis cooperating.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #315: METRON-464: Force co-location of all Metron com...

[dlyle65535]

Github user dlyle65535 commented on the issue:

https://github.com/apache/incubator-metron/pull/315
  
@justinleet - if you're still good with my latest change, I'll go ahead and 
merge this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #322: METRON-513: Ambari MP Metainfo should no...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/322


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #326: METRON-510: Update the bro_index elasticsearch ...

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/incubator-metron/pull/326
  
Not the best, but from my install node, inside the docker container I used 
to install, I inserted the updated template to my ES cluster via:
`curl --data 
"@/root/incubator-metron/metron-deployment/roles/metron_elasticsearch_templates/files/es_templates/bro_index.template"
 -XPUT server5:9200/_template/bro_index`

Then I put a message on the indexing topic with a response_body_len of 
4261543936 (a value I saw cause this issue in the past):
`echo ${jsonBlob} | 
/usr/hdp/2.4.0.0-169/kafka/bin/kafka-console-producer.sh --topic indexing 
--broker-list server1:6667,server2:6667,server3:6667,server4:6667`

And watched the indexing topic, monitoring for the value in the 
response_body_len field:
`/usr/hdp/2.4.0.0-169/kafka/bin/kafka-console-consumer.sh --zookeeper $zk 
--topic indexing | grep 4261543936`

Finally, I successfully searched for a response_body_len of 4261543936 in 
kibana, as well as a more general response_body_len:>2147483647, which provided 
valid responses.


I also pursued testing this in quick-dev but it knocked over my laptop, so 
I'll be setting up a VM to do that in the future...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #329: METRON-148 compress logs with logrotate

[dlyle65535]

Github user dlyle65535 commented on the issue:

https://github.com/apache/incubator-metron/pull/329
  
Looks good to me, Otto. How did you test it?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #315: METRON-464: Force co-location of all Met...

[dlyle65535]

Github user dlyle65535 closed the pull request at:

https://github.com/apache/incubator-metron/pull/315


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #315: METRON-464: Force co-location of all Met...

[dlyle65535]

GitHub user dlyle65535 reopened a pull request:

https://github.com/apache/incubator-metron/pull/315

METRON-464: Force co-location of all Metron components

Now the Ambari Add Service will show an error in the case that:

a) The Metron Parsers service is not deployed with the Storm Supervisor and 
Kafka Broker.
b) Any of the other Metron services aren't installed to the host containing 
Metron Parsers.

Tested on Docker cluster following procedure here:[ Metron 
MPack](https://www.evernote.com/l/AhLFVR-9CsFIYYnOnF43BlxSsT4F856qwaY)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dlyle65535/incubator-metron METRON-464

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/315.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #315


commit b40f0d649f461a9adb82495e912f7cc0558d8503
Author: David Lyle 
Date:   2016-10-17T23:28:46Z

METRON-464: Force co-location of all Metron components

commit 7fcf88c2335fdc5eaa5e90cf3416d3e216e62593
Author: David Lyle 
Date:   2016-10-26T16:11:50Z

Address PR review comments-
Change colocation for storm and kafka to error
Correct MySQL location error message

commit a290b373cdbef130c96aa47c013a5fb6b99b52d8
Author: David Lyle 
Date:   2016-10-26T17:48:36Z

Merge branch 'master' into METRON-464

commit 813701b62d4537e6b709ab399ae994bb589eb7a3
Author: David Lyle 
Date:   2016-10-26T20:44:49Z

Fix post-merge issues.

commit 001cd76181524088667d9264259a10977183dd62
Author: David Lyle 
Date:   2016-10-27T13:22:46Z

Leave KAFKA_BROKER co-location as ERROR.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #315: METRON-464: Force co-location of all Metron com...

[dlyle65535]

Github user dlyle65535 commented on the issue:

https://github.com/apache/incubator-metron/pull/315
  
reopening for travis error. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #315: METRON-464: Force co-location of all Metron com...

[anandsubbu]

Github user anandsubbu commented on the issue:

https://github.com/apache/incubator-metron/pull/315
  
Thank you @dlyle65535 for a nice fix.

I'm +1 (non-binding)

I did tests with the modified commit and here are my observations:

1) The metron_mpack jar does not get generated during ``mvn clean package`` 
now.

2) Fresh deploy of HDP + Metron at once
- In the beginning the Metron Parser error about Kafka appears, and then 
changes to a warn message about Storm Supervisor. This is working as expected. 
- The co-location error for the MySQL Server, Metron Enrichment and Metron 
Indexing is shown when any of these are not on the same host as Metron Parsers
- In the event that a user _still_ hits **Next** button when any of the 
above Error exists, the Ambari wizard pops up a **Validation Issues** window 
that allows the users to Continue Anyway. This is the way Ambari is designed to 
work.
- However that said, the **Validation Issues** windows does appear even 
when the WARN message about Storm Supervisor is present. One would have to 
"Continue Anyway" to proceed. I'm not sure if there is anything that can be 
done for this scenario.
- Clicked through the wizard and was able to bring up the cluster fine 
without issues.

3) Deploy with a existing HDP cluster - Adding Metron services subsequently
- In the **Assign Masters** page, the ERROR and WARN messages are gone when 
all the colocation criteria are met.
- In my case, I did have an issue where the host where I was installing did 
not have Supervisors running. I had to go back to Ambari and use the 
Host->Actions screen to add a storm supervisor to the node. Likewise, I had to 
also use the Host->MetronHost->Add Clients option to install HBASE client, 
which is required during service startup, since the MetronHost did not have the 
clients installed during the initial HDP deployment.
- Clicked through the wizard and was able to bring up the cluster fine 
without issues


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #329: METRON-148 compress logs with logrotate

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/329
  
kicking travis


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #329: METRON-148 compress logs with logrotate

[ottobackwards]

Github user ottobackwards closed the pull request at:

https://github.com/apache/incubator-metron/pull/329


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #329: METRON-148 compress logs with logrotate

[ottobackwards]

GitHub user ottobackwards reopened a pull request:

https://github.com/apache/incubator-metron/pull/329

METRON-148 compress logs with logrotate

Add compress configuration for logrotate.d/ configurations

Tested using the logrotate -f /etc/logrotate.d/metron-* commands

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ottobackwards/incubator-metron METRON-148

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/329.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #329


commit b44146cfceb542fbeed4e812a8fb9a62fa29bfbe
Author: Otto Fowler 
Date:   2016-10-27T01:50:13Z

METRON-148 compress logs with logrotate




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #329: METRON-148 compress logs with logrotate

[ottobackwards]

GitHub user ottobackwards opened a pull request:

https://github.com/apache/incubator-metron/pull/329

METRON-148 compress logs with logrotate

Add compress configuration for logrotate.d/ configurations

Tested using the logrotate -f /etc/logrotate.d/metron-* commands

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ottobackwards/incubator-metron METRON-148

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/329.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #329


commit b44146cfceb542fbeed4e812a8fb9a62fa29bfbe
Author: Otto Fowler 
Date:   2016-10-27T01:50:13Z

METRON-148 compress logs with logrotate




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Metron REST API Architecture and Design

[zeo...@gmail.com]

Another thought that came to me recently - Has there been any consideration
of using some of the newer standards such as pxGrid
, DXL
, etc. to integrate with other
systems and allow data loading/sharing?  I'm not familiar enough with those
solutions to know if it fits with the current Metron API architecture, but
at least I wanted to put it out there.

Jon

On Mon, Oct 24, 2016 at 11:00 AM larry mccay  wrote:

> I think this is a reasonable direction for Metron.
>
> It probably makes sense to make sure that your services can accept identity
> propagation from Knox so that they can also be proxied along with Hadoop
> APIs.
>
> FWIW - discussing whether a JAXRS programming model is something wanted by
> the community wouldn't be a difficult thing to do.
> It hasn't been discussed to this point which is why it isn't documented -
> this is primarily because there hasn't been any explicit demand for it.
>
>
>
> On Mon, Oct 24, 2016 at 10:51 AM, zeo...@gmail.com 
> wrote:
>
> > Ok, that sounds good to me, I primarily wanted to see whether or not it
> was
> > attempted and if it hit a technical roadblock.  Thanks,
> >
> > Jon
> >
> > On Mon, Oct 24, 2016 at 10:11 AM Ryan Merriman 
> > wrote:
> >
> > > There is also this comment in that Jira:
> > >
> > > "Adding the JAXRS services to knox is really easy but we haven't really
> > > discussed whether it should be a programming model aspect of Knox in
> the
> > > community"
> > >
> > > I think that would need to be worked out before we move services into
> > Knox,
> > > if we decide we should do that.
> > >
> > >
> > >
> > > On Mon, Oct 24, 2016 at 9:06 AM, Ryan Merriman 
> > > wrote:
> > >
> > > > I spent some time researching the Knox documentation and building
> > custom
> > > > services (hosted in Knox) was not well-documented.  Spring is a great
> > > > choice for that and I didn't really get any other feedback on which
> > > > application development framework to use.  So that's what I went
> with.
> > > >
> > > > I think we should plan on adding Knox in front to leverage all the
> nice
> > > > security features and integrations.  That is how most Knox
> integrations
> > > > (HFDS, Storm, etc) are architected.
> > > >
> > > > Ryan
> > > >
> > > > On Mon, Oct 24, 2016 at 8:37 AM, zeo...@gmail.com 
> > > > wrote:
> > > >
> > > >> So it looks like, for now, you are not pursuing Knox (per comments
> in
> > > >> METRON-503 and then PR 316).  Is there a reason for that?
> > > >>
> > > >> Jon
> > > >>
> > > >> On Fri, Oct 14, 2016 at 5:59 PM zeo...@gmail.com 
> > > >> wrote:
> > > >>
> > > >> > Good question :)
> > > >> >
> > > >> > On Fri, Oct 14, 2016, 17:07 Ryan Merriman 
> > > wrote:
> > > >> >
> > > >> > Jon,
> > > >> >
> > > >> > It wasn't intentional, I ran out of time and wanted to get
> something
> > > out
> > > >> > there.  I think it certainly could be open ended though.  Where
> > should
> > > >> the
> > > >> > REST API project be located?
> > > >> >
> > > >> > Ryan
> > > >> >
> > > >> > On Thu, Oct 13, 2016 at 7:32 PM, zeo...@gmail.com <
> zeo...@gmail.com
> > >
> > > >> > wrote:
> > > >> >
> > > >> > > Along the lines of:
> > > >> > > • Must be deployed to a machine with adequate resources so that
> > > >> resource
> > > >> > > contention is avoided.
> > > >> > > • Will need network access to all other services within Metron
> > > >> > >
> > > >> > > Has there been any consideration of a "Metron Manager" node?  In
> > the
> > > >> old
> > > >> > > TP2
> > > >> > > bare metal install guide
> > > >> > >  > > >> > > Metron+Installation+on+an+Ambari-Managed+Cluster>
> > > >> > > it mentions a "Metron Installer," but I could see the needs for
> > that
> > > >> sort
> > > >> > > of a system expanding to have the following roles:
> > > >> > > - API
> > > >> > > - Metron UI
> > > >> > > - Metron Installer/upgrades
> > > >> > > - Edge/Gateway Node for data loading
> > > >> > > - Clients
> > > >> > >
> > > >> > > Also, at the end it ends mid-sentence under "Organization within
> > > >> Metron,"
> > > >> > > was that intended to be open ended?
> > > >> > >
> > > >> > > Jon
> > > >> > >
> > > >> > > On Thu, Oct 13, 2016 at 6:10 PM Ryan Merriman <
> > merrim...@gmail.com>
> > > >> > wrote:
> > > >> > >
> > > >> > > > I created a Jira to track this new feature at
> > > >> > > > https://issues.apache.org/jira/browse/METRON-503.  I also
> > started
> > > >> and
> > > >> > > > attached an architecture doc to that Jira with some of my
> ideas
> > > >> about
> > > >> > how
> > > >> > > > we should implement it.  Please feel free to review and
> comment
> > or
> > > >> add
> > > >> > to
> > > >> > > > it.  Looking forward to everyone's ideas and feedback.
> > > >> > > >
> > > >> > > 

[GitHub] incubator-metron pull request #315: METRON-464: Force co-location of all Met...

[anandsubbu]

Github user anandsubbu commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/315#discussion_r85333482
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/0.2.1BETA/service_advisor.py
 ---
@@ -54,11 +54,11 @@ def getServiceComponentLayoutValidations(self, 
services, hosts):
 #Metron Must Co-locate with KAFKA_BROKER and STORM_SUPERVISOR
 if metronParsersHost not in kafkaBrokers:
 message = "Metron must be colocated with an instance of KAFKA 
BROKER"
-items.append({ "type": 'host-component', "level": 'ERROR', 
"message": message, "component-name": 'METRON_PARSERS', "host": 
metronParsersHost })
+items.append({ "type": 'host-component', "level": 'WARN', 
"message": message, "component-name": 'METRON_PARSERS', "host": 
metronParsersHost })
--- End diff --

Hi @dlyle65535, can we leave this as an ERROR, instead of changing to WARN, 
since the KAFKA_BROKER selection happens on this screen.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #315: METRON-464: Force co-location of all Metron com...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/incubator-metron/pull/315
  
I'm +1 (by inspection in conjunction with Anand's testing and the most 
recent fixes).  Great work figuring this out.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---