GitHub user cestella reopened a pull request:
https://github.com/apache/metron/pull/621
METRON-1001: Allow metron to ingest parser metadata along with data
## Contributor Comments
Currently, we only ingest data in Metron. Often, there is valuable metadata
constructed up-stream
Github user cestella closed the pull request at:
https://github.com/apache/metron/pull/621
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user cestella closed the pull request at:
https://github.com/apache/metron/pull/621
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
GitHub user cestella reopened a pull request:
https://github.com/apache/metron/pull/621
METRON-1001: Allow metron to ingest parser metadata along with data
## Contributor Comments
Currently, we only ingest data in Metron. Often, there is valuable metadata
constructed up-stream
GitHub user cestella reopened a pull request:
https://github.com/apache/metron/pull/621
METRON-1001: Allow metron to ingest parser metadata along with data
## Contributor Comments
Currently, we only ingest data in Metron. Often, there is valuable metadata
constructed up-stream
Github user cestella closed the pull request at:
https://github.com/apache/metron/pull/621
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
GitHub user cestella reopened a pull request:
https://github.com/apache/metron/pull/621
METRON-1001: Allow metron to ingest parser metadata along with data
## Contributor Comments
Currently, we only ingest data in Metron. Often, there is valuable metadata
constructed up-stream
GitHub user cestella reopened a pull request:
https://github.com/apache/metron/pull/621
METRON-1001: Allow metron to ingest parser metadata along with data
## Contributor Comments
Currently, we only ingest data in Metron. Often, there is valuable metadata
constructed up-stream
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/621
Hopefully the addition of documentation and a comprehensive test plan will
help soothe the rest of the confusion here.
---
If your project is set up for it, you can reply to this email and have
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/621
# TESTING PLAN
Testing Instructions beyond the normal smoke test (i.e. letting data
flow through to the indices and checking them).
# Preliminaries
Since I will use
Github user cestella closed the pull request at:
https://github.com/apache/metron/pull/621
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
GitHub user cestella reopened a pull request:
https://github.com/apache/metron/pull/621
METRON-1001: Allow metron to ingest parser metadata along with data
## Contributor Comments
Currently, we only ingest data in Metron. Often, there is valuable metadata
constructed up-stream
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/621#discussion_r123594763
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
---
@@ -52,6 +50,8 @@
public class
GitHub user cestella reopened a pull request:
https://github.com/apache/metron/pull/621
METRON-1001: Allow metron to ingest parser metadata along with data
## Contributor Comments
Currently, we only ingest data in Metron. Often, there is valuable metadata
constructed up-stream
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/550
sorry, the status of this PR is that none of my work seems to have
accomplished anything here. This is a placeholder for me to try other stuff.
---
If your project is set up for it, you can reply
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/621#discussion_r123514965
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/topology/ParserTopologyBuilder.java
---
@@ -132,7 +132,14 @@ public
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/621#discussion_r123512990
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/topology/ParserTopologyBuilder.java
---
@@ -132,7 +132,14 @@ public
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/621#discussion_r123512057
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
---
@@ -118,6 +116,27 @@ protected void
GitHub user cestella opened a pull request:
https://github.com/apache/metron/pull/621
METRON-1001: Allow metron to ingest parser metadata along with data
## Contributor Comments
Currently, we only ingest data in Metron. Often, there is valuable metadata
constructed up-stream
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/620
So, given that we're planning on moving to the REST API to enable
index-specific capabilities, can you detail exactly which API calls that this
PR will require? We can then create JIRAs
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/620#discussion_r122813875
--- Diff:
metron-interface/metron-alerts/src/app/service/save-search.service.ts ---
@@ -0,0 +1,191 @@
+/**
+ * Licensed to the Apache Software
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/608
This is great work, @nickwallen Nice refactoring and good functionality
changes.
Just curious for the uninitiated, if I'm running a non-X520 but another
DPDK supported card
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/620#discussion_r122807607
--- Diff:
metron-interface/metron-alerts/src/app/service/cluster-metadata.service.ts ---
@@ -0,0 +1,59 @@
+/**
+ * Licensed to the Apache Software
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/620
@ottobackwards yes, that's exactly what I mean. I want to make sure this
work is teed up to easily use the rest-api.
---
If your project is set up for it, you can reply to this email and have
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/620
I think it's ok to not integrate with REST for the first cut, but I'd like
to see the UI layer architected in such a way that when we do integrate with
the REST layer, it'll not span more than
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/617
+1 by inspection, great job @simonellistonball
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/602
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
GitHub user cestella reopened a pull request:
https://github.com/apache/metron/pull/615
METRON-995: Temporary variables in stellar enrichments which are maps do
not function as expected
## Contributor Comments
If you construct a variable which returns a map in a stellar
GitHub user cestella reopened a pull request:
https://github.com/apache/metron/pull/615
METRON-995: Temporary variables in stellar enrichments which are maps do
not function as expected
## Contributor Comments
If you construct a variable which returns a map in a stellar
Github user cestella closed the pull request at:
https://github.com/apache/metron/pull/615
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
GitHub user cestella opened a pull request:
https://github.com/apache/metron/pull/615
METRON-995: Temporary variables in stellar enrichments which are maps do
not function as expected
## Contributor Comments
If you construct a variable which returns a map in a stellar
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/606#discussion_r121049130
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/StellarCompiler.java
---
@@ -76,14 +93,80 @@ public Expression
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/609
@justinleet yes, agreed. I wanted to get this functionality in and have it
be backwards compatible for the short term, but ultimately, I think that it's a
better approach and we should work toward
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/609#discussion_r120932266
--- Diff: metron-platform/metron-enrichment/README.md ---
@@ -71,40 +73,94 @@ The `fieldMap`contents are of interest because they
contain the routing
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/609#discussion_r120924836
--- Diff:
metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/adapters/stellar/StellarAdapterTest.java
---
@@ -0,0 +1,134
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/609#discussion_r120924708
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/message/JSONFromPosition.java
---
@@ -40,10 +40,12 @@ public
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/602#discussion_r120890884
--- Diff: metron-deployment/roles/ambari_config/vars/single_node_vm.yml ---
@@ -102,7 +102,7 @@ configurations:
required_configurations
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/530
Regarding creating a separate parser, I mean, you have two telemetries that
are both JSON Maps, so you have 2 instances of the JSONMap parser topologies
running with different names.
---
If your
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/610
Ok, for testing this, I'd suggest we go through #468 's test plan. That
touches every place we use Stellar atm, I think.
---
If your project is set up for it, you can reply to this email and have
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/530
@ottobackwards they will be
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/530
I tend to agree with @ottobackwards that this is split. For better or
worse, we've preferred in this project to submit PRs to keep master fully
functional. I don't know of a better split
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/606#discussion_r120623833
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/StellarCompiler.java
---
@@ -76,14 +92,80 @@ public Expression
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/606#discussion_r120623722
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/StellarCompiler.java
---
@@ -40,14 +40,22 @@
import static
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/606#discussion_r120622230
--- Diff:
metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/WindowProcessor.java
---
@@ -89,7 +89,7 @@ public
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/612
+1 by inspection. This is good work; we inherited a lot of these from the
initial opensoc import, so normalizing and reducing the dead weight is a
valuable contribution. Thanks @justinleet
Github user cestella closed the pull request at:
https://github.com/apache/metron/pull/609
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
GitHub user cestella reopened a pull request:
https://github.com/apache/metron/pull/609
METRON-987: Allow stellar enrichments to be specified by a list as well as
a map
## Contributor Comments
Currently, stellar enrichments are specified by a map associating fields
GitHub user cestella opened a pull request:
https://github.com/apache/metron/pull/609
METRON-987: Allow stellar enrichments to be specified by a list as well as
a map
## Contributor Comments
Currently, stellar enrichments are specified by a map associating fields
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/601
+1 by inspection
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/572
Just piling in here and saying thanks @zezutom for the contribution. This
looks great!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/607#discussion_r120106798
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormStatusServiceImpl.java
---
@@ -49,6 +44,11 @@ public
GitHub user cestella opened a pull request:
https://github.com/apache/metron/pull/606
METRON-980: Short circuit operations for Stellar
## Contributor Comments
Stellar does not currently contain short circuit operations. In most
languages, this is an important optimization
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/600
I will happily assign over credit to you. Thanks for catching the bug
@justinleet
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/595#discussion_r117610139
--- Diff:
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/JoinBolt.java
---
@@ -90,11 +93,24 @@ public void prepare(Map
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/591
@JonZeolla Just let me know if you want me to commit in the meantime. I'd
be happy to, but I'll understand if you want to use this to test whether your
commit rights are restored.
---
If your
GitHub user cestella reopened a pull request:
https://github.com/apache/metron/pull/596
METRON-965: In the case where we specify the syncpolicy in the HDFS Writer,
we do not properly clone and end up syncing for every record
## Contributor Comments
Right now the SyncPolicy
GitHub user cestella opened a pull request:
https://github.com/apache/metron/pull/597
METRON-966: Pcap topology does not commit offsets
## Contributor Comments
Due to a bug in the storm-kafka-client in 1.1.0, offsets are not committed
in the pcap topology. This can be worked
Github user cestella closed the pull request at:
https://github.com/apache/metron/pull/596
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
GitHub user cestella reopened a pull request:
https://github.com/apache/metron/pull/596
METRON-965: In the case where we specify the syncpolicy in the HDFS Writer,
we do not properly clone and end up syncing for every record
## Contributor Comments
Right now the SyncPolicy
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/596#discussion_r117392489
--- Diff:
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/hdfs/HdfsWriter.java
---
@@ -85,7 +85,17 @@ public void init(Map
GitHub user cestella opened a pull request:
https://github.com/apache/metron/pull/592
METRON-961: Scary periodic exceptions in pcap during sync should be handled
and explained
## Contributor Comments
Right now, we have a couple scary exceptions that may happen periodically
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/585
+1 by inspection, great addition.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/585
Regarding the test data, it's not a sequence file in the format suitable
for reading in PcapInspector. Depending on the test case, we construct the
appropriate kafka representation. The value
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/585
@mmiklavc it depends on which test case you're talking about. We have two
modes of operation in the pcap topology and 2 test cases in the integration
test and these are defined by the flux
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/585#discussion_r116791165
--- Diff:
metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/PartitionHDFSWriter.java
---
@@ -183,14 +219,14 @@ private void
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/585#discussion_r116791055
--- Diff:
metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/deserializer/KeyValueDeserializer.java
---
@@ -28,6 +26,17
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/574
+1 by inspection
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/574
Wow, great addition. We have definitely not tested the HDFS Writer
sufficiently. ;)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/584
@ctramnitz While we don't really test on 1.1 yet, this shouldn't have an
impact on the topologies running there.
---
If your project is set up for it, you can reply to this email and have your
GitHub user cestella reopened a pull request:
https://github.com/apache/metron/pull/584
METRON-950: Migrate storm-kafka-client to 1.1
## Contributor Comments
There are MAJOR performance issues with the storm-kafka-client. Throughput
is roughly an order of magnitude faster
Github user cestella closed the pull request at:
https://github.com/apache/metron/pull/584
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/584#discussion_r116584769
--- Diff:
metron-platform/metron-storm-kafka-override/src/main/java/org/apache/storm/kafka/spout/internal/Timer.java
---
@@ -0,0 +1,58
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/588#discussion_r116583655
--- Diff:
metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/SquidIntegrationTest.java
---
@@ -29,6 +29,11 @@ String
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/589#discussion_r116582720
--- Diff:
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/hdfs/HdfsWriter.java
---
@@ -111,6 +121,16 @@ public BulkWriterResponse
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/588#discussion_r116582169
--- Diff:
metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/SquidIntegrationTest.java
---
@@ -29,6 +29,11 @@ String
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/584#discussion_r116581475
--- Diff:
metron-platform/metron-storm-kafka-override/src/main/java/org/apache/storm/kafka/spout/internal/Timer.java
---
@@ -0,0 +1,58
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/584#discussion_r116579916
--- Diff:
metron-platform/metron-storm-kafka/src/main/java/org/apache/metron/storm/kafka/flux/SimpleStormKafkaBuilder.java
---
@@ -202,31 +240,4 @@ public
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/584
@mmiklavc it is. To give guarantees beyond that, would require us to shade
and relocate the `metron-kafka-spout` project, which I did not want to do
without some good justification.
---
If your
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/584#discussion_r116574125
--- Diff:
metron-platform/metron-storm-kafka-override/src/main/java/org/apache/storm/kafka/spout/internal/Timer.java
---
@@ -0,0 +1,58
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/584#discussion_r116574075
--- Diff:
metron-platform/metron-storm-kafka/src/main/java/org/apache/metron/storm/kafka/flux/SimpleStormKafkaBuilder.java
---
@@ -165,13 +177,39
GitHub user cestella opened a pull request:
https://github.com/apache/metron/pull/589
METRON-955: Make the default sync policy for HDFS Writer be based on the
batch size
## Contributor Comments
Right now, we do a sync per record written in HDFS. This has performance
penalties
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/588
Yep, testing being a main one. YOu want to try out a parser without having
it shove its data into `enrichment`
---
If your project is set up for it, you can reply to this email and have your
GitHub user cestella opened a pull request:
https://github.com/apache/metron/pull/588
METRON-954: Create ability to change output topic of parsers from the CLI
## Contributor Comments
Right now, changing the output of the kafka topic for the parsers involves
either adjusting
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/587
agreed; there is more to this than just this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/584
The test plan for this should be mainly a smoke test..make sure data flows
through the default topologies in full dev.
---
If your project is set up for it, you can reply to this email
GitHub user cestella opened a pull request:
https://github.com/apache/incubator-metron/pull/584
METRON-950: Migrate storm-kafka-client to 1.1
## Contributor Comments
There are MAJOR performance issues with the storm-kafka-client. Throughput
is roughly an order of magnitude
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/566
+1 by inspection. this looks great
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/554
+1, great job, Mike. Thank you very much for taking this up.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
GitHub user cestella reopened a pull request:
https://github.com/apache/incubator-metron/pull/550
METRON-890: Intermittent unit test errors in shutting down Storm in memory
component
## Contributor Comments
Cross your fingers. This may or may not work. Please don't merge
Github user cestella closed the pull request at:
https://github.com/apache/incubator-metron/pull/550
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/550
wait, no, we're using 1.0.3 already.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/550
@ottobackwards oh man, that may be it. I'll see what bumping the storm
version does.
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user cestella closed the pull request at:
https://github.com/apache/incubator-metron/pull/550
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
GitHub user cestella reopened a pull request:
https://github.com/apache/incubator-metron/pull/550
METRON-890: Intermittent unit test errors in shutting down Storm in memory
component
## Contributor Comments
Cross your fingers. This may or may not work. Please don't merge
GitHub user cestella reopened a pull request:
https://github.com/apache/incubator-metron/pull/550
METRON-890: Intermittent unit test errors in shutting down Storm in memory
component
## Contributor Comments
Cross your fingers. This may or may not work. Please don't merge
Github user cestella closed the pull request at:
https://github.com/apache/incubator-metron/pull/550
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
GitHub user cestella reopened a pull request:
https://github.com/apache/incubator-metron/pull/550
METRON-890: Intermittent unit test errors in shutting down Storm in memory
component
## Contributor Comments
Cross your fingers. This may or may not work. Please don't merge
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/541
Yeah, no `0x` for specifying the regex syntax. I'll update the function
docs to point to the syntax guide. Also, I'm going to give a bit of a better
effort at the testing too. Good
Github user cestella closed the pull request at:
https://github.com/apache/incubator-metron/pull/550
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
GitHub user cestella reopened a pull request:
https://github.com/apache/incubator-metron/pull/550
METRON-890: Intermittent unit test errors in shutting down Storm in memory
component
## Contributor Comments
Cross your fingers. This may or may not work. Please don't merge
601 - 700 of 729 matches
Mail list logo
