[GitHub] metron pull request #863: METRON-1347: Indexing Topology should fail tuples ...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/863#discussion_r161087829 --- Diff: metron-platform/metron-indexing/README.md --- @@ -15,6 +15,12 @@ Indices are written in batch and the batch size and batch timeout are specified [Sensor Indexing Configuration](#sensor-indexing-configuration) via the `batchSize` and `batchTimeout` parameters. These configs are variable by sensor type. +## Minimal Assumptions for Message Structure + +At minimum, a message should have a `sensor.type` field. --- End diff -- whoops, yes, typo. corrected. ---
[GitHub] metron pull request #863: METRON-1347: Indexing Topology should fail tuples ...
Github user mmiklavc commented on a diff in the pull request: https://github.com/apache/metron/pull/863#discussion_r160250862 --- Diff: metron-platform/metron-indexing/README.md --- @@ -15,6 +15,12 @@ Indices are written in batch and the batch size and batch timeout are specified [Sensor Indexing Configuration](#sensor-indexing-configuration) via the `batchSize` and `batchTimeout` parameters. These configs are variable by sensor type. +## Minimal Assumptions for Message Structure + +At minimum, a message should have a `sensor.type` field. --- End diff -- sensor.type - did you mean source.type? ---
[GitHub] metron pull request #863: METRON-1347: Indexing Topology should fail tuples ...
Github user simonellistonball commented on a diff in the pull request:
https://github.com/apache/metron/pull/863#discussion_r156676868
--- Diff:
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/bolt/BulkMessageWriterBolt.java
---
@@ -229,17 +239,30 @@ public void execute(Tuple tuple) {
LOG.trace("Writing enrichment message: {}", message);
WriterConfiguration writerConfiguration =
configurationTransformation.apply(
new IndexingWriterConfiguration(bulkMessageWriter.getName(),
getConfigurations()));
- if(writerConfiguration.isDefault(sensorType)) {
-//want to warn, but not fail the tuple
-collector.reportError(new Exception("WARNING: Default and (likely)
unoptimized writer config used for " + bulkMessageWriter.getName() + " writer
and sensor " + sensorType));
+ if(sensorType == null) {
--- End diff --
Strictly speaking that's true, but by convention original_string should be
required. There is a broader topic about what should be required, but that
certainly doesn't belong in a comment on a PR.
---
[GitHub] metron pull request #863: METRON-1347: Indexing Topology should fail tuples ...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/863#discussion_r156676155 --- Diff: metron-platform/metron-indexing/README.md --- @@ -15,6 +15,12 @@ Indices are written in batch and the batch size and batch timeout are specified [Sensor Indexing Configuration](#sensor-indexing-configuration) via the `batchSize` and `batchTimeout` parameters. These configs are variable by sensor type. --- End diff -- So, strictly speaking messages really only require `source.type` (which I typo'd) and `timestamp` (which I should add). I'll fix that, but did I miss anything? ---
[GitHub] metron pull request #863: METRON-1347: Indexing Topology should fail tuples ...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/863#discussion_r156675356
--- Diff:
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/bolt/BulkMessageWriterBolt.java
---
@@ -229,17 +239,30 @@ public void execute(Tuple tuple) {
LOG.trace("Writing enrichment message: {}", message);
WriterConfiguration writerConfiguration =
configurationTransformation.apply(
new IndexingWriterConfiguration(bulkMessageWriter.getName(),
getConfigurations()));
- if(writerConfiguration.isDefault(sensorType)) {
-//want to warn, but not fail the tuple
-collector.reportError(new Exception("WARNING: Default and (likely)
unoptimized writer config used for " + bulkMessageWriter.getName() + " writer
and sensor " + sensorType));
+ if(sensorType == null) {
--- End diff --
Sure thing. Really the only two required are `timestamp` and
`source.type`. Did I miss any?
---
[GitHub] metron pull request #863: METRON-1347: Indexing Topology should fail tuples ...
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/863#discussion_r156674159
--- Diff:
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/bolt/BulkMessageWriterBolt.java
---
@@ -229,17 +239,30 @@ public void execute(Tuple tuple) {
LOG.trace("Writing enrichment message: {}", message);
WriterConfiguration writerConfiguration =
configurationTransformation.apply(
new IndexingWriterConfiguration(bulkMessageWriter.getName(),
getConfigurations()));
- if(writerConfiguration.isDefault(sensorType)) {
-//want to warn, but not fail the tuple
-collector.reportError(new Exception("WARNING: Default and (likely)
unoptimized writer config used for " + bulkMessageWriter.getName() + " writer
and sensor " + sensorType));
+ if(sensorType == null) {
--- End diff --
@ottobackwards which fields should we validate here?
---
[GitHub] metron pull request #863: METRON-1347: Indexing Topology should fail tuples ...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/863#discussion_r156372312 --- Diff: metron-platform/metron-indexing/README.md --- @@ -15,6 +15,12 @@ Indices are written in batch and the batch size and batch timeout are specified [Sensor Indexing Configuration](#sensor-indexing-configuration) via the `batchSize` and `batchTimeout` parameters. These configs are variable by sensor type. --- End diff -- Do we care, here in particular, that we have different required fields listed? Should this not be cumulative and include the required fields coming out of the parsers? ---
[GitHub] metron pull request #863: METRON-1347: Indexing Topology should fail tuples ...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/863#discussion_r156372868
--- Diff:
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/bolt/BulkMessageWriterBolt.java
---
@@ -229,17 +239,30 @@ public void execute(Tuple tuple) {
LOG.trace("Writing enrichment message: {}", message);
WriterConfiguration writerConfiguration =
configurationTransformation.apply(
new IndexingWriterConfiguration(bulkMessageWriter.getName(),
getConfigurations()));
- if(writerConfiguration.isDefault(sensorType)) {
-//want to warn, but not fail the tuple
-collector.reportError(new Exception("WARNING: Default and (likely)
unoptimized writer config used for " + bulkMessageWriter.getName() + " writer
and sensor " + sensorType));
+ if(sensorType == null) {
--- End diff --
Maybe we should validate all the required fields?
---
