[05/50] [abbrv] metron git commit: METRON-1594: KafkaWriter is asynchronous and may lose data on node failure (mmiklavc via mmiklavc) closes apache/metron#1045
METRON-1594: KafkaWriter is asynchronous and may lose data on node failure (mmiklavc via mmiklavc) closes apache/metron#1045 Project: http://git-wip-us.apache.org/repos/asf/metron/repo Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/523c38cf Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/523c38cf Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/523c38cf Branch: refs/heads/feature/METRON-1416-upgrade-solr Commit: 523c38cf6399e2e3974a51a2cd0fe47e096b0bdf Parents: b6808f7 Author: mmiklavc Authored: Wed Jun 6 15:40:55 2018 -0600 Committer: Michael Miklavcic Committed: Wed Jun 6 15:40:55 2018 -0600 -- metron-analytics/metron-profiler/README.md | 15 ++ .../src/main/flux/profiler/remote.yaml | 3 +- metron-platform/metron-common/README.md | 5 + .../metron/common/bolt/ConfiguredBolt.java | 14 +- .../common/bolt/ConfiguredEnrichmentBolt.java | 11 +- .../common/bolt/ConfiguredIndexingBolt.java | 13 +- .../common/bolt/ConfiguredParserBolt.java | 13 +- .../common/bolt/ConfiguredProfilerBolt.java | 9 +- .../common/configuration/Configurations.java| 7 +- .../configuration/EnrichmentConfigurations.java | 26 +++ .../configuration/IndexingConfigurations.java | 12 +- .../configuration/ParserConfigurations.java | 1 + .../profiler/ProfilerConfigurations.java| 25 +++ .../writer/ConfigurationStrategy.java | 44 + .../writer/ConfigurationsStrategies.java| 144 +++ .../writer/EnrichmentWriterConfiguration.java | 110 .../writer/ParserWriterConfiguration.java | 6 +- .../writer/ProfilerWriterConfiguration.java | 109 .../configuration/ParserConfigurationsTest.java | 120 + .../profiler/ProfilerConfigTest.java| 13 +- .../writer/ConfigurationsStrategiesTest.java| 79 + .../EnrichmentWriterConfigurationTest.java | 54 ++ .../writer/IndexingWriterConfigurationTest.java | 70 .../writer/ParserWriterConfigurationTest.java | 72 .../writer/ProfilerWriterConfigurationTest.java | 54 ++ .../writer/IndexingWriterConfigurationTest.java | 70 .../writer/ParserWriterConfigurationTest.java | 38 metron-platform/metron-enrichment/README.md | 15 +- .../main/flux/enrichment/remote-splitjoin.yaml | 9 +- .../main/flux/enrichment/remote-unified.yaml| 9 +- .../bolt/BulkMessageWriterBoltTest.java | 64 --- .../src/main/flux/indexing/batch/remote.yaml| 4 +- .../flux/indexing/random_access/remote.yaml | 4 +- metron-platform/metron-parsers/README.md| 6 +- .../apache/metron/parsers/bolt/ParserBolt.java | 108 +++- .../metron/parsers/bolt/WriterHandler.java | 63 +-- .../metron/parsers/bolt/ParserBoltTest.java | 176 +-- .../integration/WriterBoltIntegrationTest.java | 6 +- .../metron/writer/bolt/BatchTimeoutHelper.java | 6 +- .../writer/bolt/BulkMessageWriterBolt.java | 48 ++--- .../apache/metron/writer/kafka/KafkaWriter.java | 92 +++--- 41 files changed, 1426 insertions(+), 321 deletions(-) -- http://git-wip-us.apache.org/repos/asf/metron/blob/523c38cf/metron-analytics/metron-profiler/README.md -- diff --git a/metron-analytics/metron-profiler/README.md b/metron-analytics/metron-profiler/README.md index 79cdd44..1a17e10 100644 --- a/metron-analytics/metron-profiler/README.md +++ b/metron-analytics/metron-profiler/README.md @@ -538,6 +538,8 @@ The Profiler runs as an independent Storm topology. The configuration for the P | [`profiler.hbase.batch`](#profilerhbasebatch) | The number of puts that are written to HBase in a single batch. | [`profiler.hbase.flush.interval.seconds`](#profilerhbaseflushintervalseconds) | The maximum number of seconds between batch writes to HBase. | [`topology.kryo.register`](#topologykryoregister) | Storm will use Kryo serialization for these classes. +| [`profiler.writer.batchSize`](#profilerwriterbatchsize) | The number of records to batch when writing to Kakfa. +| [`profiler.writer.batchTimeout`](#profilerwriterbatchtimeout) | The timeout in ms for batching when writing to Kakfa. ### `profiler.input.topic` @@ -852,6 +854,19 @@ More information on accessing profile data can be found in the [Profiler Client] More information on using the [`STATS_*` functions in Stellar can be found here](../../metron-platform/metron-common). +### `profiler.writer.batchSize` + +*Default*: 15 + +The number of records to batch when writing to Kakfa. This is managed in the global configuration and does not require a
[05/50] [abbrv] metron git commit: METRON-1594: KafkaWriter is asynchronous and may lose data on node failure (mmiklavc via mmiklavc) closes apache/metron#1045
METRON-1594: KafkaWriter is asynchronous and may lose data on node failure (mmiklavc via mmiklavc) closes apache/metron#1045 Project: http://git-wip-us.apache.org/repos/asf/metron/repo Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/523c38cf Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/523c38cf Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/523c38cf Branch: refs/heads/feature/METRON-1554-pcap-query-panel Commit: 523c38cf6399e2e3974a51a2cd0fe47e096b0bdf Parents: b6808f7 Author: mmiklavc Authored: Wed Jun 6 15:40:55 2018 -0600 Committer: Michael Miklavcic Committed: Wed Jun 6 15:40:55 2018 -0600 -- metron-analytics/metron-profiler/README.md | 15 ++ .../src/main/flux/profiler/remote.yaml | 3 +- metron-platform/metron-common/README.md | 5 + .../metron/common/bolt/ConfiguredBolt.java | 14 +- .../common/bolt/ConfiguredEnrichmentBolt.java | 11 +- .../common/bolt/ConfiguredIndexingBolt.java | 13 +- .../common/bolt/ConfiguredParserBolt.java | 13 +- .../common/bolt/ConfiguredProfilerBolt.java | 9 +- .../common/configuration/Configurations.java| 7 +- .../configuration/EnrichmentConfigurations.java | 26 +++ .../configuration/IndexingConfigurations.java | 12 +- .../configuration/ParserConfigurations.java | 1 + .../profiler/ProfilerConfigurations.java| 25 +++ .../writer/ConfigurationStrategy.java | 44 + .../writer/ConfigurationsStrategies.java| 144 +++ .../writer/EnrichmentWriterConfiguration.java | 110 .../writer/ParserWriterConfiguration.java | 6 +- .../writer/ProfilerWriterConfiguration.java | 109 .../configuration/ParserConfigurationsTest.java | 120 + .../profiler/ProfilerConfigTest.java| 13 +- .../writer/ConfigurationsStrategiesTest.java| 79 + .../EnrichmentWriterConfigurationTest.java | 54 ++ .../writer/IndexingWriterConfigurationTest.java | 70 .../writer/ParserWriterConfigurationTest.java | 72 .../writer/ProfilerWriterConfigurationTest.java | 54 ++ .../writer/IndexingWriterConfigurationTest.java | 70 .../writer/ParserWriterConfigurationTest.java | 38 metron-platform/metron-enrichment/README.md | 15 +- .../main/flux/enrichment/remote-splitjoin.yaml | 9 +- .../main/flux/enrichment/remote-unified.yaml| 9 +- .../bolt/BulkMessageWriterBoltTest.java | 64 --- .../src/main/flux/indexing/batch/remote.yaml| 4 +- .../flux/indexing/random_access/remote.yaml | 4 +- metron-platform/metron-parsers/README.md| 6 +- .../apache/metron/parsers/bolt/ParserBolt.java | 108 +++- .../metron/parsers/bolt/WriterHandler.java | 63 +-- .../metron/parsers/bolt/ParserBoltTest.java | 176 +-- .../integration/WriterBoltIntegrationTest.java | 6 +- .../metron/writer/bolt/BatchTimeoutHelper.java | 6 +- .../writer/bolt/BulkMessageWriterBolt.java | 48 ++--- .../apache/metron/writer/kafka/KafkaWriter.java | 92 +++--- 41 files changed, 1426 insertions(+), 321 deletions(-) -- http://git-wip-us.apache.org/repos/asf/metron/blob/523c38cf/metron-analytics/metron-profiler/README.md -- diff --git a/metron-analytics/metron-profiler/README.md b/metron-analytics/metron-profiler/README.md index 79cdd44..1a17e10 100644 --- a/metron-analytics/metron-profiler/README.md +++ b/metron-analytics/metron-profiler/README.md @@ -538,6 +538,8 @@ The Profiler runs as an independent Storm topology. The configuration for the P | [`profiler.hbase.batch`](#profilerhbasebatch) | The number of puts that are written to HBase in a single batch. | [`profiler.hbase.flush.interval.seconds`](#profilerhbaseflushintervalseconds) | The maximum number of seconds between batch writes to HBase. | [`topology.kryo.register`](#topologykryoregister) | Storm will use Kryo serialization for these classes. +| [`profiler.writer.batchSize`](#profilerwriterbatchsize) | The number of records to batch when writing to Kakfa. +| [`profiler.writer.batchTimeout`](#profilerwriterbatchtimeout) | The timeout in ms for batching when writing to Kakfa. ### `profiler.input.topic` @@ -852,6 +854,19 @@ More information on accessing profile data can be found in the [Profiler Client] More information on using the [`STATS_*` functions in Stellar can be found here](../../metron-platform/metron-common). +### `profiler.writer.batchSize` + +*Default*: 15 + +The number of records to batch when writing to Kakfa. This is managed in the global configuration and does not require a
