[
https://issues.apache.org/jira/browse/METRON-673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bas van de Lustgraaf updated METRON-673:
----------------------------------------
Description:
metron-parser is unable to connect to a Kerberized Kafka cluster for consuming
and producing purposes.
The initial error from the storm worker.log that indicated it was not working:
{noformat}
2017-01-13 15:31:39.793 o.a.s.k.PartitionManager [INFO] Read partition
information from: /suricata/partition_0 --> null
{noformat}
This error is a known error, and could be solved by following the instruction
on the following page
http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_storm-component-guide/content/storm-kafka-kerb.html.
(In this case the the lack of spoutConfig.securityProtocol is the problem).
Also take into account that 2.5.0.0 has a issue with the KafkaSpout reading
from a secured Kafka cluster (attached: Hortonworks Technical Alert: Storm
kafkaspout to secure Kafka issue in HDP 2.5.0).
After changing the pom.xml of the incubator-project, to package the
metron-parsers JAR with the storm-kafka dependency version 1.0.1.2.5.3.0-37, I
tried to deploy the metron-parsers topology (attached:
extra_kafka_spout_config.json).
{noformat}
storm jar metron-parsers-0.3.0-uber.jar
org.apache.metron.parsers.topology.ParserTopologyCLI -k kn00:6667 -z kn00:2181
-s suricata –esc extra_kafka_spout_config.json
{noformat}
To set the spoutConfig.securityProtocol=PLAINTEXTSASL we used the
metron-parsers -esc parameter to pass the securityProtocol setting to the
topology. Unfortunately, Metron only allows a predefined list of parameters
that can be passed to the KafkaSpout. This approach resulted in an
IllegalArgumentException (source: storm jar).
{noformat}
java.lang.IllegalArgumentException: Configuration keys for spout config must be
one of:
retryDelayMaxMs,retryDelayMultiplier,retryInitialDelayMs,stateUpdateIntervalMs,bufferSizeBytes,fetchMaxWait,fetchSizeBytes,maxOffsetBehind,metricsTimeBucketSizeInSecs,socketTimeoutMs
at
org.apache.metron.common.spout.kafka.SpoutConfigOptions.coerceMap(SpoutConfigOptions.java:68)
at
org.apache.metron.parsers.topology.ParserTopologyCLI.readSpoutConfig(ParserTopologyCLI.java:340)
at
org.apache.metron.parsers.topology.ParserTopologyCLI.main(ParserTopologyCLI.java:291)
Caused by: java.lang.IllegalArgumentException: No enum constant
org.apache.metron.common.spout.kafka.SpoutConfigOptions.securityProtocol
at java.lang.Enum.valueOf(Enum.java:238)
at
org.apache.metron.common.spout.kafka.SpoutConfigOptions.valueOf(SpoutConfigOptions.java:28)
at
org.apache.metron.common.spout.kafka.SpoutConfigOptions.coerceMap(SpoutConfigOptions.java:64)
... 2 more
{noformat}
I have solved this by changing the Metron code, as showed below (or see the
git_diff.txt).
{noformat}
vi
/metron-platform/metron-common/src/main/java/org/apache/metron/common/spout/kafka/SpoutConfigOptions.java
### add line to function SpoutConfigOptions
securityProtocol( (config, val) -> config.securityProtocol = convertVal(val,
String.class)),
{noformat}
And change the KafkaProducer, to make it aware of the SASL_PLAINTEXT option.
{noformat}
vi
/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/kafka/KafkaWriter.java
b/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/kafka/KafkaWriter.java
### add line to class KafkaWriter
producerConfig.put("security.protocol", "SASL_PLAINTEXT");
{noformat}
The problem with my version is that the change for the producer is hard coded.
The ideal option is to make this as a parameter when starting the parser
topology, like with the consumer.
This also may prevent you running the enrichment or index part of metron.
was:
metron-parser is unable to connect to a Kerberized Kafka cluster for consuming
and producing purposes.
The initial error from the storm worker.log that indicated it was not working:
{noformat}
2017-01-13 15:31:39.793 o.a.s.k.PartitionManager [INFO] Read partition
information from: /suricata/partition_0 --> null
{noformat}
This error is a known error, and could be solved by following the instruction
on the following page
http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_storm-component-guide/content/storm-kafka-kerb.html.
(In this case the the lack of spoutConfig.securityProtocol is the problem).
Also take into account that 2.5.0.0 has a issue with the KafkaSpout reading
from a secured Kafka cluster (attached: Hortonworks Technical Alert: Storm
kafkaspout to secure Kafka issue in HDP 2.5.0).
After changing the pom.xml of the incubator-project, to package the
metron-parsers JAR with the storm-kafka dependency version 1.0.1.2.5.3.0-37, I
tried to deploy the metron-parsers topology (attached:
extra_kafka_spout_config.json).
{noformat}
storm jar metron-parsers-0.3.0-uber.jar
org.apache.metron.parsers.topology.ParserTopologyCLI -k kn00:6667 -z kn00:2181
-s suricata –esc extra_kafka_spout_config.json
{noformat}
To set the spoutConfig.securityProtocol=PLAINTEXTSASL we used the
metron-parsers -esc parameter to pass the securityProtocol setting to the
topology. Unfortunately, Metron only allows a predefined list of parameters
that can be passed to the KafkaSpout. This approach resulted in an
IllegalArgumentException (source: storm jar).
{noformat}
java.lang.IllegalArgumentException: Configuration keys for spout config must be
one of:
retryDelayMaxMs,retryDelayMultiplier,retryInitialDelayMs,stateUpdateIntervalMs,bufferSizeBytes,fetchMaxWait,fetchSizeBytes,maxOffsetBehind,metricsTimeBucketSizeInSecs,socketTimeoutMs
at
org.apache.metron.common.spout.kafka.SpoutConfigOptions.coerceMap(SpoutConfigOptions.java:68)
at
org.apache.metron.parsers.topology.ParserTopologyCLI.readSpoutConfig(ParserTopologyCLI.java:340)
at
org.apache.metron.parsers.topology.ParserTopologyCLI.main(ParserTopologyCLI.java:291)
Caused by: java.lang.IllegalArgumentException: No enum constant
org.apache.metron.common.spout.kafka.SpoutConfigOptions.securityProtocol
at java.lang.Enum.valueOf(Enum.java:238)
at
org.apache.metron.common.spout.kafka.SpoutConfigOptions.valueOf(SpoutConfigOptions.java:28)
at
org.apache.metron.common.spout.kafka.SpoutConfigOptions.coerceMap(SpoutConfigOptions.java:64)
... 2 more
{noformat}
I have solved this by changing the Metron code, as showed below (or see the
git_diff.txt).
{noformat}
vi
/metron-platform/metron-common/src/main/java/org/apache/metron/common/spout/kafka/SpoutConfigOptions.java
### add line to function SpoutConfigOptions
securityProtocol( (config, val) -> config.securityProtocol = convertVal(val,
String.class)),
{noformat}
And change the KafkaProducer, to make it aware of the SASL_PLAINTEXT option.
{noformat}
vi
/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/kafka/KafkaWriter.java
b/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/kafka/KafkaWriter.java
### add line to class KafkaWriter
producerConfig.put("security.protocol", "SASL_PLAINTEXT");
{noformat}
The problem with my version is that the change for the producer is hard coded.
The ideal option is to make this as a parameter when starting the parser
topology, like with the consumer.
> Unable to connect to a Secured Kafka cluster
> --------------------------------------------
>
> Key: METRON-673
> URL: https://issues.apache.org/jira/browse/METRON-673
> Project: Metron
> Issue Type: Improvement
> Affects Versions: 0.3.0
> Reporter: Bas van de Lustgraaf
> Attachments: extra_kafka_spout_config.json, git_diff.txt, Hortonworks
> Technical Alert Storm kafkaspout to secure Kafka issue in HDP 2.5.0.msg
>
>
> metron-parser is unable to connect to a Kerberized Kafka cluster for
> consuming and producing purposes.
> The initial error from the storm worker.log that indicated it was not working:
> {noformat}
> 2017-01-13 15:31:39.793 o.a.s.k.PartitionManager [INFO] Read partition
> information from: /suricata/partition_0 --> null
> {noformat}
> This error is a known error, and could be solved by following the instruction
> on the following page
> http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_storm-component-guide/content/storm-kafka-kerb.html.
> (In this case the the lack of spoutConfig.securityProtocol is the problem).
> Also take into account that 2.5.0.0 has a issue with the KafkaSpout reading
> from a secured Kafka cluster (attached: Hortonworks Technical Alert: Storm
> kafkaspout to secure Kafka issue in HDP 2.5.0).
> After changing the pom.xml of the incubator-project, to package the
> metron-parsers JAR with the storm-kafka dependency version 1.0.1.2.5.3.0-37,
> I tried to deploy the metron-parsers topology (attached:
> extra_kafka_spout_config.json).
> {noformat}
> storm jar metron-parsers-0.3.0-uber.jar
> org.apache.metron.parsers.topology.ParserTopologyCLI -k kn00:6667 -z
> kn00:2181 -s suricata –esc extra_kafka_spout_config.json
> {noformat}
> To set the spoutConfig.securityProtocol=PLAINTEXTSASL we used the
> metron-parsers -esc parameter to pass the securityProtocol setting to the
> topology. Unfortunately, Metron only allows a predefined list of parameters
> that can be passed to the KafkaSpout. This approach resulted in an
> IllegalArgumentException (source: storm jar).
> {noformat}
> java.lang.IllegalArgumentException: Configuration keys for spout config must
> be one of:
> retryDelayMaxMs,retryDelayMultiplier,retryInitialDelayMs,stateUpdateIntervalMs,bufferSizeBytes,fetchMaxWait,fetchSizeBytes,maxOffsetBehind,metricsTimeBucketSizeInSecs,socketTimeoutMs
> at
> org.apache.metron.common.spout.kafka.SpoutConfigOptions.coerceMap(SpoutConfigOptions.java:68)
> at
> org.apache.metron.parsers.topology.ParserTopologyCLI.readSpoutConfig(ParserTopologyCLI.java:340)
> at
> org.apache.metron.parsers.topology.ParserTopologyCLI.main(ParserTopologyCLI.java:291)
> Caused by: java.lang.IllegalArgumentException: No enum constant
> org.apache.metron.common.spout.kafka.SpoutConfigOptions.securityProtocol
> at java.lang.Enum.valueOf(Enum.java:238)
> at
> org.apache.metron.common.spout.kafka.SpoutConfigOptions.valueOf(SpoutConfigOptions.java:28)
> at
> org.apache.metron.common.spout.kafka.SpoutConfigOptions.coerceMap(SpoutConfigOptions.java:64)
> ... 2 more
> {noformat}
> I have solved this by changing the Metron code, as showed below (or see the
> git_diff.txt).
> {noformat}
> vi
> /metron-platform/metron-common/src/main/java/org/apache/metron/common/spout/kafka/SpoutConfigOptions.java
> ### add line to function SpoutConfigOptions
> securityProtocol( (config, val) -> config.securityProtocol = convertVal(val,
> String.class)),
> {noformat}
> And change the KafkaProducer, to make it aware of the SASL_PLAINTEXT option.
> {noformat}
> vi
> /metron-platform/metron-writer/src/main/java/org/apache/metron/writer/kafka/KafkaWriter.java
>
> b/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/kafka/KafkaWriter.java
> ### add line to class KafkaWriter
> producerConfig.put("security.protocol", "SASL_PLAINTEXT");
> {noformat}
> The problem with my version is that the change for the producer is hard
> coded. The ideal option is to make this as a parameter when starting the
> parser topology, like with the consumer.
> This also may prevent you running the enrichment or index part of metron.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
