[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 Docker setup for testi...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:


https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r240249191
  
--- Diff: docker/scripts/download_sample_pcaps.sh ---
@@ -0,0 +1,105 @@
+#!/usr/bin/env bash
+
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+shopt -s nocasematch
+
+#
+# Downloads sample pcap files to the data directory
+#
+
+function help {
+  echo " "
+  echo "usage: ${0}"
+  echo "--data-path[REQURIED] The pcap data path"
+  echo "-h/--help  Usage information."
+  echo " "
+  echo " "
+}
+
+DATA_PATH=
+
+# handle command line options
+for i in "$@"; do
+  case $i in
+  #
+  # DATA_PATH
+  #
+  #
+--data-path=*)
+  DATA_PATH="${i#*=}"
+  shift # past argument=value
+;;
+
+  #
+  # -h/--help
+  #
+-h | --help)
+  help
+  exit 0
+  shift # past argument with no value
+;;
+
+  #
+  # Unknown option
+  #
+*)
+  UNKNOWN_OPTION="${i#*=}"
+  echo "Error: unknown option: $UNKNOWN_OPTION"
+  help
+;;
+  esac
+done
+
+if [[ -z "$DATA_PATH" ]]; then
+  echo "DATA_PATH must be passed"
+  exit 1
+fi
+
+echo "Running download_sample_pcaps with "
+echo "DATA_PATH = $DATA_PATH"
+echo "==="
+
+for folder in nitroba example-traffic ssh ftp radius rfb; do
+  if [[ ! -d "${DATA_PATH}"/${folder} ]]; then
+mkdir -p "${DATA_PATH}"/${folder}
+  fi
+done
+
+if [[ ! -f "${DATA_PATH}"/example-traffic/exercise-traffic.pcap ]]; then
+  wget https://www.bro.org/static/traces/exercise-traffic.pcap -O 
"${DATA_PATH}"/example-traffic/exercise-traffic.pcap
--- End diff --

I had thought of the pcaps as being external to the images, that you may 
have different sets of pcaps that you want to test with, so leaving that 
configurable would be a good thing


---

[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 Docker setup for testi...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:


https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r240248699
  
--- Diff: docker/in_docker_scripts/build_bro_plugin.sh ---
@@ -0,0 +1,48 @@
+#!/usr/bin/env bash
+
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+shopt -s nocasematch
+
+#
+# Runs bro-package to build and install the plugin
+#
+
+cd /root/code || exit 1
+
+
+make clean
+
+rc=$?; if [[ ${rc} != 0 ]]; then
+  echo "ERROR cleaning project ${rc}" >>"${RUN_LOG_PATH}"
+  exit ${rc}
+fi
+
+cd /root || exit 1
+
+echo "" >>"${RUN_LOG_PATH}" 2>&1
+bro-pkg install code --force | tee "${RUN_LOG_PATH}"
--- End diff --

Because one of the uses of this is to regression test changes in the plugin 
itself.  Therefore you may _not_ rebuild the bro image, but want to rebuild the 
plugin


---

[GitHub] metron issue #1292: METRON-1925 Provide Verbose View of Profile Results in R...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1292
  
@nickwallen What I mean is that the returned value has the query parameters 
in it, so you have the data and the query you used for it.  Please excuse me if 
that is already the case


---

[GitHub] metron-bro-plugin-kafka issue #21: METRON-1911 Docker setup for testing bro ...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron-bro-plugin-kafka/pull/21
  
ok, this is fixed.  The bashSupport formatter in IntelliJ break the array 
assignment.  Sorry, i didn't redeploy after a format and  document run.

Also, I don't think your pcap directories match.   And- I have found I need 
to pass full paths for both of those scripts/parameters.

When I ran your line I ended up downloading to  a folder '~' in my current 
directory.

Sorry for the trouble.  If you could suggest a good bash formatter that 
would be super.




---

[GitHub] metron pull request #1245: METRON-1795: Initial Commit for Regular Expressio...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/1245#discussion_r239860797
  
--- Diff: 
metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/regex/RegularExpressionsParserTest.java
 ---
@@ -0,0 +1,152 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more 
contributor license
+ * agreements. See the NOTICE file distributed with this work for 
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the Apache 
License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the 
License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software 
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 
ANY KIND, either express
+ * or implied. See the License for the specific language governing 
permissions and limitations under
+ * the License.
+ */
+package org.apache.metron.parsers.regex;
+
+import org.json.simple.JSONObject;
+import org.json.simple.parser.JSONParser;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.junit.Assert.assertTrue;
+
+public class RegularExpressionsParserTest {
+
+  private RegularExpressionsParser regularExpressionsParser;
+  private JSONObject parserConfig;
+
+  @Before
+  public void setUp() throws Exception {
+regularExpressionsParser = new RegularExpressionsParser();
+  }
+
+  @Test
+  public void testSSHDParse() throws Exception {
+String message =
+"<38>Jun 20 15:01:17 deviceName sshd[11672]: Accepted publickey 
for prod from 22.22.22.22 port 5 ssh2";
+
+parserConfig = getJsonConfig(
+
Paths.get("src/test/resources/config/RegularExpressionsParserConfig.json").toString());
--- End diff --

Ok, this pr is actually simpler:  https://github.com/apache/metron/pull/1175


---

[GitHub] metron pull request #1245: METRON-1795: Initial Commit for Regular Expressio...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/1245#discussion_r239859491
  
--- Diff: 
metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/regex/RegularExpressionsParserTest.java
 ---
@@ -0,0 +1,152 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more 
contributor license
+ * agreements. See the NOTICE file distributed with this work for 
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the Apache 
License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the 
License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software 
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 
ANY KIND, either express
+ * or implied. See the License for the specific language governing 
permissions and limitations under
+ * the License.
+ */
+package org.apache.metron.parsers.regex;
+
+import org.json.simple.JSONObject;
+import org.json.simple.parser.JSONParser;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.junit.Assert.assertTrue;
+
+public class RegularExpressionsParserTest {
+
+  private RegularExpressionsParser regularExpressionsParser;
+  private JSONObject parserConfig;
+
+  @Before
+  public void setUp() throws Exception {
+regularExpressionsParser = new RegularExpressionsParser();
+  }
+
+  @Test
+  public void testSSHDParse() throws Exception {
+String message =
+"<38>Jun 20 15:01:17 deviceName sshd[11672]: Accepted publickey 
for prod from 22.22.22.22 port 5 ssh2";
+
+parserConfig = getJsonConfig(
+
Paths.get("src/test/resources/config/RegularExpressionsParserConfig.json").toString());
--- End diff --

The integration tests have a standard setup.  You have to do a few things, 
off the top of my head they are ( again check the pr and that parser for 
details):

- write the IntegrationTest that derives from the base 
- create a default sample configuration for your parser and put it in the 
configuration area
- add in the raw and parsed data in the integration testing module data 
directory for comparison


---

[GitHub] metron pull request #1245: METRON-1795: Initial Commit for Regular Expressio...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/1245#discussion_r239847486
  
--- Diff: 
metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/regex/RegularExpressionsParserTest.java
 ---
@@ -0,0 +1,152 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more 
contributor license
+ * agreements. See the NOTICE file distributed with this work for 
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the Apache 
License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the 
License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software 
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 
ANY KIND, either express
+ * or implied. See the License for the specific language governing 
permissions and limitations under
+ * the License.
+ */
+package org.apache.metron.parsers.regex;
+
+import org.json.simple.JSONObject;
+import org.json.simple.parser.JSONParser;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.junit.Assert.assertTrue;
+
+public class RegularExpressionsParserTest {
+
+  private RegularExpressionsParser regularExpressionsParser;
+  private JSONObject parserConfig;
+
+  @Before
+  public void setUp() throws Exception {
+regularExpressionsParser = new RegularExpressionsParser();
+  }
+
+  @Test
+  public void testSSHDParse() throws Exception {
+String message =
+"<38>Jun 20 15:01:17 deviceName sshd[11672]: Accepted publickey 
for prod from 22.22.22.22 port 5 ssh2";
+
+parserConfig = getJsonConfig(
+
Paths.get("src/test/resources/config/RegularExpressionsParserConfig.json").toString());
--- End diff --

When writing a new parser, it is important that you also implement the 
integration tests.  An example of a parser submittal that does this is : 
https://github.com/apache/metron/pull/1279


---

[GitHub] metron issue #1245: METRON-1795: Initial Commit for Regular Expressions Pars...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1245
  
Given we have the 5424 parser, and the 3164 parser in PR already, with 
chaining, perhaps this parser would be cleaner and easier to configure and 
understand if it was re-positioned ( wrt syslog ) as being a chained parser, 
that parser the MSG portion of either upstream parser.

Then your examples could be a bit simpler.



---

[GitHub] metron issue #1292: METRON-1925 Provide Verbose View of Profile Results in R...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1292
  
Could the return be a full json document, that includes the query 
parameters? I can see  doing these things and writing to file, and wanting more 
than just the data, but having the meta data ( query ) as well.



---

[GitHub] metron-bro-plugin-kafka issue #21: METRON-1911 Docker setup for testing bro ...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron-bro-plugin-kafka/pull/21
  
@nickwallen @JonZeolla, with the readme done, this is now ready for review


---

[GitHub] metron-bro-plugin-kafka issue #21: METRON-1911 Docker setup for testing bro ...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron-bro-plugin-kafka/pull/21
  
@nickwallen @JonZeolla, with the readme done, this is now ready for review


---

[GitHub] metron issue #1291: METRON-1922: Escaping incorrectly handled in current aes...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1291
  
+1 pending updating the dependency csv file


---

[GitHub] metron-bro-plugin-kafka issue #21: METRON-1911 [WIP] Docker setup for testin...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron-bro-plugin-kafka/pull/21
  
current send to end cmd is:

```bash
./scripts/download_sample_pcaps.sh --data-path=A_FULL_PATH &&  
./example_script.sh --skip-docker-build --leave-running --data-path=A_FULL_PATH 
&& ./scripts/docker_execute_process_data_dir.sh &&  
./scripts/docker_run_consume_bro_kafka.sh
```


---

[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:


https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r239090484
  
--- Diff: docker/in_docker_scripts/build_bro_plugin.sh ---
@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+shopt -s nocasematch
+
+cd /root || exit 1
+echo "" >> "${RUN_LOG_PATH}" 2>&1
+bro-pkg install code --force | tee "${RUN_LOG_PATH}"
+echo "" >> "${RUN_LOG_PATH}" 2>&1
+
+echo "" >> "${RUN_LOG_PATH}" 2>&1
+bro -N Apache::Kafka | tee v
+echo "" >> "${RUN_LOG_PATH}" 2>&1
+
+echo "@load packages" >> /usr/local/bro/share/bro/site/local.bro
+echo "redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, Conn::LOG, 
DPD::LOG, FTP::LOG, Files::LOG, Known::CERTS_LOG, SMTP::LOG, SSL::LOG, 
Weird::LOG, Notice::LOG, DHCP::LOG, SSH::LOG, Software::LOG, RADIUS::LOG, 
X509::LOG, Known::DEVICES_LOG, RFB::LOG, Stats::LOG, CaptureLoss::LOG, 
SIP::LOG);/'" >> /usr/local/bro/share/bro/site/local.bro
+echo "redef Kafka::topic_name = \"bro\";" >> 
/usr/local/bro/share/bro/site/local.bro
+echo "redef Kafka::tag_json = T;" >> 
/usr/local/bro/share/bro/site/local.bro
+echo "redef Kafka::kafka_conf = table([\"metadata.broker.list\"] = 
\"kafka:9092\");" >> /usr/local/bro/share/bro/site/local.bro
+echo "redef Kafka::logs_to_exclude = set(Conn::LOG, DHCP::LOG);" >> 
/usr/local/bro/share/bro/site/local.bro
+echo "redef Known::cert_tracking = ALL_HOSTS;" >> 
/usr/local/bro/share/bro/site/local.bro
+echo "redef Software::asset_tracking = ALL_HOSTS;" >> 
/usr/local/bro/share/bro/site/local.bro
+sed -i '86 a @load policy/protocols/dhcp/known-devices-and-hostnames.bro' 
/usr/local/bro/share/bro/site/local.bro
--- End diff --

done


---

[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:


https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238851540
  
--- Diff: docker/example_script.sh ---
@@ -0,0 +1,177 @@
+#!/usr/bin/env bash
+
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+shopt -s nocasematch
+
+CREATED_NETWORK_FLAG=false
+RAN_ZK_CONTAINER=false
+RAN_KAFKA_CONTAINER=false
+CREATED_BRO_CONTAINER=false
+RAN_BRO_CONTAINER=false
+
+SKIP_REBUILD_BRO=false
+LEAVE_RUNNING=false
+
+SCRIPT_DIR=./scripts
+CONTAINER_DIR=./containers/bro-localbuild-container
+CONTAINER_NAME=
+LOG_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && cd logs 
&& pwd )"
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo "--skip-docker-build Skip build of bro docker 
machine."
+ echo "--leave-running Do not stop containers after 
script.  The cleanup_containers.sh script should be run when done."
+ echo "-h/--help   Usage information."
+ echo " "
+ echo " "
+}
+
+function shutdown {
+
+  if [[ "$RAN_BRO_CONTAINER" = true ]]; then
+"${SCRIPT_DIR}"/stop_container.sh --container-name=bro
--- End diff --

the container name is not variable here, even though there is a var defined 
it isn't passed


---

[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:


https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238850336
  
--- Diff: docker/example_script.sh ---
@@ -0,0 +1,177 @@
+#!/usr/bin/env bash
+
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+shopt -s nocasematch
+
+CREATED_NETWORK_FLAG=false
+RAN_ZK_CONTAINER=false
+RAN_KAFKA_CONTAINER=false
+CREATED_BRO_CONTAINER=false
+RAN_BRO_CONTAINER=false
+
+SKIP_REBUILD_BRO=false
+LEAVE_RUNNING=false
+
+SCRIPT_DIR=./scripts
+CONTAINER_DIR=./containers/bro-localbuild-container
+CONTAINER_NAME=
+LOG_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && cd logs 
&& pwd )"
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo "--skip-docker-build Skip build of bro docker 
machine."
+ echo "--leave-running Do not stop containers after 
script.  The cleanup_containers.sh script should be run when done."
+ echo "-h/--help   Usage information."
+ echo " "
+ echo " "
+}
+
+function shutdown {
+
+  if [[ "$RAN_BRO_CONTAINER" = true ]]; then
+"${SCRIPT_DIR}"/stop_container.sh --container-name=bro
+  fi
+
+  if [[ "$RAN_KAFKA_CONTAINER" = true ]]; then
+"${SCRIPT_DIR}"/stop_container.sh --container-name=kafka
+  fi
+
+  if [[ "$RAN_ZK_CONTAINER" = true ]]; then
+"${SCRIPT_DIR}"/stop_container.sh --container-name=zookeeper
+  fi
+
+  if [[ "$CREATED_NETWORK_FLAG" = true ]]; then
+"${SCRIPT_DIR}"/destroy_docker_network.sh --network-name=bro-network
+  fi
+}
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+
+ #
+ # FORCE_DOCKER_BUILD
+ #
+ #   --skip-docker-build
+ #
+   --skip-docker-build)
+   SKIP_REBUILD_BRO=true
+   shift # past argument
+  ;;
+
+  #
+  # LEAVE_RUNNING
+  #
+  #   --leave-running
+  #
+--leave-running)
+LEAVE_RUNNING=true
+shift # past argument
+   ;;
+
+ #
+ # -h/--help
+ #
+  -h|--help)
+   help
+   exit 0
+   shift # past argument with no value
+  ;;
+ esac
+done
+EXTRA_ARGS="$@"
+echo "Running with "
+echo "SKIP_REBUILD_BRO = $SKIP_REBUILD_BRO"
+echo "==="
+
+# create the network
+bash "${SCRIPT_DIR}"/create_docker_network.sh --network-name=bro-network
+rc=$?; if [[ ${rc} != 0 ]]; then
+  shutdown
+  exit ${rc}
+else
+  CREATED_NETWORK_FLAG=true
+fi
+
+
+
+# run the zookeeper container
+bash "${SCRIPT_DIR}"/run_zookeeper_container.sh --network-name=bro-network
+rc=$?; if [[ ${rc} != 0 ]]; then
+  shutdown
+  exit ${rc}
+else
+  RAN_ZK_CONTAINER=true
+fi
+
+# run the kafka container
+bash "${SCRIPT_DIR}"/run_kafka_container.sh --network-name=bro-network
+rc=$?; if [[ ${rc} != 0 ]]; then
+  shutdown
+  exit ${rc}
+else
+  RAN_KAFKA_CONTAINER=true
+fi
+
+#build the bro container
+if [[ "$SKIP_REBUILD_BRO" = false ]] ; then
+  bash "${SCRIPT_DIR}"/build_container.sh \
+--container-directory="${CONTAINER_DIR}" \
+--container-name=bro-docker-container:latest
--- End diff --

done


---

[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:


https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238850284
  
--- Diff: docker/example_script.sh ---
@@ -0,0 +1,177 @@
+#!/usr/bin/env bash
+
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+shopt -s nocasematch
+
+CREATED_NETWORK_FLAG=false
+RAN_ZK_CONTAINER=false
+RAN_KAFKA_CONTAINER=false
+CREATED_BRO_CONTAINER=false
+RAN_BRO_CONTAINER=false
+
+SKIP_REBUILD_BRO=false
+LEAVE_RUNNING=false
+
+SCRIPT_DIR=./scripts
+CONTAINER_DIR=./containers/bro-localbuild-container
+CONTAINER_NAME=
+LOG_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && cd logs 
&& pwd )"
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo "--skip-docker-build Skip build of bro docker 
machine."
+ echo "--leave-running Do not stop containers after 
script.  The cleanup_containers.sh script should be run when done."
+ echo "-h/--help   Usage information."
+ echo " "
+ echo " "
+}
+
+function shutdown {
+
+  if [[ "$RAN_BRO_CONTAINER" = true ]]; then
+"${SCRIPT_DIR}"/stop_container.sh --container-name=bro
+  fi
+
+  if [[ "$RAN_KAFKA_CONTAINER" = true ]]; then
+"${SCRIPT_DIR}"/stop_container.sh --container-name=kafka
+  fi
+
+  if [[ "$RAN_ZK_CONTAINER" = true ]]; then
+"${SCRIPT_DIR}"/stop_container.sh --container-name=zookeeper
+  fi
+
+  if [[ "$CREATED_NETWORK_FLAG" = true ]]; then
+"${SCRIPT_DIR}"/destroy_docker_network.sh --network-name=bro-network
+  fi
+}
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+
+ #
+ # FORCE_DOCKER_BUILD
+ #
+ #   --skip-docker-build
+ #
+   --skip-docker-build)
+   SKIP_REBUILD_BRO=true
+   shift # past argument
+  ;;
+
+  #
+  # LEAVE_RUNNING
+  #
+  #   --leave-running
+  #
+--leave-running)
+LEAVE_RUNNING=true
+shift # past argument
+   ;;
+
+ #
+ # -h/--help
+ #
+  -h|--help)
+   help
+   exit 0
+   shift # past argument with no value
+  ;;
+ esac
+done
+EXTRA_ARGS="$@"
+echo "Running with "
+echo "SKIP_REBUILD_BRO = $SKIP_REBUILD_BRO"
+echo "==="
+
+# create the network
+bash "${SCRIPT_DIR}"/create_docker_network.sh --network-name=bro-network
+rc=$?; if [[ ${rc} != 0 ]]; then
+  shutdown
+  exit ${rc}
+else
+  CREATED_NETWORK_FLAG=true
+fi
+
+
+
+# run the zookeeper container
+bash "${SCRIPT_DIR}"/run_zookeeper_container.sh --network-name=bro-network
+rc=$?; if [[ ${rc} != 0 ]]; then
+  shutdown
+  exit ${rc}
+else
+  RAN_ZK_CONTAINER=true
+fi
+
+# run the kafka container
+bash "${SCRIPT_DIR}"/run_kafka_container.sh --network-name=bro-network
+rc=$?; if [[ ${rc} != 0 ]]; then
+  shutdown
+  exit ${rc}
+else
+  RAN_KAFKA_CONTAINER=true
+fi
+
+#build the bro container
+if [[ "$SKIP_REBUILD_BRO" = false ]] ; then
+  bash "${SCRIPT_DIR}"/build_container.sh \
+--container-directory="${CONTAINER_DIR}" \
+--container-name=bro-docker-container:latest
+
+  rc=$?; if [[ ${rc} != 0 ]]; then
+shutdown
+exit ${rc}
+  else
+CREATED_BRO_CONTAINER=true
+  fi
+fi
+
+
+#run the bro container
+#and optionally the passed script _IN_ the container
+bash "${SCRIPT_DIR}"/run_bro_container.sh 
--container-path="${CONTAINER_DIR}" \
+  --container-name=bro-docker-container:latest \
--- End diff --

done


---

[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:


https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238850232
  
--- Diff: docker/scripts/cleanup_containers.sh ---
@@ -0,0 +1,30 @@
+#!/usr/bin/env bash
+
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+shopt -s nocasematch
+
+SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+
+"${SCRIPT_DIR}"/stop_container.sh --container-name=bro
--- End diff --

done


---

[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:


https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238848200
  
--- Diff: docker/example_script.sh ---
@@ -0,0 +1,177 @@
+#!/usr/bin/env bash
+
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+shopt -s nocasematch
+
+CREATED_NETWORK_FLAG=false
+RAN_ZK_CONTAINER=false
+RAN_KAFKA_CONTAINER=false
+CREATED_BRO_CONTAINER=false
+RAN_BRO_CONTAINER=false
+
+SKIP_REBUILD_BRO=false
+LEAVE_RUNNING=false
+
+SCRIPT_DIR=./scripts
+CONTAINER_DIR=./containers/bro-localbuild-container
+CONTAINER_NAME=
+LOG_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && cd logs 
&& pwd )"
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo "--skip-docker-build Skip build of bro docker 
machine."
+ echo "--leave-running Do not stop containers after 
script.  The cleanup_containers.sh script should be run when done."
+ echo "-h/--help   Usage information."
+ echo " "
+ echo " "
+}
+
+function shutdown {
+
+  if [[ "$RAN_BRO_CONTAINER" = true ]]; then
+"${SCRIPT_DIR}"/stop_container.sh --container-name=bro
--- End diff --

done


---

[GitHub] metron-bro-plugin-kafka issue #21: METRON-1911 [WIP] Docker setup for testin...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron-bro-plugin-kafka/pull/21
  
@JonZeolla , I was going to ask you about that, what do you think the best 
practice is for these permissions?


---

[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:


https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238827202
  
--- Diff: docker/run_test.sh.template ---
@@ -0,0 +1,199 @@
+#!/usr/bin/env bash
--- End diff --

i changed it


---

[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:


https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238826943
  
--- Diff: docker/scripts/process_data_docker.sh ---
@@ -0,0 +1,29 @@
+#!/usr/bin/env bash
+
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+shopt -s nocasematch
+
+echo "exectuting proces_data_dir.sh in the bro docker container"
--- End diff --

fixed


---

[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:


https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238806017
  
--- Diff: docker/scripts/download_sample_pcaps.sh ---
@@ -0,0 +1,101 @@
+#!/usr/bin/env bash
+
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo "--data-paththe pcap data path"
+ echo "-h/--help   Usage information."
+ echo " "
+ echo " "
+}
+
+DATA_PATH=
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+ #
+ # DATA_PATH
+ #
+ #
+  --data-path=*)
+   DATA_PATH="${i#*=}"
+   shift # past argument=value
+  ;;
+
+ #
+ # -h/--help
+ #
+  -h|--help)
+   help
+   exit 0
+   shift # past argument with no value
+  ;;
+
+ #
+ # Unknown option
+ #
+  *)
+   UNKNOWN_OPTION="${i#*=}"
+   echo "Error: unknown option: $UNKNOWN_OPTION"
+   help
+  ;;
+ esac
+done
+
+if [[ -z "$DATA_PATH" ]]; then
+  echo "DATA_PATH must be passed"
+  exit 1
+fi
+
+echo "Running with "
+echo "DATA_PATH = $DATA_PATH"
+echo "==="
+
+for folder in nitroba example-traffic ssh ftp radius rfb; do
+  if [[ ! -d ${DATA_PATH}/${folder} ]]; then
+mkdir -p ${DATA_PATH}/${folder}
+  fi
+done
+
+if [[ ! -f ${DATA_PATH}/example-traffic/exercise-traffic.pcap ]]; then
+  wget https://www.bro.org/static/traces/exercise-traffic.pcap -O 
${DATA_PATH}/example-traffic/exercise-traffic.pcap
+fi
+if [[ ! -f ${DATA_PATH}/nitroba/nitroba.pcap ]]; then
+  wget 
http://downloads.digitalcorpora.org/corpora/network-packet-dumps/2008-nitroba/nitroba.pcap
 -O ${DATA_PATH}/nitroba/nitroba.pcap
+fi
+if [[ ! -f ${DATA_PATH}/ssh/ssh.pcap ]]; then
+  wget https://www.bro.org/static/traces/ssh.pcap -O 
${DATA_PATH}/ssh/ssh.pcap
+fi
+if [[ ! -f ${DATA_PATH}/ftp/ftp.pcap ]]; then
+  wget 
https://github.com/markofu/pcaps/blob/master/PracticalPacketAnalysis/ppa-capture-files/ftp.pcap?raw=true
 -O ${DATA_PATH}/ftp/ftp.pcap
+fi
+if [[ ! -f ${DATA_PATH}/radius/radius_localhost.pcapng ]]; then
+  wget 
https://github.com/EmpowerSecurityAcademy/wireshark/blob/master/radius_localhost.pcapng?raw=true
 -O ${DATA_PATH}/radius/radius_localhost.pcapng
--- End diff --

as above I'm not sure which repo you mean.  


---

[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:


https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238805671
  
--- Diff: docker/scripts/download_sample_pcaps.sh ---
@@ -0,0 +1,101 @@
+#!/usr/bin/env bash
+
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo "--data-paththe pcap data path"
+ echo "-h/--help   Usage information."
+ echo " "
+ echo " "
+}
+
+DATA_PATH=
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+ #
+ # DATA_PATH
+ #
+ #
+  --data-path=*)
+   DATA_PATH="${i#*=}"
+   shift # past argument=value
+  ;;
+
+ #
+ # -h/--help
+ #
+  -h|--help)
+   help
+   exit 0
+   shift # past argument with no value
+  ;;
+
+ #
+ # Unknown option
+ #
+  *)
+   UNKNOWN_OPTION="${i#*=}"
+   echo "Error: unknown option: $UNKNOWN_OPTION"
+   help
+  ;;
+ esac
+done
+
+if [[ -z "$DATA_PATH" ]]; then
+  echo "DATA_PATH must be passed"
+  exit 1
+fi
+
+echo "Running with "
+echo "DATA_PATH = $DATA_PATH"
+echo "==="
+
+for folder in nitroba example-traffic ssh ftp radius rfb; do
+  if [[ ! -d ${DATA_PATH}/${folder} ]]; then
+mkdir -p ${DATA_PATH}/${folder}
+  fi
+done
+
+if [[ ! -f ${DATA_PATH}/example-traffic/exercise-traffic.pcap ]]; then
+  wget https://www.bro.org/static/traces/exercise-traffic.pcap -O 
${DATA_PATH}/example-traffic/exercise-traffic.pcap
--- End diff --

what repo?


---

[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:


https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238805529
  
--- Diff: docker/scripts/create_docker_network.sh ---
@@ -0,0 +1,73 @@
+#!/usr/bin/env bash
+
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+shopt -s nocasematch
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo "--network-name  [REQUIRED] The docker network 
name"
+ echo "-h/--help   Usage information."
+ echo " "
+}
+
+NETWORK_NAME=
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+  #
+  # NETWORK_NAME
+  #
+  #
+  #
+--network-name=*)
+NETWORK_NAME="${i#*=}"
+shift # past argument=value
+   ;;
+ #
+ # -h/--help
+ #
+  -h|--help)
+   help
+   exit 0
+   shift # past argument with no value
+  ;;
+
+ #
+ # Unknown option
+ #
+  *)
+   UNKNOWN_OPTION="${i#*=}"
+   echo "Error: unknown option: $UNKNOWN_OPTION"
+   help
+  ;;
+ esac
+done
+
+if [[ -z "$NETWORK_NAME" ]]; then
+  echo "NETWORK_NAME must be passed"
+  exit 1
+fi
+
+docker network create "${NETWORK_NAME}"
+rc=$?; if [[ ${rc} != 0 ]]; then
+exit ${rc};
--- End diff --

oops


---

[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:


https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238805123
  
--- Diff: docker/in_docker_scripts/wait-for-it.sh ---
@@ -0,0 +1,161 @@
+#!/usr/bin/env bash
+#   Use this script to test if a given TCP host/port are available
+
+cmdname=$(basename $0)
--- End diff --

this works, and I don't want to change this script without great cause


---

[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:


https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238804842
  
--- Diff: docker/in_docker_scripts/build_bro_plugin.sh ---
@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+shopt -s nocasematch
+
+cd /root || exit 1
+echo "" >> "${RUN_LOG_PATH}" 2>&1
+bro-pkg install code --force | tee "${RUN_LOG_PATH}"
+echo "" >> "${RUN_LOG_PATH}" 2>&1
+
+echo "" >> "${RUN_LOG_PATH}" 2>&1
+bro -N Apache::Kafka | tee v
--- End diff --

oops


---

[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:


https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238804501
  
--- Diff: docker/in_docker_scripts/build_bro_plugin.sh ---
@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+shopt -s nocasematch
+
+cd /root || exit 1
+echo "" >> "${RUN_LOG_PATH}" 2>&1
+bro-pkg install code --force | tee "${RUN_LOG_PATH}"
--- End diff --

ok


---

[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:


https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238803331
  
--- Diff: docker/run_test.sh.template ---
@@ -0,0 +1,199 @@
+#!/usr/bin/env bash
--- End diff --

so the idea is that this is a sample, that they would modify, and would not 
get committed over and over again


---

[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:


https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238803083
  
--- Diff: docker/in_docker_scripts/wait-for-it.sh ---
@@ -0,0 +1,161 @@
+#!/usr/bin/env bash
+#   Use this script to test if a given TCP host/port are available
--- End diff --

all set


---

[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:


https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238802060
  
--- Diff: docker/in_docker_scripts/wait-for-it.sh ---
@@ -0,0 +1,161 @@
+#!/usr/bin/env bash
+#   Use this script to test if a given TCP host/port are available
--- End diff --

it is from our metron-contrib/docker, he may have gotten it from there.  
I'll have to check the L


---

[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:


https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238801412
  
--- Diff: docker/in_docker_scripts/process_data_dir.sh ---
@@ -0,0 +1,32 @@
+#!/usr/bin/env bash
+
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+shopt -s nocasematch
+
+cd /root || exit 1
+echo "" >> $RUN_LOG_PATH 2>&1
+
+if [ ! -d /root/data ] ; then
+  echo "DATA_PATH has not been set and mapped" >> $RUN_LOG_PATH 2&>1
+  exit 1
+fi
+
+ls /root/data
--- End diff --

no debugging


---

[GitHub] metron-bro-plugin-kafka issue #21: METRON-1911 [WIP] Docker setup for testin...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron-bro-plugin-kafka/pull/21
  
We will also have scripts that do docker exec etc to read from kafka etc.


---

[GitHub] metron-bro-plugin-kafka issue #21: METRON-1911 [WIP] Docker setup for testin...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron-bro-plugin-kafka/pull/21
  
The idea, at least at this point is that when in the container you will 
have the 'built in scripts' that we have, and whatever scripts you have mapped 
into the container too.  the run_test.sh.template currently doesn't pass 
through the script dir parameter, I have to build it out at that top level.

So this will allow the user to write their own test scripts, maintained 
outside the code etc.



---

[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:


https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238431951
  
--- Diff: docker/containers/bro-localbuild-container/Dockerfile ---
@@ -0,0 +1,49 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+FROM centos:7
+WORKDIR /root
+
+# install development tools
+RUN yum -y groupinstall "Development Tools"
+RUN yum -y install cmake make gcc gcc-c++ flex bison libpcap libpcap-devel 
openssl-devel python-devel swig zlib-devel perl cyrus-sasl cyrus-sasl-devel 
cyrus-sasl-gssapi git
+
+# install bro
+RUN curl -L https://www.bro.org/downloads/bro-2.5.5.tar.gz | tar xvz
+WORKDIR bro-2.5.5/
+RUN ./configure
+RUN make
+RUN make install
+ENV PATH="${PATH}:/usr/local/bro/bin"
+
+# install pip
+RUN yum -y update
+RUN yum -y install epel-release
+RUN yum -y install python-pip
+RUN pip install --upgrade pip
+
+# install bro-pkg
+RUN pip install bro-pkg
+ENV PATH="${PATH}:/usr/bin"
+RUN bro-pkg autoconfig
+
+# install librdkafka
+RUN curl -L https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz | 
tar xvz
+WORKDIR librdkafka-0.9.4/
--- End diff --

all set


---

[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:


https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238431923
  
--- Diff: docker/in_docker_scripts/build_bro.sh ---
@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
--- End diff --

all set


---

[GitHub] metron-bro-plugin-kafka issue #21: METRON-1911 [WIP] Docker setup for testin...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron-bro-plugin-kafka/pull/21
  
I had a lot of trouble with compose, so I did it with plain run and a 
shared network.
We can look at compose later.


---

[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...

[ottobackwards]

GitHub user ottobackwards opened a pull request:

https://github.com/apache/metron-bro-plugin-kafka/pull/21

METRON-1911 [WIP] Docker setup for testing bro plugin

Status>

To try:
```
  ~/src/apache/forks/metron-bro-plugin-kafka   docker  cd docker
  ~/src/apache/forks/metron-bro-plugin-kafka/docker   docker  
cp run_test.sh.template run_test.sh
 ~/src/apache/forks/metron-bro-plugin-kafka/docker   docker  
./run_test.sh
```
This will:

- create a network
- start kafka and zookeeper containers
- create bro container ( compile bro and kafka plugin )
- run bro container  -> drop you in a shell

then in the container
`>./built_in_scripts/run_test.sh`
Which will build the bro plugin and test it.


## todo

- start the bro container in the background
- add host side scripts to docker exec 'foo'  where foo is a script in the 
bro container
   - execute the plugin build that way
- configure bro for kafka
- run script to bro -r some pcaps


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron's Bro kafka writer 
plugin.

In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [na] Have you ensured that the full suite of tests and checks have been 
executed via:
  ```
  bro-pkg test $GITHUB_USERNAME/metron-bro-plugin-kafka --version $BRANCH
  ```
- [-] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [-] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [-] Have you verified the basic functionality of the build by building 
and running locally with Apache Metron's [Vagrant full-dev 
environment](https://github.com/apache/metron/tree/master/metron-deployment/development/centos6)
 or the equivalent?


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ottobackwards/metron-bro-plugin-kafka docker

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron-bro-plugin-kafka/pull/21.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #21


commit f7e464c6353da3b131c3c610ba154b435a24ee95
Author: Otto Fowler 
Date:   2018-12-03T20:23:13Z

initial docker testing harness




---

[GitHub] metron issue #1288: METRON-1916 Stellar Classpath Function Resolver Should H...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1288
  
Can we step back?  What dependencies where not present?  That may be the 
real issue for the users.


---

[GitHub] metron issue #1288: METRON-1916 Stellar Classpath Function Resolver Should H...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1288
  
In my mind we don't have a current state where Stellar is running but not 
all the functions in the class path are loaded.   This is different.  Before we 
would have crashed starting up.  Now we will run and crash later.


---

[GitHub] metron issue #1288: METRON-1916 Stellar Classpath Function Resolver Should H...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1288
  
@cestella should check in on this


---

[GitHub] metron issue #1288: METRON-1916 Stellar Classpath Function Resolver Should H...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1288
  
So, my main concern here is the state of stellar _after_ loading in this 
condition.  What happens if you call a function that failed to load?  What 
happens if it is nested in a match or a lambda?

Right now, it blows up, but when loaded everything is known to be 'ok'.  So 
this introduces a new state where we are loaded, but things that were supposed 
to load ( stellar functions introduced to the class path ) have failed.

I'm not sure this is that simple.  Would we _ever_ for example accept this 
state in the topology?


---

[GitHub] metron issue #1287: METORN-1915: Disable vagrant hostmanager in the developm...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1287
  
what breaks specifically?
having all the metron ports etc point to localhost seems to me that it 
would also break things or have conflicts etc.



---

[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1261
  
The base option is like quick dev, but quick dev also had metron installed, 
so starting it involved replacing the bits that where in there, we didn't 
maintain quick dev ansible w/o the metron stuff and 'other' ansible with the 
base.

I think that is a good idea, but for a follow on to this.This effort 
itself is or may be just a first step to a couple of things.

The base image would not replace this, but be the base image that this 
functionality uses.



---

[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/1261#discussion_r236385492
  
--- Diff: 
metron-deployment/development/centos6_docker_build/build_and_run.sh ---
@@ -0,0 +1,144 @@
+#!/usr/bin/env bash
+
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo "--skip-vagrant-up   skip vagrant up"
+ echo "--force-docker-buildforce build docker machine"
+ echo "--skip-tags='tag,tag2,tag3' the ansible skip tags"
+ echo "-h/--help   Usage information."
+ echo " "
+ echo "example: to skip vagrant up and force docker build with two tags"
+ echo "   build_and_run.sh -skip-vagrant-up --force-docker-build 
--skip-tags='solr,sensors'"
+ echo " "
+}
+
+SKIP_VAGRANT_UP=false
+FORCE_DOCKER_BUILD=false
+A_SKIP_TAGS="sensors,solr"
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+ #
+ # SKIP_VAGRANT_UP
+ #
+ #
+  --skip-vagrant-up)
+   SKIP_VAGRANT_UP=true
+   shift # past argument
+  ;;
+
+ #
+ # FORCE_DOCKER_BUILD
+ #
+ #   --force-docker-build
+ #
+   --force-docker-build)
+   FORCE_DOCKER_BUILD=true
+   shift # past argument
+  ;;
+
+ #
+ # SKIP_TAGS
+ #
+ #   --skip-tags='foo,bar'
+ #
+   --skip-tags=*)
+   A_SKIP_TAGS="${i#*=}"
+   shift # past argument=value
+  ;;
+
+ #
+ # -h/--help
+ #
+  -h|--help)
+   help
+   exit 0
+   shift # past argument with no value
+  ;;
+
+ #
+ # Unknown option
+ #
+  *)
+   UNKNOWN_OPTION="${i#*=}"
+   echo "Error: unknown option: $UNKNOWN_OPTION"
+   help
+  ;;
+ esac
+done
+
+echo "Running with "
+echo "SKIP_VAGRANT_UP= $SKIP_VAGRANT_UP"
+echo "FORCE_DOCKER_BUILD = $FORCE_DOCKER_BUILD"
+echo "SKIP_TAGS  = $A_SKIP_TAGS"
+echo "==="
+
+if [[ "$SKIP_VAGRANT_UP" = false ]]; then
+ vagrant up
+ rc=$?; if [[ $rc != 0 ]]; then exit $rc; fi
+fi
+
+VAGRANT_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+ANSIBLE_PATH=${VAGRANT_PATH}/ansible
+VAGRANT_KEY_PATH=${VAGRANT_PATH}/.vagrant/machines/node1/virtualbox
+
+# move over to the docker area
+cd ../docker || exit 1
+
+# Give the option to not build the docker container, which can take some 
time and not be necessary
+if [[ "$FORCE_DOCKER_BUILD" = true ]]; then
+ echo "docker build"
+ docker build -t metron-build-docker:latest .
+fi
+
+if [[ ! -d ~/.m2 ]]; then
+ mkdir ~/.m2
+fi
+
+DATE=`date`
+LOG_DATE=${DATE// /_}
+LOGNAME="metron-build-${LOG_DATE}.log"
+
+# get the node1 ip address so we can add it to the docker hosts
+NODE1_IP=$(awk '/^\s*hosts/{flag=1; next} /}]/{flag=0} flag' 
${VAGRANT_PATH}/Vagrantfile | grep  "^\s*ip:" | awk -F'"' '{print $2}')
--- End diff --

Fixed


---

[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/1261#discussion_r236375985
  
--- Diff: 
metron-deployment/development/centos6_docker_build/build_and_run.sh ---
@@ -0,0 +1,144 @@
+#!/usr/bin/env bash
+
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo "--skip-vagrant-up   skip vagrant up"
+ echo "--force-docker-buildforce build docker machine"
+ echo "--skip-tags='tag,tag2,tag3' the ansible skip tags"
+ echo "-h/--help   Usage information."
+ echo " "
+ echo "example: to skip vagrant up and force docker build with two tags"
+ echo "   build_and_run.sh -skip-vagrant-up --force-docker-build 
--skip-tags='solr,sensors'"
+ echo " "
+}
+
+SKIP_VAGRANT_UP=false
+FORCE_DOCKER_BUILD=false
+A_SKIP_TAGS="sensors,solr"
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+ #
+ # SKIP_VAGRANT_UP
+ #
+ #
+  --skip-vagrant-up)
+   SKIP_VAGRANT_UP=true
+   shift # past argument
+  ;;
+
+ #
+ # FORCE_DOCKER_BUILD
+ #
+ #   --force-docker-build
+ #
+   --force-docker-build)
+   FORCE_DOCKER_BUILD=true
+   shift # past argument
+  ;;
+
+ #
+ # SKIP_TAGS
+ #
+ #   --skip-tags='foo,bar'
+ #
+   --skip-tags=*)
+   A_SKIP_TAGS="${i#*=}"
+   shift # past argument=value
+  ;;
+
+ #
+ # -h/--help
+ #
+  -h|--help)
+   help
+   exit 0
+   shift # past argument with no value
+  ;;
+
+ #
+ # Unknown option
+ #
+  *)
+   UNKNOWN_OPTION="${i#*=}"
+   echo "Error: unknown option: $UNKNOWN_OPTION"
+   help
+  ;;
+ esac
+done
+
+echo "Running with "
+echo "SKIP_VAGRANT_UP= $SKIP_VAGRANT_UP"
+echo "FORCE_DOCKER_BUILD = $FORCE_DOCKER_BUILD"
+echo "SKIP_TAGS  = $A_SKIP_TAGS"
+echo "==="
+
+if [[ "$SKIP_VAGRANT_UP" = false ]]; then
+ vagrant up
+ rc=$?; if [[ $rc != 0 ]]; then exit $rc; fi
+fi
+
+VAGRANT_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+ANSIBLE_PATH=${VAGRANT_PATH}/ansible
+VAGRANT_KEY_PATH=${VAGRANT_PATH}/.vagrant/machines/node1/virtualbox
+
+# move over to the docker area
+cd ../docker || exit 1
+
+# Give the option to not build the docker container, which can take some 
time and not be necessary
+if [[ "$FORCE_DOCKER_BUILD" = true ]]; then
+ echo "docker build"
+ docker build -t metron-build-docker:latest .
+fi
+
+if [[ ! -d ~/.m2 ]]; then
+ mkdir ~/.m2
+fi
+
+DATE=`date`
+LOG_DATE=${DATE// /_}
+LOGNAME="metron-build-${LOG_DATE}.log"
+
+# get the node1 ip address so we can add it to the docker hosts
+NODE1_IP=$(awk '/^\s*hosts/{flag=1; next} /}]/{flag=0} flag' 
${VAGRANT_PATH}/Vagrantfile | grep  "^\s*ip:" | awk -F'"' '{print $2}')
--- End diff --

ahhh, yes, I did it for both.



---

[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1261
  
I think spellcheck is integrated into travis, so what we _can_ do is make 
shellcheck a build dependency and execute it through maven in the poms


---

[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1261
  
@mmiklavc I have it integrated into IntelliJ, though it isn't as obvious as 
java errors


---

[GitHub] metron issue #1249: METRON-1815: Separate metron-parsers into metron-parsers...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1249
  
Let's go with what @nickwallen said


---

[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/1261#discussion_r236301587
  
--- Diff: 
metron-deployment/development/centos6_docker_build/build_and_run.sh ---
@@ -0,0 +1,144 @@
+#!/usr/bin/env bash
+
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo "--skip-vagrant-up   skip vagrant up"
+ echo "--force-docker-buildforce build docker machine"
+ echo "--skip-tags='tag,tag2,tag3' the ansible skip tags"
+ echo "-h/--help   Usage information."
+ echo " "
+ echo "example: to skip vagrant up and force docker build with two tags"
+ echo "   build_and_run.sh -skip-vagrant-up --force-docker-build 
--skip-tags='solr,sensors'"
+ echo " "
+}
+
+SKIP_VAGRANT_UP=false
+FORCE_DOCKER_BUILD=false
+A_SKIP_TAGS="sensors,solr"
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+ #
+ # SKIP_VAGRANT_UP
+ #
+ #
+  --skip-vagrant-up)
+   SKIP_VAGRANT_UP=true
+   shift # past argument
+  ;;
+
+ #
+ # FORCE_DOCKER_BUILD
+ #
+ #   --force-docker-build
+ #
+   --force-docker-build)
+   FORCE_DOCKER_BUILD=true
+   shift # past argument
+  ;;
+
+ #
+ # SKIP_TAGS
+ #
+ #   --skip-tags='foo,bar'
+ #
+   --skip-tags=*)
+   A_SKIP_TAGS="${i#*=}"
+   shift # past argument=value
+  ;;
+
+ #
+ # -h/--help
+ #
+  -h|--help)
+   help
+   exit 0
+   shift # past argument with no value
+  ;;
+
+ #
+ # Unknown option
+ #
+  *)
+   UNKNOWN_OPTION="${i#*=}"
+   echo "Error: unknown option: $UNKNOWN_OPTION"
+   help
+  ;;
+ esac
+done
+
+echo "Running with "
+echo "SKIP_VAGRANT_UP= $SKIP_VAGRANT_UP"
+echo "FORCE_DOCKER_BUILD = $FORCE_DOCKER_BUILD"
+echo "SKIP_TAGS  = $A_SKIP_TAGS"
+echo "==="
+
+if [[ "$SKIP_VAGRANT_UP" = false ]]; then
+ vagrant up
+ rc=$?; if [[ $rc != 0 ]]; then exit $rc; fi
+fi
+
+VAGRANT_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+ANSIBLE_PATH=${VAGRANT_PATH}/ansible
+VAGRANT_KEY_PATH=${VAGRANT_PATH}/.vagrant/machines/node1/virtualbox
+
+# move over to the docker area
+cd ../docker || exit 1
+
+# Give the option to not build the docker container, which can take some 
time and not be necessary
+if [[ "$FORCE_DOCKER_BUILD" = true ]]; then
+ echo "docker build"
+ docker build -t metron-build-docker:latest .
+fi
+
+if [[ ! -d ~/.m2 ]]; then
+ mkdir ~/.m2
+fi
+
+DATE=`date`
+LOG_DATE=${DATE// /_}
+LOGNAME="metron-build-${LOG_DATE}.log"
+
+# get the node1 ip address so we can add it to the docker hosts
+NODE1_IP=$(awk '/^\s*hosts/{flag=1; next} /}]/{flag=0} flag' 
${VAGRANT_PATH}/Vagrantfile | grep  "^\s*ip:" | awk -F'"' '{print $2}')
--- End diff --

this change breaks the statement.  I disabled the warning.


---

[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/1261#discussion_r236292888
  
--- Diff: 
metron-deployment/development/centos6_docker_build/docker_run_ansible.sh ---
@@ -0,0 +1,46 @@
+#!/usr/bin/env bash
+
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+
+#
+# This script runs IN the docker container
+#
+
+cd /root/metron || exit 1
+
+# make sure we have the right c++ tools
+source /opt/rh/devtoolset-6/enable
+
+# give the option to skip building metron, in case they have already done 
so
+#read -p "  build metron? [yN] " -n 1 -r
+#echo
+#if [[ $REPLY =~ ^[Yy]$ ]]; then
+# USE TAGS
+#fi
--- End diff --

yes


---

[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/1261#discussion_r236262632
  
--- Diff: 
metron-deployment/development/centos6_docker_build/run_docker_only.sh ---
@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+VAGRANT_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+echo "setting the ansible configuration path"
+ANSIBLE_PATH=${VAGRANT_PATH}/ansible
+echo ${ANSIBLE_PATH}
+echo "setting the ssh key"
+VAGRANT_KEY_PATH=`pwd`/.vagrant/machines/node1/virtualbox
+echo ${VAGRANT_KEY_PATH}
+
+# move over to the docker area
+cd ../docker || exit 1
+pwd
--- End diff --

oh, the pwd I guess not, this is a debugging leftover.  I'll remove


---

[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/1261#discussion_r236262449
  
--- Diff: 
metron-deployment/development/centos6_docker_build/run_docker_only.sh ---
@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+VAGRANT_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+echo "setting the ansible configuration path"
+ANSIBLE_PATH=${VAGRANT_PATH}/ansible
+echo ${ANSIBLE_PATH}
+echo "setting the ssh key"
+VAGRANT_KEY_PATH=`pwd`/.vagrant/machines/node1/virtualbox
+echo ${VAGRANT_KEY_PATH}
+
+# move over to the docker area
+cd ../docker || exit 1
+pwd
--- End diff --

Not sure what you mean


---

[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/1261#discussion_r236262118
  
--- Diff: 
metron-deployment/development/centos6_docker_build/build_and_run.sh ---
@@ -0,0 +1,144 @@
+#!/usr/bin/env bash
+
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo "--skip-vagrant-up   skip vagrant up"
+ echo "--force-docker-buildforce build docker machine"
+ echo "--skip-tags='tag,tag2,tag3' the ansible skip tags"
+ echo "-h/--help   Usage information."
+ echo " "
+ echo "example: to skip vagrant up and force docker build with two tags"
+ echo "   build_and_run.sh -skip-vagrant-up --force-docker-build 
--skip-tags='solr,sensors'"
--- End diff --

ok


---

[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/1261#discussion_r236261928
  
--- Diff: 
metron-deployment/development/centos6_docker_build/docker_run_ansible.sh ---
@@ -0,0 +1,46 @@
+#!/usr/bin/env bash
+
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+
+#
+# This script runs IN the docker container
+#
+
+cd /root/metron || exit 1
+
+# make sure we have the right c++ tools
+source /opt/rh/devtoolset-6/enable
--- End diff --

ok


---

[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/1261#discussion_r236261518
  
--- Diff: 
metron-deployment/development/centos6_docker_build/build_and_run.sh ---
@@ -0,0 +1,144 @@
+#!/usr/bin/env bash
+
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo "--skip-vagrant-up   skip vagrant up"
+ echo "--force-docker-buildforce build docker machine"
+ echo "--skip-tags='tag,tag2,tag3' the ansible skip tags"
+ echo "-h/--help   Usage information."
+ echo " "
+ echo "example: to skip vagrant up and force docker build with two tags"
+ echo "   build_and_run.sh -skip-vagrant-up --force-docker-build 
--skip-tags='solr,sensors'"
+ echo " "
+}
+
+SKIP_VAGRANT_UP=false
+FORCE_DOCKER_BUILD=false
+A_SKIP_TAGS="sensors,solr"
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+ #
+ # SKIP_VAGRANT_UP
+ #
+ #
+  --skip-vagrant-up)
+   SKIP_VAGRANT_UP=true
+   shift # past argument
+  ;;
+
+ #
+ # FORCE_DOCKER_BUILD
+ #
+ #   --force-docker-build
+ #
+   --force-docker-build)
+   FORCE_DOCKER_BUILD=true
+   shift # past argument
+  ;;
+
+ #
+ # SKIP_TAGS
+ #
+ #   --skip-tags='foo,bar'
+ #
+   --skip-tags=*)
+   A_SKIP_TAGS="${i#*=}"
+   shift # past argument=value
+  ;;
+
+ #
+ # -h/--help
+ #
+  -h|--help)
+   help
+   exit 0
+   shift # past argument with no value
+  ;;
+
+ #
+ # Unknown option
+ #
+  *)
+   UNKNOWN_OPTION="${i#*=}"
+   echo "Error: unknown option: $UNKNOWN_OPTION"
+   help
+  ;;
+ esac
+done
+
+echo "Running with "
+echo "SKIP_VAGRANT_UP= $SKIP_VAGRANT_UP"
+echo "FORCE_DOCKER_BUILD = $FORCE_DOCKER_BUILD"
+echo "SKIP_TAGS  = $A_SKIP_TAGS"
+echo "==="
+
+if [[ "$SKIP_VAGRANT_UP" = false ]]; then
+ vagrant up
+ rc=$?; if [[ $rc != 0 ]]; then exit $rc; fi
+fi
+
+VAGRANT_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+ANSIBLE_PATH=${VAGRANT_PATH}/ansible
+VAGRANT_KEY_PATH=${VAGRANT_PATH}/.vagrant/machines/node1/virtualbox
+
+# move over to the docker area
+cd ../docker || exit 1
+
+# Give the option to not build the docker container, which can take some 
time and not be necessary
+if [[ "$FORCE_DOCKER_BUILD" = true ]]; then
+ echo "docker build"
+ docker build -t metron-build-docker:latest .
+fi
+
+if [[ ! -d ~/.m2 ]]; then
+ mkdir ~/.m2
+fi
+
+DATE=`date`
--- End diff --

ok



---

[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/1261#discussion_r236261064
  
--- Diff: 
metron-deployment/development/centos6_docker_build/build_and_run.sh ---
@@ -0,0 +1,144 @@
+#!/usr/bin/env bash
+
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo "--skip-vagrant-up   skip vagrant up"
+ echo "--force-docker-buildforce build docker machine"
+ echo "--skip-tags='tag,tag2,tag3' the ansible skip tags"
+ echo "-h/--help   Usage information."
+ echo " "
+ echo "example: to skip vagrant up and force docker build with two tags"
+ echo "   build_and_run.sh -skip-vagrant-up --force-docker-build 
--skip-tags='solr,sensors'"
+ echo " "
+}
+
+SKIP_VAGRANT_UP=false
+FORCE_DOCKER_BUILD=false
+A_SKIP_TAGS="sensors,solr"
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+ #
+ # SKIP_VAGRANT_UP
+ #
+ #
+  --skip-vagrant-up)
+   SKIP_VAGRANT_UP=true
+   shift # past argument
+  ;;
+
+ #
+ # FORCE_DOCKER_BUILD
+ #
+ #   --force-docker-build
+ #
+   --force-docker-build)
+   FORCE_DOCKER_BUILD=true
+   shift # past argument
+  ;;
+
+ #
+ # SKIP_TAGS
+ #
+ #   --skip-tags='foo,bar'
+ #
+   --skip-tags=*)
+   A_SKIP_TAGS="${i#*=}"
+   shift # past argument=value
+  ;;
+
+ #
+ # -h/--help
+ #
+  -h|--help)
+   help
+   exit 0
+   shift # past argument with no value
+  ;;
+
+ #
+ # Unknown option
+ #
+  *)
+   UNKNOWN_OPTION="${i#*=}"
+   echo "Error: unknown option: $UNKNOWN_OPTION"
+   help
+  ;;
+ esac
+done
+
+echo "Running with "
+echo "SKIP_VAGRANT_UP= $SKIP_VAGRANT_UP"
+echo "FORCE_DOCKER_BUILD = $FORCE_DOCKER_BUILD"
+echo "SKIP_TAGS  = $A_SKIP_TAGS"
+echo "==="
+
+if [[ "$SKIP_VAGRANT_UP" = false ]]; then
+ vagrant up
+ rc=$?; if [[ $rc != 0 ]]; then exit $rc; fi
+fi
+
+VAGRANT_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+ANSIBLE_PATH=${VAGRANT_PATH}/ansible
+VAGRANT_KEY_PATH=${VAGRANT_PATH}/.vagrant/machines/node1/virtualbox
+
+# move over to the docker area
+cd ../docker || exit 1
+
+# Give the option to not build the docker container, which can take some 
time and not be necessary
+if [[ "$FORCE_DOCKER_BUILD" = true ]]; then
+ echo "docker build"
+ docker build -t metron-build-docker:latest .
+fi
+
+if [[ ! -d ~/.m2 ]]; then
+ mkdir ~/.m2
+fi
+
+DATE=`date`
+LOG_DATE=${DATE// /_}
+LOGNAME="metron-build-${LOG_DATE}.log"
+
+# get the node1 ip address so we can add it to the docker hosts
+NODE1_IP=$(awk '/^\s*hosts/{flag=1; next} /}]/{flag=0} flag' 
${VAGRANT_PATH}/Vagrantfile | grep  "^\s*ip:" | awk -F'"' '{print $2}')
--- End diff --

ok


---

[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/1261#discussion_r236260538
  
--- Diff: 
metron-deployment/development/centos6_docker_build/build_and_run.sh ---
@@ -0,0 +1,144 @@
+#!/usr/bin/env bash
+
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo "--skip-vagrant-up   skip vagrant up"
+ echo "--force-docker-buildforce build docker machine"
+ echo "--skip-tags='tag,tag2,tag3' the ansible skip tags"
+ echo "-h/--help   Usage information."
+ echo " "
+ echo "example: to skip vagrant up and force docker build with two tags"
+ echo "   build_and_run.sh -skip-vagrant-up --force-docker-build 
--skip-tags='solr,sensors'"
+ echo " "
+}
+
+SKIP_VAGRANT_UP=false
+FORCE_DOCKER_BUILD=false
+A_SKIP_TAGS="sensors,solr"
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+ #
+ # SKIP_VAGRANT_UP
+ #
+ #
+  --skip-vagrant-up)
+   SKIP_VAGRANT_UP=true
+   shift # past argument
+  ;;
+
+ #
+ # FORCE_DOCKER_BUILD
+ #
+ #   --force-docker-build
+ #
+   --force-docker-build)
+   FORCE_DOCKER_BUILD=true
+   shift # past argument
+  ;;
+
+ #
+ # SKIP_TAGS
+ #
+ #   --skip-tags='foo,bar'
+ #
+   --skip-tags=*)
+   A_SKIP_TAGS="${i#*=}"
+   shift # past argument=value
+  ;;
+
+ #
+ # -h/--help
+ #
+  -h|--help)
+   help
+   exit 0
+   shift # past argument with no value
+  ;;
+
+ #
+ # Unknown option
+ #
+  *)
+   UNKNOWN_OPTION="${i#*=}"
--- End diff --

To make sure we consume the option and value


---

[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/1261#discussion_r236260621
  
--- Diff: 
metron-deployment/development/centos6_docker_build/build_and_run.sh ---
@@ -0,0 +1,144 @@
+#!/usr/bin/env bash
+
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo "--skip-vagrant-up   skip vagrant up"
+ echo "--force-docker-buildforce build docker machine"
+ echo "--skip-tags='tag,tag2,tag3' the ansible skip tags"
+ echo "-h/--help   Usage information."
+ echo " "
+ echo "example: to skip vagrant up and force docker build with two tags"
+ echo "   build_and_run.sh -skip-vagrant-up --force-docker-build 
--skip-tags='solr,sensors'"
+ echo " "
+}
+
+SKIP_VAGRANT_UP=false
+FORCE_DOCKER_BUILD=false
+A_SKIP_TAGS="sensors,solr"
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+ #
+ # SKIP_VAGRANT_UP
+ #
+ #
+  --skip-vagrant-up)
+   SKIP_VAGRANT_UP=true
+   shift # past argument
+  ;;
+
+ #
+ # FORCE_DOCKER_BUILD
+ #
+ #   --force-docker-build
+ #
+   --force-docker-build)
+   FORCE_DOCKER_BUILD=true
+   shift # past argument
+  ;;
+
+ #
+ # SKIP_TAGS
+ #
+ #   --skip-tags='foo,bar'
+ #
+   --skip-tags=*)
+   A_SKIP_TAGS="${i#*=}"
+   shift # past argument=value
+  ;;
+
+ #
+ # -h/--help
+ #
+  -h|--help)
+   help
+   exit 0
+   shift # past argument with no value
+  ;;
+
+ #
+ # Unknown option
+ #
+  *)
+   UNKNOWN_OPTION="${i#*=}"
--- End diff --

This is kind of boilerplate


---

[GitHub] metron pull request #1279: METRON-1893 Syslog RFC-3164 parser

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/1279#discussion_r236062925
  
--- Diff: metron-platform/metron-parsers/src/main/resources/META-INF/NOTICE 
---
@@ -37,4 +37,18 @@ Copyright 2006-2011 Google, Inc.
Apache Software Foundation that were originally developed at 
iClick, Inc.,
software copyright (c) 1999.
 
+ (ASLv2) simple-syslog-5424
+The following NOTICE information applies:
 
+simple-syslog-5424
+https://github.com/palindromicity/simple-syslog-5424
+
+Copyright 2018 simple-syslog-5424 authors.
+
+  (ASLv2) simple-syslog-5424
--- End diff --

nice catch, fixed. Thanks!


---

[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1261
  
@nickwallen Done



---

[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1261
  
@nickwallen, yeah,  I did prompts as I went along debugging.  I was 
thinking that folks may not like them.
I'll parameterize things.


---

[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1261
  
If you create an issue for your vagrant base machine with our hadoop / 
ambari already in it, you can assign it to me. @justinleet 



---

[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1261
  
It is possible to imagine a number of scenarios, including that, but also 
needing to build with new hadoop versions ( can't loose build from scratch ).

There are a number of things we can do down the road.

I think this work is going to help people enough in the near term to land 
it, while we discuss longer term refactoring and workflow.





---

[GitHub] metron issue #1249: METRON-1815: Separate metron-parsers into metron-parsers...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1249
  
wait, you don't have a plan??? lol. nm.  just wondering.  I have some old 
things around this, maybe I'll follow on.


---

[GitHub] metron issue #1249: METRON-1815: Separate metron-parsers into metron-parsers...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1249
  
I was going to suggest that we may want do it, but your use of _conflating_ 
has dissuaded me.
Have you a list of 'post' things to add that to perhaps?


---

[GitHub] metron issue #1249: METRON-1815: Separate metron-parsers into metron-parsers...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1249
  
All the hard coded paths for test resources should be removed.  Similar to 
how we build the test data paths for the integration tests.



---

[GitHub] metron-bro-plugin-kafka issue #18: Remove version from show-plugin btest out...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron-bro-plugin-kafka/pull/18
  
Thanks for the taking the time you did to submit the PR @dopheide-esnet. 

-1

Filling out the template to say "Yes reviewer, I tested, built, and check 
the lic etc " isn't a lot to ask and it helps the community, the reviewers a 
great deal.  Having the courtesy to follow a project/communities policies when 
submitting is not a strange thing to expect.

You can close this. 

Thanks again.


---

[GitHub] metron issue #1271: Updated org.apache.rat from 0.13-SNAPSHOT to 0.13

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1271
  
Hi, thanks for the contribution!  Can you have a look at filling out the pr 
template?  This this it is especially important to look at the part about 
saying you have tried:
`mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh `

Thanks!


---

[GitHub] metron issue #1268: METRON-1877: Nested IF ELSE statements can cause parse e...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1268
  
+1, great job


---

[GitHub] metron pull request #1268: METRON-1877: Nested IF ELSE statements can cause ...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/1268#discussion_r234305980
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/BasicStellarTest.java
 ---
@@ -916,6 +916,53 @@ public void testShortCircuit_conditional() throws 
Exception {
 Assert.assertEquals("foo", run("if NOT('metron' in [ 'metron', 
'metronicus'] ) then THROW('expression') else 'foo'", new HashMap<>()));
   }
 
--- End diff --

yeah, that is the idea.  I had to add more and more permutations ( I added 
tests for issues as @jjmeyer0 found them, or as @cestella found them )
If you have one mega case that works that is fine.  I started out 
differently, because I had things that did not work at the beginning and I 
added later ( MAP support etc )


---

[GitHub] metron pull request #1268: METRON-1877: Nested IF ELSE statements can cause ...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/1268#discussion_r234293803
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/BasicStellarTest.java
 ---
@@ -916,6 +916,53 @@ public void testShortCircuit_conditional() throws 
Exception {
 Assert.assertEquals("foo", run("if NOT('metron' in [ 'metron', 
'metronicus'] ) then THROW('expression') else 'foo'", new HashMap<>()));
   }
 
--- End diff --

if you look at https://github.com/apache/metron/pull/814, which was when I 
did checkpoint work for match, I had a lot of fallout to account for.  Maybe 
you can take a peak at the match tests.


---

[GitHub] metron pull request #1268: METRON-1877: Nested IF ELSE statements can cause ...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/1268#discussion_r234290697
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/BasicStellarTest.java
 ---
@@ -916,6 +916,53 @@ public void testShortCircuit_conditional() throws 
Exception {
 Assert.assertEquals("foo", run("if NOT('metron' in [ 'metron', 
'metronicus'] ) then THROW('expression') else 'foo'", new HashMap<>()));
   }
 
--- End diff --

These tests are good, but these evaluations should be tested with other 
things in the things in the if s

IF SOME_MORE_COMPLEX_FUNCTION THEN IF LAMBDA THEN MATCH ELSE B ELSE C
To make sure we are not messing the stack up.


---

[GitHub] metron pull request #1268: METRON-1877: Nested IF ELSE statements can cause ...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/1268#discussion_r234289745
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/StellarCompiler.java
 ---
@@ -219,14 +220,23 @@ public Object apply(ExpressionState state) {
   //short circuit the if/then/else
   instanceDeque.pop();
   if((Boolean)curr.getValue()) {
-//choose then
-skipElse = true;
+//choose then.  Need to make sure we're keeping track of 
nesting.
+skipElseCount++;
   } else {
 //choose else
+// Need to count in case we see another if-else, to avoid 
breaking on wrong else.
+int innerIfCount = 0;
 while (it.hasNext()) {
   Token t = it.next();
+  if (t.getUnderlyingType() == IfExpr.class) {
+innerIfCount++;
+  }
--- End diff --

wouldn't if, else if be more clear here?


---

[GitHub] metron issue #1265: METRON-1874 Create a Parser Debugger

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1265
  
Then maybe a function to evaluate stellar
EVALUATE_TRANSFORM(config,PARSER_PARSE(parser, input)


---

[GitHub] metron issue #1265: METRON-1874 Create a Parser Debugger

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1265
  
maybe that can be a follow on, it would be much better to load from disk -> 
split lines than to open an editor and cut and past from your sample log.


---

[GitHub] metron pull request #1249: METRON-1815: Separate metron-parsers into metron-...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/1249#discussion_r234230655
  
--- Diff: README.md ---
@@ -132,7 +132,7 @@ component and Apache Kafka as the unified data bus.
 
 Some high level links to the relevant subparts of the architecture, for
 more information:
--- End diff --

Fair enough, it just stuck out to me.  Kafka is a part of the storm 
support, not the parsing ( we could have a udp bolt or something, or a nifi 
bolt ) with where we are going.


---

[GitHub] metron issue #1265: METRON-1874 Create a Parser Debugger

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1265
  
Can we test parser chains?


---

[GitHub] metron issue #1265: METRON-1874 Create a Parser Debugger

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1265
  
Can we load files from disk?  It would be nice to not have had to setup 
kafka etc.


---

[GitHub] metron pull request #1249: METRON-1815: Separate metron-parsers into metron-...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/1249#discussion_r233920015
  
--- Diff: README.md ---
@@ -132,7 +132,7 @@ component and Apache Kafka as the unified data bus.
 
 Some high level links to the relevant subparts of the architecture, for
 more information:
--- End diff --

Going forward in this effort, as it is more compassable, it may _not_ be 
kafka that is providing the data, should we explicitly name kafka here?


---

[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1261
  
anyone have any ideas of the best way to time these things?


---

[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1261
  
We could also use more tags, for example I may want to skip building the 
java, but not skip building the RPMs.  Think of a dev flow -> I make my change, 
run my local tests and want to spin up full dev.  It is already built, but 
needs the rpms, I should be able to make ansible skip the compile/package of 
java and still do the rpms/debs


---

[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1261
  
@nickwallen That is an option, but not something I would pick as the goal 
from the outset if you know what I mean.


---

[GitHub] metron issue #1249: METRON-1815: Separate metron-parsers into metron-parsers...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1249
  
@justinleet I'm fine with that


---

[GitHub] metron issue #1260: METRON-1868: metron-committer-common incorrectly checkin...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1260
  
+1, sorry I missed this


---

[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1261
  
@mmiklavc That is basically correct. Except that the ansible version is 
2.5, since it only applies to this build, and allows for the yaml log 
formatting.  

Also, in the latest version, the ansible once again does the clean and 
build as opposed to the script.  I had a lot of problems getting the c++ picked 
up from ansible and moved the build out of it for the time, but the idea was 
always to have ansible run the metron_build, and that has returned.

The reasoning for the prompts to build the vagrant box and the docker ->
- if you are using this during development, IE> we are working ON ansible 
or ON docker, you may fail in the docker or ansible stage without modifying the 
vm, and thus not need to vagrant up again.  Likewise, you may not need to 
rebuild the docker machine if you have not made changes, _or_ you may in fact 
need to.   I added these flags as I developed.



---

[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1261
  
@nickwallen I did not think of that.  I was improving the process that 
stands today.  I think in a world where the posted image exists, we would still 
want the ability to try the latest ( to verify a fix pre-release etc ).


---

[GitHub] metron issue #1249: METRON-1815: Separate metron-parsers into metron-parsers...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1249
  
I'm ok with changing parsing to something else, just not parsers.


---

[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...

[ottobackwards]

GitHub user ottobackwards opened a pull request:

https://github.com/apache/metron/pull/1261

METRON-1860 [WIP] new developer option for ansible in docker to deploy to 
vagrant

The goal of this PR is to provide a new "full_dev" option for new and old 
users that does not require as much setup and version matching to try Metron's 
full dev environment.

Currently, the vagrant up command runs ansible locally, on the host 
machine, to build and deploy metron.  This means that the user must not only 
have Vagrant, Virtual Box and Docker, but must also have all the tools 
necessary to build metron ( maven, java, c++ 11 etc ) and run ansible ( python 
and others ).  It has been a common source of problems for new users to get 
started with Metron because of version or setup problems.

This PR introduces a new metron-deployment/development option which tries 
to address this problem, and make it possible for the user to only have 
Vagrant, VirtualBox and Docker ( along with a local copy of the source ) to be 
able to run full dev.

The new option starts the Vagrant VM, but does not run ansible in it. 
Instead it runs a docker container which contains all the tools/versions 
necessary, and that container is what runs ansible.

This is a WIP, I am looking for initial feedback.

##Testing
Have the correct versions of vagrant, virtual box and docker installed and 
running
```
cd $METRON_SRC_ROOT/metron-deployment/development/centos6_docker_build
./build_and_run.sh
```
Answer yes to building the vagrant box.
Answer yes to building the docker machine
Go grab a coffee.

The end result should be full dev running in the vagrant instance.

### Differences
- This does not support skip tags passed on the cli
- This does not support provision

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [-] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [-] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [-] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [-] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ottobackwards/metron vagrant-docker

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/1261.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1261


commit 1ac44c0d78b25113a5ebe807baccf160ef9af139
Author: Otto Fowler 
Date:   2018-11-09T03:51:56Z

Initial commit - This provides an environment where only vagrant and docker 
( and a copy of the metron codebase ) are required to run the Metron
full-dev vm with it's default setup.

This is the initial work, there will be refactorings

commit f54cc493f8e045d5f0ca669258f56a5576b92ad1
Author: Otto Fowler 
Date:   2018-11-13T17:32:38Z

refactored locations




---

[GitHub] metron issue #1171: METRON-1740 make parser support CONFIG and SYSTEM log ty...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1171
  
@nickwallen can we get a +1 on this?  @JonZeolla you are a +1 but not 
explicitly?


---

[GitHub] metron pull request #1014: METRON-1563 : Base Stellar assign for feature bra...

[ottobackwards]

Github user ottobackwards closed the pull request at:

https://github.com/apache/metron/pull/1014


---

[GitHub] metron issue #1014: METRON-1563 : Base Stellar assign for feature branch

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1014
  
This is in, not sure why it isn't closed


---

[GitHub] metron issue #1258: METRON-1864 fix issue where daylight savings breaks test...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1258
  
I think that the test logically tests what we are concerned about, that the 
format of the produced string matches the desired format.  Testing that the 
value is explicitly x,y,z isn't testing the format, it is testing something 
else.

I propose, with a mind to getting master and pr's building again, that I 
comment as requested but not change the test, and that any other ideas for 
improvement and or expansion of the tests be put to a jira for later work.


---

[GitHub] metron pull request #1258: METRON-1864 fix issue where daylight savings brea...

[ottobackwards]

GitHub user ottobackwards opened a pull request:

https://github.com/apache/metron/pull/1258

METRON-1864 fix issue where daylight savings breaks tests

Thanks to @justinleet for the fix.  I would have use a totally awesome but 
overdone regex.

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [-] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [-] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [-] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [-] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [-] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ottobackwards/metron 
stellar-datefunction-tz-fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/1258.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1258


commit b89068d2a52e5e6ceb7c925d6ca3e14ee8fc15f6
Author: Otto Fowler 
Date:   2018-11-08T16:12:40Z

fix issue where daylight savings breaks tests




---

[GitHub] metron issue #1249: METRON-1815: Separate metron-parsers into metron-parsers...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1249
  
Yes @justinleet exactly that.  Sorry I didn't explain it more betterer


---

[GitHub] metron issue #1255: METRON-1858 Fix Fireeye formatting and some optimization

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1255
  
Right @justinleet, I forgot about that.


---

[GitHub] metron issue #1251: METRON-1853: Add shutdown hook to Stellar BaseFunctionRe...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1251
  
+1


---

[GitHub] metron pull request #1255: METRON-1858 Fix Fireeye formatting and some optim...

[ottobackwards]

GitHub user ottobackwards opened a pull request:

https://github.com/apache/metron/pull/1255

METRON-1858  Fix Fireeye formatting and some optimization

Reformat and fix checkstyle.
Changed to not compile patterns every parse()

All tests should pass as before, no behavior changed.


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [-] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [-] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [-] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [-] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [-] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [-] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ottobackwards/metron fireeye-cleanup

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/1255.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1255


commit 1f6ced34aee604814c924d70e10a50418822abd8
Author: Otto Fowler 
Date:   2018-11-06T15:10:35Z

Initial Formatting

commit e314cd75c050f7170bb5334bb68dc76334fd26f9
Author: Otto Fowler 
Date:   2018-11-06T15:12:10Z

use methodhandles for logger

commit 5da4500290006c2b4082ce4e447f7fc0cb4035d5
Author: Otto Fowler 
Date:   2018-11-06T15:38:45Z

more cleanup and refactoring




---

[GitHub] metron issue #1251: METRON-1853: Add shutdown hook to Stellar BaseFunctionRe...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1251
  
That is great



---

[GitHub] metron issue #1249: METRON-1815: Separate metron-parsers into metron-parsers...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1249
  
@justinleet that is what I am saying as well.


---