[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 Docker setup for testi...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r240249191 --- Diff: docker/scripts/download_sample_pcaps.sh --- @@ -0,0 +1,105 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +shopt -s nocasematch + +# +# Downloads sample pcap files to the data directory +# + +function help { + echo " " + echo "usage: ${0}" + echo "--data-path[REQURIED] The pcap data path" + echo "-h/--help Usage information." + echo " " + echo " " +} + +DATA_PATH= + +# handle command line options +for i in "$@"; do + case $i in + # + # DATA_PATH + # + # +--data-path=*) + DATA_PATH="${i#*=}" + shift # past argument=value +;; + + # + # -h/--help + # +-h | --help) + help + exit 0 + shift # past argument with no value +;; + + # + # Unknown option + # +*) + UNKNOWN_OPTION="${i#*=}" + echo "Error: unknown option: $UNKNOWN_OPTION" + help +;; + esac +done + +if [[ -z "$DATA_PATH" ]]; then + echo "DATA_PATH must be passed" + exit 1 +fi + +echo "Running download_sample_pcaps with " +echo "DATA_PATH = $DATA_PATH" +echo "===" + +for folder in nitroba example-traffic ssh ftp radius rfb; do + if [[ ! -d "${DATA_PATH}"/${folder} ]]; then +mkdir -p "${DATA_PATH}"/${folder} + fi +done + +if [[ ! -f "${DATA_PATH}"/example-traffic/exercise-traffic.pcap ]]; then + wget https://www.bro.org/static/traces/exercise-traffic.pcap -O "${DATA_PATH}"/example-traffic/exercise-traffic.pcap --- End diff -- I had thought of the pcaps as being external to the images, that you may have different sets of pcaps that you want to test with, so leaving that configurable would be a good thing ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 Docker setup for testi...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r240248699 --- Diff: docker/in_docker_scripts/build_bro_plugin.sh --- @@ -0,0 +1,48 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +shopt -s nocasematch + +# +# Runs bro-package to build and install the plugin +# + +cd /root/code || exit 1 + + +make clean + +rc=$?; if [[ ${rc} != 0 ]]; then + echo "ERROR cleaning project ${rc}" >>"${RUN_LOG_PATH}" + exit ${rc} +fi + +cd /root || exit 1 + +echo "" >>"${RUN_LOG_PATH}" 2>&1 +bro-pkg install code --force | tee "${RUN_LOG_PATH}" --- End diff -- Because one of the uses of this is to regression test changes in the plugin itself. Therefore you may _not_ rebuild the bro image, but want to rebuild the plugin ---
[GitHub] metron issue #1292: METRON-1925 Provide Verbose View of Profile Results in R...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1292 @nickwallen What I mean is that the returned value has the query parameters in it, so you have the data and the query you used for it. Please excuse me if that is already the case ---
[GitHub] metron-bro-plugin-kafka issue #21: METRON-1911 Docker setup for testing bro ...
Github user ottobackwards commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/21 ok, this is fixed. The bashSupport formatter in IntelliJ break the array assignment. Sorry, i didn't redeploy after a format and document run. Also, I don't think your pcap directories match. And- I have found I need to pass full paths for both of those scripts/parameters. When I ran your line I ended up downloading to a folder '~' in my current directory. Sorry for the trouble. If you could suggest a good bash formatter that would be super. ---
[GitHub] metron pull request #1245: METRON-1795: Initial Commit for Regular Expressio...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1245#discussion_r239860797 --- Diff: metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/regex/RegularExpressionsParserTest.java --- @@ -0,0 +1,152 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more contributor license + * agreements. See the NOTICE file distributed with this work for additional information regarding + * copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance with the License. You may obtain a + * copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed under the License + * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express + * or implied. See the License for the specific language governing permissions and limitations under + * the License. + */ +package org.apache.metron.parsers.regex; + +import org.json.simple.JSONObject; +import org.json.simple.parser.JSONParser; +import org.junit.Before; +import org.junit.Test; + +import java.nio.file.Files; +import java.nio.file.Paths; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +import static org.junit.Assert.assertTrue; + +public class RegularExpressionsParserTest { + + private RegularExpressionsParser regularExpressionsParser; + private JSONObject parserConfig; + + @Before + public void setUp() throws Exception { +regularExpressionsParser = new RegularExpressionsParser(); + } + + @Test + public void testSSHDParse() throws Exception { +String message = +"<38>Jun 20 15:01:17 deviceName sshd[11672]: Accepted publickey for prod from 22.22.22.22 port 5 ssh2"; + +parserConfig = getJsonConfig( + Paths.get("src/test/resources/config/RegularExpressionsParserConfig.json").toString()); --- End diff -- Ok, this pr is actually simpler: https://github.com/apache/metron/pull/1175 ---
[GitHub] metron pull request #1245: METRON-1795: Initial Commit for Regular Expressio...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1245#discussion_r239859491 --- Diff: metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/regex/RegularExpressionsParserTest.java --- @@ -0,0 +1,152 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more contributor license + * agreements. See the NOTICE file distributed with this work for additional information regarding + * copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance with the License. You may obtain a + * copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed under the License + * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express + * or implied. See the License for the specific language governing permissions and limitations under + * the License. + */ +package org.apache.metron.parsers.regex; + +import org.json.simple.JSONObject; +import org.json.simple.parser.JSONParser; +import org.junit.Before; +import org.junit.Test; + +import java.nio.file.Files; +import java.nio.file.Paths; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +import static org.junit.Assert.assertTrue; + +public class RegularExpressionsParserTest { + + private RegularExpressionsParser regularExpressionsParser; + private JSONObject parserConfig; + + @Before + public void setUp() throws Exception { +regularExpressionsParser = new RegularExpressionsParser(); + } + + @Test + public void testSSHDParse() throws Exception { +String message = +"<38>Jun 20 15:01:17 deviceName sshd[11672]: Accepted publickey for prod from 22.22.22.22 port 5 ssh2"; + +parserConfig = getJsonConfig( + Paths.get("src/test/resources/config/RegularExpressionsParserConfig.json").toString()); --- End diff -- The integration tests have a standard setup. You have to do a few things, off the top of my head they are ( again check the pr and that parser for details): - write the IntegrationTest that derives from the base - create a default sample configuration for your parser and put it in the configuration area - add in the raw and parsed data in the integration testing module data directory for comparison ---
[GitHub] metron pull request #1245: METRON-1795: Initial Commit for Regular Expressio...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1245#discussion_r239847486 --- Diff: metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/regex/RegularExpressionsParserTest.java --- @@ -0,0 +1,152 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more contributor license + * agreements. See the NOTICE file distributed with this work for additional information regarding + * copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance with the License. You may obtain a + * copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed under the License + * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express + * or implied. See the License for the specific language governing permissions and limitations under + * the License. + */ +package org.apache.metron.parsers.regex; + +import org.json.simple.JSONObject; +import org.json.simple.parser.JSONParser; +import org.junit.Before; +import org.junit.Test; + +import java.nio.file.Files; +import java.nio.file.Paths; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +import static org.junit.Assert.assertTrue; + +public class RegularExpressionsParserTest { + + private RegularExpressionsParser regularExpressionsParser; + private JSONObject parserConfig; + + @Before + public void setUp() throws Exception { +regularExpressionsParser = new RegularExpressionsParser(); + } + + @Test + public void testSSHDParse() throws Exception { +String message = +"<38>Jun 20 15:01:17 deviceName sshd[11672]: Accepted publickey for prod from 22.22.22.22 port 5 ssh2"; + +parserConfig = getJsonConfig( + Paths.get("src/test/resources/config/RegularExpressionsParserConfig.json").toString()); --- End diff -- When writing a new parser, it is important that you also implement the integration tests. An example of a parser submittal that does this is : https://github.com/apache/metron/pull/1279 ---
[GitHub] metron issue #1245: METRON-1795: Initial Commit for Regular Expressions Pars...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1245 Given we have the 5424 parser, and the 3164 parser in PR already, with chaining, perhaps this parser would be cleaner and easier to configure and understand if it was re-positioned ( wrt syslog ) as being a chained parser, that parser the MSG portion of either upstream parser. Then your examples could be a bit simpler. ---
[GitHub] metron issue #1292: METRON-1925 Provide Verbose View of Profile Results in R...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1292 Could the return be a full json document, that includes the query parameters? I can see doing these things and writing to file, and wanting more than just the data, but having the meta data ( query ) as well. ---
[GitHub] metron-bro-plugin-kafka issue #21: METRON-1911 Docker setup for testing bro ...
Github user ottobackwards commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/21 @nickwallen @JonZeolla, with the readme done, this is now ready for review ---
[GitHub] metron-bro-plugin-kafka issue #21: METRON-1911 Docker setup for testing bro ...
Github user ottobackwards commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/21 @nickwallen @JonZeolla, with the readme done, this is now ready for review ---
[GitHub] metron issue #1291: METRON-1922: Escaping incorrectly handled in current aes...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1291 +1 pending updating the dependency csv file ---
[GitHub] metron-bro-plugin-kafka issue #21: METRON-1911 [WIP] Docker setup for testin...
Github user ottobackwards commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/21 current send to end cmd is: ```bash ./scripts/download_sample_pcaps.sh --data-path=A_FULL_PATH && ./example_script.sh --skip-docker-build --leave-running --data-path=A_FULL_PATH && ./scripts/docker_execute_process_data_dir.sh && ./scripts/docker_run_consume_bro_kafka.sh ``` ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r239090484 --- Diff: docker/in_docker_scripts/build_bro_plugin.sh --- @@ -0,0 +1,39 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +shopt -s nocasematch + +cd /root || exit 1 +echo "" >> "${RUN_LOG_PATH}" 2>&1 +bro-pkg install code --force | tee "${RUN_LOG_PATH}" +echo "" >> "${RUN_LOG_PATH}" 2>&1 + +echo "" >> "${RUN_LOG_PATH}" 2>&1 +bro -N Apache::Kafka | tee v +echo "" >> "${RUN_LOG_PATH}" 2>&1 + +echo "@load packages" >> /usr/local/bro/share/bro/site/local.bro +echo "redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, Conn::LOG, DPD::LOG, FTP::LOG, Files::LOG, Known::CERTS_LOG, SMTP::LOG, SSL::LOG, Weird::LOG, Notice::LOG, DHCP::LOG, SSH::LOG, Software::LOG, RADIUS::LOG, X509::LOG, Known::DEVICES_LOG, RFB::LOG, Stats::LOG, CaptureLoss::LOG, SIP::LOG);/'" >> /usr/local/bro/share/bro/site/local.bro +echo "redef Kafka::topic_name = \"bro\";" >> /usr/local/bro/share/bro/site/local.bro +echo "redef Kafka::tag_json = T;" >> /usr/local/bro/share/bro/site/local.bro +echo "redef Kafka::kafka_conf = table([\"metadata.broker.list\"] = \"kafka:9092\");" >> /usr/local/bro/share/bro/site/local.bro +echo "redef Kafka::logs_to_exclude = set(Conn::LOG, DHCP::LOG);" >> /usr/local/bro/share/bro/site/local.bro +echo "redef Known::cert_tracking = ALL_HOSTS;" >> /usr/local/bro/share/bro/site/local.bro +echo "redef Software::asset_tracking = ALL_HOSTS;" >> /usr/local/bro/share/bro/site/local.bro +sed -i '86 a @load policy/protocols/dhcp/known-devices-and-hostnames.bro' /usr/local/bro/share/bro/site/local.bro --- End diff -- done ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238851540 --- Diff: docker/example_script.sh --- @@ -0,0 +1,177 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +shopt -s nocasematch + +CREATED_NETWORK_FLAG=false +RAN_ZK_CONTAINER=false +RAN_KAFKA_CONTAINER=false +CREATED_BRO_CONTAINER=false +RAN_BRO_CONTAINER=false + +SKIP_REBUILD_BRO=false +LEAVE_RUNNING=false + +SCRIPT_DIR=./scripts +CONTAINER_DIR=./containers/bro-localbuild-container +CONTAINER_NAME= +LOG_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && cd logs && pwd )" + +function help { + echo " " + echo "usage: ${0}" + echo "--skip-docker-build Skip build of bro docker machine." + echo "--leave-running Do not stop containers after script. The cleanup_containers.sh script should be run when done." + echo "-h/--help Usage information." + echo " " + echo " " +} + +function shutdown { + + if [[ "$RAN_BRO_CONTAINER" = true ]]; then +"${SCRIPT_DIR}"/stop_container.sh --container-name=bro --- End diff -- the container name is not variable here, even though there is a var defined it isn't passed ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238850336 --- Diff: docker/example_script.sh --- @@ -0,0 +1,177 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +shopt -s nocasematch + +CREATED_NETWORK_FLAG=false +RAN_ZK_CONTAINER=false +RAN_KAFKA_CONTAINER=false +CREATED_BRO_CONTAINER=false +RAN_BRO_CONTAINER=false + +SKIP_REBUILD_BRO=false +LEAVE_RUNNING=false + +SCRIPT_DIR=./scripts +CONTAINER_DIR=./containers/bro-localbuild-container +CONTAINER_NAME= +LOG_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && cd logs && pwd )" + +function help { + echo " " + echo "usage: ${0}" + echo "--skip-docker-build Skip build of bro docker machine." + echo "--leave-running Do not stop containers after script. The cleanup_containers.sh script should be run when done." + echo "-h/--help Usage information." + echo " " + echo " " +} + +function shutdown { + + if [[ "$RAN_BRO_CONTAINER" = true ]]; then +"${SCRIPT_DIR}"/stop_container.sh --container-name=bro + fi + + if [[ "$RAN_KAFKA_CONTAINER" = true ]]; then +"${SCRIPT_DIR}"/stop_container.sh --container-name=kafka + fi + + if [[ "$RAN_ZK_CONTAINER" = true ]]; then +"${SCRIPT_DIR}"/stop_container.sh --container-name=zookeeper + fi + + if [[ "$CREATED_NETWORK_FLAG" = true ]]; then +"${SCRIPT_DIR}"/destroy_docker_network.sh --network-name=bro-network + fi +} + +# handle command line options +for i in "$@"; do + case $i in + + # + # FORCE_DOCKER_BUILD + # + # --skip-docker-build + # + --skip-docker-build) + SKIP_REBUILD_BRO=true + shift # past argument + ;; + + # + # LEAVE_RUNNING + # + # --leave-running + # +--leave-running) +LEAVE_RUNNING=true +shift # past argument + ;; + + # + # -h/--help + # + -h|--help) + help + exit 0 + shift # past argument with no value + ;; + esac +done +EXTRA_ARGS="$@" +echo "Running with " +echo "SKIP_REBUILD_BRO = $SKIP_REBUILD_BRO" +echo "===" + +# create the network +bash "${SCRIPT_DIR}"/create_docker_network.sh --network-name=bro-network +rc=$?; if [[ ${rc} != 0 ]]; then + shutdown + exit ${rc} +else + CREATED_NETWORK_FLAG=true +fi + + + +# run the zookeeper container +bash "${SCRIPT_DIR}"/run_zookeeper_container.sh --network-name=bro-network +rc=$?; if [[ ${rc} != 0 ]]; then + shutdown + exit ${rc} +else + RAN_ZK_CONTAINER=true +fi + +# run the kafka container +bash "${SCRIPT_DIR}"/run_kafka_container.sh --network-name=bro-network +rc=$?; if [[ ${rc} != 0 ]]; then + shutdown + exit ${rc} +else + RAN_KAFKA_CONTAINER=true +fi + +#build the bro container +if [[ "$SKIP_REBUILD_BRO" = false ]] ; then + bash "${SCRIPT_DIR}"/build_container.sh \ +--container-directory="${CONTAINER_DIR}" \ +--container-name=bro-docker-container:latest --- End diff -- done ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238850284 --- Diff: docker/example_script.sh --- @@ -0,0 +1,177 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +shopt -s nocasematch + +CREATED_NETWORK_FLAG=false +RAN_ZK_CONTAINER=false +RAN_KAFKA_CONTAINER=false +CREATED_BRO_CONTAINER=false +RAN_BRO_CONTAINER=false + +SKIP_REBUILD_BRO=false +LEAVE_RUNNING=false + +SCRIPT_DIR=./scripts +CONTAINER_DIR=./containers/bro-localbuild-container +CONTAINER_NAME= +LOG_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && cd logs && pwd )" + +function help { + echo " " + echo "usage: ${0}" + echo "--skip-docker-build Skip build of bro docker machine." + echo "--leave-running Do not stop containers after script. The cleanup_containers.sh script should be run when done." + echo "-h/--help Usage information." + echo " " + echo " " +} + +function shutdown { + + if [[ "$RAN_BRO_CONTAINER" = true ]]; then +"${SCRIPT_DIR}"/stop_container.sh --container-name=bro + fi + + if [[ "$RAN_KAFKA_CONTAINER" = true ]]; then +"${SCRIPT_DIR}"/stop_container.sh --container-name=kafka + fi + + if [[ "$RAN_ZK_CONTAINER" = true ]]; then +"${SCRIPT_DIR}"/stop_container.sh --container-name=zookeeper + fi + + if [[ "$CREATED_NETWORK_FLAG" = true ]]; then +"${SCRIPT_DIR}"/destroy_docker_network.sh --network-name=bro-network + fi +} + +# handle command line options +for i in "$@"; do + case $i in + + # + # FORCE_DOCKER_BUILD + # + # --skip-docker-build + # + --skip-docker-build) + SKIP_REBUILD_BRO=true + shift # past argument + ;; + + # + # LEAVE_RUNNING + # + # --leave-running + # +--leave-running) +LEAVE_RUNNING=true +shift # past argument + ;; + + # + # -h/--help + # + -h|--help) + help + exit 0 + shift # past argument with no value + ;; + esac +done +EXTRA_ARGS="$@" +echo "Running with " +echo "SKIP_REBUILD_BRO = $SKIP_REBUILD_BRO" +echo "===" + +# create the network +bash "${SCRIPT_DIR}"/create_docker_network.sh --network-name=bro-network +rc=$?; if [[ ${rc} != 0 ]]; then + shutdown + exit ${rc} +else + CREATED_NETWORK_FLAG=true +fi + + + +# run the zookeeper container +bash "${SCRIPT_DIR}"/run_zookeeper_container.sh --network-name=bro-network +rc=$?; if [[ ${rc} != 0 ]]; then + shutdown + exit ${rc} +else + RAN_ZK_CONTAINER=true +fi + +# run the kafka container +bash "${SCRIPT_DIR}"/run_kafka_container.sh --network-name=bro-network +rc=$?; if [[ ${rc} != 0 ]]; then + shutdown + exit ${rc} +else + RAN_KAFKA_CONTAINER=true +fi + +#build the bro container +if [[ "$SKIP_REBUILD_BRO" = false ]] ; then + bash "${SCRIPT_DIR}"/build_container.sh \ +--container-directory="${CONTAINER_DIR}" \ +--container-name=bro-docker-container:latest + + rc=$?; if [[ ${rc} != 0 ]]; then +shutdown +exit ${rc} + else +CREATED_BRO_CONTAINER=true + fi +fi + + +#run the bro container +#and optionally the passed script _IN_ the container +bash "${SCRIPT_DIR}"/run_bro_container.sh --container-path="${CONTAINER_DIR}" \ + --container-name=bro-docker-container:latest \ --- End diff -- done ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238850232 --- Diff: docker/scripts/cleanup_containers.sh --- @@ -0,0 +1,30 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +shopt -s nocasematch + +SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )" + +"${SCRIPT_DIR}"/stop_container.sh --container-name=bro --- End diff -- done ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238848200 --- Diff: docker/example_script.sh --- @@ -0,0 +1,177 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +shopt -s nocasematch + +CREATED_NETWORK_FLAG=false +RAN_ZK_CONTAINER=false +RAN_KAFKA_CONTAINER=false +CREATED_BRO_CONTAINER=false +RAN_BRO_CONTAINER=false + +SKIP_REBUILD_BRO=false +LEAVE_RUNNING=false + +SCRIPT_DIR=./scripts +CONTAINER_DIR=./containers/bro-localbuild-container +CONTAINER_NAME= +LOG_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && cd logs && pwd )" + +function help { + echo " " + echo "usage: ${0}" + echo "--skip-docker-build Skip build of bro docker machine." + echo "--leave-running Do not stop containers after script. The cleanup_containers.sh script should be run when done." + echo "-h/--help Usage information." + echo " " + echo " " +} + +function shutdown { + + if [[ "$RAN_BRO_CONTAINER" = true ]]; then +"${SCRIPT_DIR}"/stop_container.sh --container-name=bro --- End diff -- done ---
[GitHub] metron-bro-plugin-kafka issue #21: METRON-1911 [WIP] Docker setup for testin...
Github user ottobackwards commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/21 @JonZeolla , I was going to ask you about that, what do you think the best practice is for these permissions? ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238827202 --- Diff: docker/run_test.sh.template --- @@ -0,0 +1,199 @@ +#!/usr/bin/env bash --- End diff -- i changed it ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238826943 --- Diff: docker/scripts/process_data_docker.sh --- @@ -0,0 +1,29 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +shopt -s nocasematch + +echo "exectuting proces_data_dir.sh in the bro docker container" --- End diff -- fixed ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238806017 --- Diff: docker/scripts/download_sample_pcaps.sh --- @@ -0,0 +1,101 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +shopt -s nocasematch + +function help { + echo " " + echo "usage: ${0}" + echo "--data-paththe pcap data path" + echo "-h/--help Usage information." + echo " " + echo " " +} + +DATA_PATH= + +# handle command line options +for i in "$@"; do + case $i in + # + # DATA_PATH + # + # + --data-path=*) + DATA_PATH="${i#*=}" + shift # past argument=value + ;; + + # + # -h/--help + # + -h|--help) + help + exit 0 + shift # past argument with no value + ;; + + # + # Unknown option + # + *) + UNKNOWN_OPTION="${i#*=}" + echo "Error: unknown option: $UNKNOWN_OPTION" + help + ;; + esac +done + +if [[ -z "$DATA_PATH" ]]; then + echo "DATA_PATH must be passed" + exit 1 +fi + +echo "Running with " +echo "DATA_PATH = $DATA_PATH" +echo "===" + +for folder in nitroba example-traffic ssh ftp radius rfb; do + if [[ ! -d ${DATA_PATH}/${folder} ]]; then +mkdir -p ${DATA_PATH}/${folder} + fi +done + +if [[ ! -f ${DATA_PATH}/example-traffic/exercise-traffic.pcap ]]; then + wget https://www.bro.org/static/traces/exercise-traffic.pcap -O ${DATA_PATH}/example-traffic/exercise-traffic.pcap +fi +if [[ ! -f ${DATA_PATH}/nitroba/nitroba.pcap ]]; then + wget http://downloads.digitalcorpora.org/corpora/network-packet-dumps/2008-nitroba/nitroba.pcap -O ${DATA_PATH}/nitroba/nitroba.pcap +fi +if [[ ! -f ${DATA_PATH}/ssh/ssh.pcap ]]; then + wget https://www.bro.org/static/traces/ssh.pcap -O ${DATA_PATH}/ssh/ssh.pcap +fi +if [[ ! -f ${DATA_PATH}/ftp/ftp.pcap ]]; then + wget https://github.com/markofu/pcaps/blob/master/PracticalPacketAnalysis/ppa-capture-files/ftp.pcap?raw=true -O ${DATA_PATH}/ftp/ftp.pcap +fi +if [[ ! -f ${DATA_PATH}/radius/radius_localhost.pcapng ]]; then + wget https://github.com/EmpowerSecurityAcademy/wireshark/blob/master/radius_localhost.pcapng?raw=true -O ${DATA_PATH}/radius/radius_localhost.pcapng --- End diff -- as above I'm not sure which repo you mean. ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238805671 --- Diff: docker/scripts/download_sample_pcaps.sh --- @@ -0,0 +1,101 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +shopt -s nocasematch + +function help { + echo " " + echo "usage: ${0}" + echo "--data-paththe pcap data path" + echo "-h/--help Usage information." + echo " " + echo " " +} + +DATA_PATH= + +# handle command line options +for i in "$@"; do + case $i in + # + # DATA_PATH + # + # + --data-path=*) + DATA_PATH="${i#*=}" + shift # past argument=value + ;; + + # + # -h/--help + # + -h|--help) + help + exit 0 + shift # past argument with no value + ;; + + # + # Unknown option + # + *) + UNKNOWN_OPTION="${i#*=}" + echo "Error: unknown option: $UNKNOWN_OPTION" + help + ;; + esac +done + +if [[ -z "$DATA_PATH" ]]; then + echo "DATA_PATH must be passed" + exit 1 +fi + +echo "Running with " +echo "DATA_PATH = $DATA_PATH" +echo "===" + +for folder in nitroba example-traffic ssh ftp radius rfb; do + if [[ ! -d ${DATA_PATH}/${folder} ]]; then +mkdir -p ${DATA_PATH}/${folder} + fi +done + +if [[ ! -f ${DATA_PATH}/example-traffic/exercise-traffic.pcap ]]; then + wget https://www.bro.org/static/traces/exercise-traffic.pcap -O ${DATA_PATH}/example-traffic/exercise-traffic.pcap --- End diff -- what repo? ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238805529 --- Diff: docker/scripts/create_docker_network.sh --- @@ -0,0 +1,73 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +shopt -s nocasematch +function help { + echo " " + echo "usage: ${0}" + echo "--network-name [REQUIRED] The docker network name" + echo "-h/--help Usage information." + echo " " +} + +NETWORK_NAME= + +# handle command line options +for i in "$@"; do + case $i in + # + # NETWORK_NAME + # + # + # +--network-name=*) +NETWORK_NAME="${i#*=}" +shift # past argument=value + ;; + # + # -h/--help + # + -h|--help) + help + exit 0 + shift # past argument with no value + ;; + + # + # Unknown option + # + *) + UNKNOWN_OPTION="${i#*=}" + echo "Error: unknown option: $UNKNOWN_OPTION" + help + ;; + esac +done + +if [[ -z "$NETWORK_NAME" ]]; then + echo "NETWORK_NAME must be passed" + exit 1 +fi + +docker network create "${NETWORK_NAME}" +rc=$?; if [[ ${rc} != 0 ]]; then +exit ${rc}; --- End diff -- oops ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238805123 --- Diff: docker/in_docker_scripts/wait-for-it.sh --- @@ -0,0 +1,161 @@ +#!/usr/bin/env bash +# Use this script to test if a given TCP host/port are available + +cmdname=$(basename $0) --- End diff -- this works, and I don't want to change this script without great cause ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238804842 --- Diff: docker/in_docker_scripts/build_bro_plugin.sh --- @@ -0,0 +1,39 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +shopt -s nocasematch + +cd /root || exit 1 +echo "" >> "${RUN_LOG_PATH}" 2>&1 +bro-pkg install code --force | tee "${RUN_LOG_PATH}" +echo "" >> "${RUN_LOG_PATH}" 2>&1 + +echo "" >> "${RUN_LOG_PATH}" 2>&1 +bro -N Apache::Kafka | tee v --- End diff -- oops ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238804501 --- Diff: docker/in_docker_scripts/build_bro_plugin.sh --- @@ -0,0 +1,39 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +shopt -s nocasematch + +cd /root || exit 1 +echo "" >> "${RUN_LOG_PATH}" 2>&1 +bro-pkg install code --force | tee "${RUN_LOG_PATH}" --- End diff -- ok ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238803331 --- Diff: docker/run_test.sh.template --- @@ -0,0 +1,199 @@ +#!/usr/bin/env bash --- End diff -- so the idea is that this is a sample, that they would modify, and would not get committed over and over again ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238803083 --- Diff: docker/in_docker_scripts/wait-for-it.sh --- @@ -0,0 +1,161 @@ +#!/usr/bin/env bash +# Use this script to test if a given TCP host/port are available --- End diff -- all set ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238802060 --- Diff: docker/in_docker_scripts/wait-for-it.sh --- @@ -0,0 +1,161 @@ +#!/usr/bin/env bash +# Use this script to test if a given TCP host/port are available --- End diff -- it is from our metron-contrib/docker, he may have gotten it from there. I'll have to check the L&N ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238801412 --- Diff: docker/in_docker_scripts/process_data_dir.sh --- @@ -0,0 +1,32 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +shopt -s nocasematch + +cd /root || exit 1 +echo "" >> $RUN_LOG_PATH 2>&1 + +if [ ! -d /root/data ] ; then + echo "DATA_PATH has not been set and mapped" >> $RUN_LOG_PATH 2&>1 + exit 1 +fi + +ls /root/data --- End diff -- no debugging ---
[GitHub] metron-bro-plugin-kafka issue #21: METRON-1911 [WIP] Docker setup for testin...
Github user ottobackwards commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/21 We will also have scripts that do docker exec etc to read from kafka etc. ---
[GitHub] metron-bro-plugin-kafka issue #21: METRON-1911 [WIP] Docker setup for testin...
Github user ottobackwards commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/21 The idea, at least at this point is that when in the container you will have the 'built in scripts' that we have, and whatever scripts you have mapped into the container too. the run_test.sh.template currently doesn't pass through the script dir parameter, I have to build it out at that top level. So this will allow the user to write their own test scripts, maintained outside the code etc. ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238431951 --- Diff: docker/containers/bro-localbuild-container/Dockerfile --- @@ -0,0 +1,49 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +FROM centos:7 +WORKDIR /root + +# install development tools +RUN yum -y groupinstall "Development Tools" +RUN yum -y install cmake make gcc gcc-c++ flex bison libpcap libpcap-devel openssl-devel python-devel swig zlib-devel perl cyrus-sasl cyrus-sasl-devel cyrus-sasl-gssapi git + +# install bro +RUN curl -L https://www.bro.org/downloads/bro-2.5.5.tar.gz | tar xvz +WORKDIR bro-2.5.5/ +RUN ./configure +RUN make +RUN make install +ENV PATH="${PATH}:/usr/local/bro/bin" + +# install pip +RUN yum -y update +RUN yum -y install epel-release +RUN yum -y install python-pip +RUN pip install --upgrade pip + +# install bro-pkg +RUN pip install bro-pkg +ENV PATH="${PATH}:/usr/bin" +RUN bro-pkg autoconfig + +# install librdkafka +RUN curl -L https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz | tar xvz +WORKDIR librdkafka-0.9.4/ --- End diff -- all set ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238431923 --- Diff: docker/in_docker_scripts/build_bro.sh --- @@ -0,0 +1,31 @@ +#!/usr/bin/env bash --- End diff -- all set ---
[GitHub] metron-bro-plugin-kafka issue #21: METRON-1911 [WIP] Docker setup for testin...
Github user ottobackwards commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/21 I had a lot of trouble with compose, so I did it with plain run and a shared network. We can look at compose later. ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
GitHub user ottobackwards opened a pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21 METRON-1911 [WIP] Docker setup for testing bro plugin Status> To try: ``` î° ~/src/apache/forks/metron-bro-plugin-kafka î° î docker î° cd docker î° ~/src/apache/forks/metron-bro-plugin-kafka/docker î° î docker î° cp run_test.sh.template run_test.sh î° ~/src/apache/forks/metron-bro-plugin-kafka/docker î° î docker î° ./run_test.sh ``` This will: - create a network - start kafka and zookeeper containers - create bro container ( compile bro and kafka plugin ) - run bro container -> drop you in a shell then in the container `>./built_in_scripts/run_test.sh` Which will build the bro plugin and test it. ## todo - start the bro container in the background - add host side scripts to docker exec 'foo' where foo is a script in the bro container - execute the plugin build that way - configure bro for kafka - run script to bro -r some pcaps ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron's Bro kafka writer plugin. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [na] Have you ensured that the full suite of tests and checks have been executed via: ``` bro-pkg test $GITHUB_USERNAME/metron-bro-plugin-kafka --version $BRANCH ``` - [-] Have you written or updated unit tests and or integration tests to verify your changes? - [-] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [-] Have you verified the basic functionality of the build by building and running locally with Apache Metron's [Vagrant full-dev environment](https://github.com/apache/metron/tree/master/metron-deployment/development/centos6) or the equivalent? You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/metron-bro-plugin-kafka docker Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron-bro-plugin-kafka/pull/21.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #21 commit f7e464c6353da3b131c3c610ba154b435a24ee95 Author: Otto Fowler Date: 2018-12-03T20:23:13Z initial docker testing harness ---
[GitHub] metron issue #1288: METRON-1916 Stellar Classpath Function Resolver Should H...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1288 Can we step back? What dependencies where not present? That may be the real issue for the users. ---
[GitHub] metron issue #1288: METRON-1916 Stellar Classpath Function Resolver Should H...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1288 In my mind we don't have a current state where Stellar is running but not all the functions in the class path are loaded. This is different. Before we would have crashed starting up. Now we will run and crash later. ---
[GitHub] metron issue #1288: METRON-1916 Stellar Classpath Function Resolver Should H...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1288 @cestella should check in on this ---
[GitHub] metron issue #1288: METRON-1916 Stellar Classpath Function Resolver Should H...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1288 So, my main concern here is the state of stellar _after_ loading in this condition. What happens if you call a function that failed to load? What happens if it is nested in a match or a lambda? Right now, it blows up, but when loaded everything is known to be 'ok'. So this introduces a new state where we are loaded, but things that were supposed to load ( stellar functions introduced to the class path ) have failed. I'm not sure this is that simple. Would we _ever_ for example accept this state in the topology? ---
[GitHub] metron issue #1287: METORN-1915: Disable vagrant hostmanager in the developm...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1287 typo in PR title ---
[GitHub] metron issue #1287: METORN-1915: Disable vagrant hostmanager in the developm...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1287 what breaks specifically? having all the metron ports etc point to localhost seems to me that it would also break things or have conflicts etc. ---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1261 The base option is like quick dev, but quick dev also had metron installed, so starting it involved replacing the bits that where in there, we didn't maintain quick dev ansible w/o the metron stuff and 'other' ansible with the base. I think that is a good idea, but for a follow on to this.This effort itself is or may be just a first step to a couple of things. The base image would not replace this, but be the base image that this functionality uses. ---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1261#discussion_r236385492 --- Diff: metron-deployment/development/centos6_docker_build/build_and_run.sh --- @@ -0,0 +1,144 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +shopt -s nocasematch + +function help { + echo " " + echo "usage: ${0}" + echo "--skip-vagrant-up skip vagrant up" + echo "--force-docker-buildforce build docker machine" + echo "--skip-tags='tag,tag2,tag3' the ansible skip tags" + echo "-h/--help Usage information." + echo " " + echo "example: to skip vagrant up and force docker build with two tags" + echo " build_and_run.sh -skip-vagrant-up --force-docker-build --skip-tags='solr,sensors'" + echo " " +} + +SKIP_VAGRANT_UP=false +FORCE_DOCKER_BUILD=false +A_SKIP_TAGS="sensors,solr" + +# handle command line options +for i in "$@"; do + case $i in + # + # SKIP_VAGRANT_UP + # + # + --skip-vagrant-up) + SKIP_VAGRANT_UP=true + shift # past argument + ;; + + # + # FORCE_DOCKER_BUILD + # + # --force-docker-build + # + --force-docker-build) + FORCE_DOCKER_BUILD=true + shift # past argument + ;; + + # + # SKIP_TAGS + # + # --skip-tags='foo,bar' + # + --skip-tags=*) + A_SKIP_TAGS="${i#*=}" + shift # past argument=value + ;; + + # + # -h/--help + # + -h|--help) + help + exit 0 + shift # past argument with no value + ;; + + # + # Unknown option + # + *) + UNKNOWN_OPTION="${i#*=}" + echo "Error: unknown option: $UNKNOWN_OPTION" + help + ;; + esac +done + +echo "Running with " +echo "SKIP_VAGRANT_UP= $SKIP_VAGRANT_UP" +echo "FORCE_DOCKER_BUILD = $FORCE_DOCKER_BUILD" +echo "SKIP_TAGS = $A_SKIP_TAGS" +echo "===" + +if [[ "$SKIP_VAGRANT_UP" = false ]]; then + vagrant up + rc=$?; if [[ $rc != 0 ]]; then exit $rc; fi +fi + +VAGRANT_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )" +ANSIBLE_PATH=${VAGRANT_PATH}/ansible +VAGRANT_KEY_PATH=${VAGRANT_PATH}/.vagrant/machines/node1/virtualbox + +# move over to the docker area +cd ../docker || exit 1 + +# Give the option to not build the docker container, which can take some time and not be necessary +if [[ "$FORCE_DOCKER_BUILD" = true ]]; then + echo "docker build" + docker build -t metron-build-docker:latest . +fi + +if [[ ! -d ~/.m2 ]]; then + mkdir ~/.m2 +fi + +DATE=`date` +LOG_DATE=${DATE// /_} +LOGNAME="metron-build-${LOG_DATE}.log" + +# get the node1 ip address so we can add it to the docker hosts +NODE1_IP=$(awk '/^\s*hosts/{flag=1; next} /}]/{flag=0} flag' ${VAGRANT_PATH}/Vagrantfile | grep "^\s*ip:" | awk -F'"' '{print $2}') --- End diff -- Fixed ---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1261#discussion_r236375985 --- Diff: metron-deployment/development/centos6_docker_build/build_and_run.sh --- @@ -0,0 +1,144 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +shopt -s nocasematch + +function help { + echo " " + echo "usage: ${0}" + echo "--skip-vagrant-up skip vagrant up" + echo "--force-docker-buildforce build docker machine" + echo "--skip-tags='tag,tag2,tag3' the ansible skip tags" + echo "-h/--help Usage information." + echo " " + echo "example: to skip vagrant up and force docker build with two tags" + echo " build_and_run.sh -skip-vagrant-up --force-docker-build --skip-tags='solr,sensors'" + echo " " +} + +SKIP_VAGRANT_UP=false +FORCE_DOCKER_BUILD=false +A_SKIP_TAGS="sensors,solr" + +# handle command line options +for i in "$@"; do + case $i in + # + # SKIP_VAGRANT_UP + # + # + --skip-vagrant-up) + SKIP_VAGRANT_UP=true + shift # past argument + ;; + + # + # FORCE_DOCKER_BUILD + # + # --force-docker-build + # + --force-docker-build) + FORCE_DOCKER_BUILD=true + shift # past argument + ;; + + # + # SKIP_TAGS + # + # --skip-tags='foo,bar' + # + --skip-tags=*) + A_SKIP_TAGS="${i#*=}" + shift # past argument=value + ;; + + # + # -h/--help + # + -h|--help) + help + exit 0 + shift # past argument with no value + ;; + + # + # Unknown option + # + *) + UNKNOWN_OPTION="${i#*=}" + echo "Error: unknown option: $UNKNOWN_OPTION" + help + ;; + esac +done + +echo "Running with " +echo "SKIP_VAGRANT_UP= $SKIP_VAGRANT_UP" +echo "FORCE_DOCKER_BUILD = $FORCE_DOCKER_BUILD" +echo "SKIP_TAGS = $A_SKIP_TAGS" +echo "===" + +if [[ "$SKIP_VAGRANT_UP" = false ]]; then + vagrant up + rc=$?; if [[ $rc != 0 ]]; then exit $rc; fi +fi + +VAGRANT_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )" +ANSIBLE_PATH=${VAGRANT_PATH}/ansible +VAGRANT_KEY_PATH=${VAGRANT_PATH}/.vagrant/machines/node1/virtualbox + +# move over to the docker area +cd ../docker || exit 1 + +# Give the option to not build the docker container, which can take some time and not be necessary +if [[ "$FORCE_DOCKER_BUILD" = true ]]; then + echo "docker build" + docker build -t metron-build-docker:latest . +fi + +if [[ ! -d ~/.m2 ]]; then + mkdir ~/.m2 +fi + +DATE=`date` +LOG_DATE=${DATE// /_} +LOGNAME="metron-build-${LOG_DATE}.log" + +# get the node1 ip address so we can add it to the docker hosts +NODE1_IP=$(awk '/^\s*hosts/{flag=1; next} /}]/{flag=0} flag' ${VAGRANT_PATH}/Vagrantfile | grep "^\s*ip:" | awk -F'"' '{print $2}') --- End diff -- ahhh, yes, I did it for both. ---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1261 I think spellcheck is integrated into travis, so what we _can_ do is make shellcheck a build dependency and execute it through maven in the poms ---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1261 @mmiklavc I have it integrated into IntelliJ, though it isn't as obvious as java errors ---
[GitHub] metron issue #1249: METRON-1815: Separate metron-parsers into metron-parsers...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1249 Let's go with what @nickwallen said ---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1261#discussion_r236301587 --- Diff: metron-deployment/development/centos6_docker_build/build_and_run.sh --- @@ -0,0 +1,144 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +shopt -s nocasematch + +function help { + echo " " + echo "usage: ${0}" + echo "--skip-vagrant-up skip vagrant up" + echo "--force-docker-buildforce build docker machine" + echo "--skip-tags='tag,tag2,tag3' the ansible skip tags" + echo "-h/--help Usage information." + echo " " + echo "example: to skip vagrant up and force docker build with two tags" + echo " build_and_run.sh -skip-vagrant-up --force-docker-build --skip-tags='solr,sensors'" + echo " " +} + +SKIP_VAGRANT_UP=false +FORCE_DOCKER_BUILD=false +A_SKIP_TAGS="sensors,solr" + +# handle command line options +for i in "$@"; do + case $i in + # + # SKIP_VAGRANT_UP + # + # + --skip-vagrant-up) + SKIP_VAGRANT_UP=true + shift # past argument + ;; + + # + # FORCE_DOCKER_BUILD + # + # --force-docker-build + # + --force-docker-build) + FORCE_DOCKER_BUILD=true + shift # past argument + ;; + + # + # SKIP_TAGS + # + # --skip-tags='foo,bar' + # + --skip-tags=*) + A_SKIP_TAGS="${i#*=}" + shift # past argument=value + ;; + + # + # -h/--help + # + -h|--help) + help + exit 0 + shift # past argument with no value + ;; + + # + # Unknown option + # + *) + UNKNOWN_OPTION="${i#*=}" + echo "Error: unknown option: $UNKNOWN_OPTION" + help + ;; + esac +done + +echo "Running with " +echo "SKIP_VAGRANT_UP= $SKIP_VAGRANT_UP" +echo "FORCE_DOCKER_BUILD = $FORCE_DOCKER_BUILD" +echo "SKIP_TAGS = $A_SKIP_TAGS" +echo "===" + +if [[ "$SKIP_VAGRANT_UP" = false ]]; then + vagrant up + rc=$?; if [[ $rc != 0 ]]; then exit $rc; fi +fi + +VAGRANT_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )" +ANSIBLE_PATH=${VAGRANT_PATH}/ansible +VAGRANT_KEY_PATH=${VAGRANT_PATH}/.vagrant/machines/node1/virtualbox + +# move over to the docker area +cd ../docker || exit 1 + +# Give the option to not build the docker container, which can take some time and not be necessary +if [[ "$FORCE_DOCKER_BUILD" = true ]]; then + echo "docker build" + docker build -t metron-build-docker:latest . +fi + +if [[ ! -d ~/.m2 ]]; then + mkdir ~/.m2 +fi + +DATE=`date` +LOG_DATE=${DATE// /_} +LOGNAME="metron-build-${LOG_DATE}.log" + +# get the node1 ip address so we can add it to the docker hosts +NODE1_IP=$(awk '/^\s*hosts/{flag=1; next} /}]/{flag=0} flag' ${VAGRANT_PATH}/Vagrantfile | grep "^\s*ip:" | awk -F'"' '{print $2}') --- End diff -- this change breaks the statement. I disabled the warning. ---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1261#discussion_r236292888 --- Diff: metron-deployment/development/centos6_docker_build/docker_run_ansible.sh --- @@ -0,0 +1,46 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + + +# +# This script runs IN the docker container +# + +cd /root/metron || exit 1 + +# make sure we have the right c++ tools +source /opt/rh/devtoolset-6/enable + +# give the option to skip building metron, in case they have already done so +#read -p " build metron? [yN] " -n 1 -r +#echo +#if [[ $REPLY =~ ^[Yy]$ ]]; then +# USE TAGS +#fi --- End diff -- yes ---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1261#discussion_r236262632 --- Diff: metron-deployment/development/centos6_docker_build/run_docker_only.sh --- @@ -0,0 +1,39 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +VAGRANT_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )" +echo "setting the ansible configuration path" +ANSIBLE_PATH=${VAGRANT_PATH}/ansible +echo ${ANSIBLE_PATH} +echo "setting the ssh key" +VAGRANT_KEY_PATH=`pwd`/.vagrant/machines/node1/virtualbox +echo ${VAGRANT_KEY_PATH} + +# move over to the docker area +cd ../docker || exit 1 +pwd --- End diff -- oh, the pwd I guess not, this is a debugging leftover. I'll remove ---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1261#discussion_r236262449 --- Diff: metron-deployment/development/centos6_docker_build/run_docker_only.sh --- @@ -0,0 +1,39 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +VAGRANT_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )" +echo "setting the ansible configuration path" +ANSIBLE_PATH=${VAGRANT_PATH}/ansible +echo ${ANSIBLE_PATH} +echo "setting the ssh key" +VAGRANT_KEY_PATH=`pwd`/.vagrant/machines/node1/virtualbox +echo ${VAGRANT_KEY_PATH} + +# move over to the docker area +cd ../docker || exit 1 +pwd --- End diff -- Not sure what you mean ---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1261#discussion_r236262354 --- Diff: metron-deployment/development/centos6_docker_build/ansible/inventory/group_vars/all --- @@ -0,0 +1,86 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# only need to build the RPMs for CentOS +metron_build_packages_cmd: "shell cd {{ metron_build_dir }}/metron-deployment && mvn clean package -DskipTests -Pbuild-rpms" --- End diff -- yes ---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1261#discussion_r236262118 --- Diff: metron-deployment/development/centos6_docker_build/build_and_run.sh --- @@ -0,0 +1,144 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +shopt -s nocasematch + +function help { + echo " " + echo "usage: ${0}" + echo "--skip-vagrant-up skip vagrant up" + echo "--force-docker-buildforce build docker machine" + echo "--skip-tags='tag,tag2,tag3' the ansible skip tags" + echo "-h/--help Usage information." + echo " " + echo "example: to skip vagrant up and force docker build with two tags" + echo " build_and_run.sh -skip-vagrant-up --force-docker-build --skip-tags='solr,sensors'" --- End diff -- ok ---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1261#discussion_r236261928 --- Diff: metron-deployment/development/centos6_docker_build/docker_run_ansible.sh --- @@ -0,0 +1,46 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + + +# +# This script runs IN the docker container +# + +cd /root/metron || exit 1 + +# make sure we have the right c++ tools +source /opt/rh/devtoolset-6/enable --- End diff -- ok ---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1261#discussion_r236261518 --- Diff: metron-deployment/development/centos6_docker_build/build_and_run.sh --- @@ -0,0 +1,144 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +shopt -s nocasematch + +function help { + echo " " + echo "usage: ${0}" + echo "--skip-vagrant-up skip vagrant up" + echo "--force-docker-buildforce build docker machine" + echo "--skip-tags='tag,tag2,tag3' the ansible skip tags" + echo "-h/--help Usage information." + echo " " + echo "example: to skip vagrant up and force docker build with two tags" + echo " build_and_run.sh -skip-vagrant-up --force-docker-build --skip-tags='solr,sensors'" + echo " " +} + +SKIP_VAGRANT_UP=false +FORCE_DOCKER_BUILD=false +A_SKIP_TAGS="sensors,solr" + +# handle command line options +for i in "$@"; do + case $i in + # + # SKIP_VAGRANT_UP + # + # + --skip-vagrant-up) + SKIP_VAGRANT_UP=true + shift # past argument + ;; + + # + # FORCE_DOCKER_BUILD + # + # --force-docker-build + # + --force-docker-build) + FORCE_DOCKER_BUILD=true + shift # past argument + ;; + + # + # SKIP_TAGS + # + # --skip-tags='foo,bar' + # + --skip-tags=*) + A_SKIP_TAGS="${i#*=}" + shift # past argument=value + ;; + + # + # -h/--help + # + -h|--help) + help + exit 0 + shift # past argument with no value + ;; + + # + # Unknown option + # + *) + UNKNOWN_OPTION="${i#*=}" + echo "Error: unknown option: $UNKNOWN_OPTION" + help + ;; + esac +done + +echo "Running with " +echo "SKIP_VAGRANT_UP= $SKIP_VAGRANT_UP" +echo "FORCE_DOCKER_BUILD = $FORCE_DOCKER_BUILD" +echo "SKIP_TAGS = $A_SKIP_TAGS" +echo "===" + +if [[ "$SKIP_VAGRANT_UP" = false ]]; then + vagrant up + rc=$?; if [[ $rc != 0 ]]; then exit $rc; fi +fi + +VAGRANT_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )" +ANSIBLE_PATH=${VAGRANT_PATH}/ansible +VAGRANT_KEY_PATH=${VAGRANT_PATH}/.vagrant/machines/node1/virtualbox + +# move over to the docker area +cd ../docker || exit 1 + +# Give the option to not build the docker container, which can take some time and not be necessary +if [[ "$FORCE_DOCKER_BUILD" = true ]]; then + echo "docker build" + docker build -t metron-build-docker:latest . +fi + +if [[ ! -d ~/.m2 ]]; then + mkdir ~/.m2 +fi + +DATE=`date` --- End diff -- ok ---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1261#discussion_r236261064 --- Diff: metron-deployment/development/centos6_docker_build/build_and_run.sh --- @@ -0,0 +1,144 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +shopt -s nocasematch + +function help { + echo " " + echo "usage: ${0}" + echo "--skip-vagrant-up skip vagrant up" + echo "--force-docker-buildforce build docker machine" + echo "--skip-tags='tag,tag2,tag3' the ansible skip tags" + echo "-h/--help Usage information." + echo " " + echo "example: to skip vagrant up and force docker build with two tags" + echo " build_and_run.sh -skip-vagrant-up --force-docker-build --skip-tags='solr,sensors'" + echo " " +} + +SKIP_VAGRANT_UP=false +FORCE_DOCKER_BUILD=false +A_SKIP_TAGS="sensors,solr" + +# handle command line options +for i in "$@"; do + case $i in + # + # SKIP_VAGRANT_UP + # + # + --skip-vagrant-up) + SKIP_VAGRANT_UP=true + shift # past argument + ;; + + # + # FORCE_DOCKER_BUILD + # + # --force-docker-build + # + --force-docker-build) + FORCE_DOCKER_BUILD=true + shift # past argument + ;; + + # + # SKIP_TAGS + # + # --skip-tags='foo,bar' + # + --skip-tags=*) + A_SKIP_TAGS="${i#*=}" + shift # past argument=value + ;; + + # + # -h/--help + # + -h|--help) + help + exit 0 + shift # past argument with no value + ;; + + # + # Unknown option + # + *) + UNKNOWN_OPTION="${i#*=}" + echo "Error: unknown option: $UNKNOWN_OPTION" + help + ;; + esac +done + +echo "Running with " +echo "SKIP_VAGRANT_UP= $SKIP_VAGRANT_UP" +echo "FORCE_DOCKER_BUILD = $FORCE_DOCKER_BUILD" +echo "SKIP_TAGS = $A_SKIP_TAGS" +echo "===" + +if [[ "$SKIP_VAGRANT_UP" = false ]]; then + vagrant up + rc=$?; if [[ $rc != 0 ]]; then exit $rc; fi +fi + +VAGRANT_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )" +ANSIBLE_PATH=${VAGRANT_PATH}/ansible +VAGRANT_KEY_PATH=${VAGRANT_PATH}/.vagrant/machines/node1/virtualbox + +# move over to the docker area +cd ../docker || exit 1 + +# Give the option to not build the docker container, which can take some time and not be necessary +if [[ "$FORCE_DOCKER_BUILD" = true ]]; then + echo "docker build" + docker build -t metron-build-docker:latest . +fi + +if [[ ! -d ~/.m2 ]]; then + mkdir ~/.m2 +fi + +DATE=`date` +LOG_DATE=${DATE// /_} +LOGNAME="metron-build-${LOG_DATE}.log" + +# get the node1 ip address so we can add it to the docker hosts +NODE1_IP=$(awk '/^\s*hosts/{flag=1; next} /}]/{flag=0} flag' ${VAGRANT_PATH}/Vagrantfile | grep "^\s*ip:" | awk -F'"' '{print $2}') --- End diff -- ok ---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1261#discussion_r236260538 --- Diff: metron-deployment/development/centos6_docker_build/build_and_run.sh --- @@ -0,0 +1,144 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +shopt -s nocasematch + +function help { + echo " " + echo "usage: ${0}" + echo "--skip-vagrant-up skip vagrant up" + echo "--force-docker-buildforce build docker machine" + echo "--skip-tags='tag,tag2,tag3' the ansible skip tags" + echo "-h/--help Usage information." + echo " " + echo "example: to skip vagrant up and force docker build with two tags" + echo " build_and_run.sh -skip-vagrant-up --force-docker-build --skip-tags='solr,sensors'" + echo " " +} + +SKIP_VAGRANT_UP=false +FORCE_DOCKER_BUILD=false +A_SKIP_TAGS="sensors,solr" + +# handle command line options +for i in "$@"; do + case $i in + # + # SKIP_VAGRANT_UP + # + # + --skip-vagrant-up) + SKIP_VAGRANT_UP=true + shift # past argument + ;; + + # + # FORCE_DOCKER_BUILD + # + # --force-docker-build + # + --force-docker-build) + FORCE_DOCKER_BUILD=true + shift # past argument + ;; + + # + # SKIP_TAGS + # + # --skip-tags='foo,bar' + # + --skip-tags=*) + A_SKIP_TAGS="${i#*=}" + shift # past argument=value + ;; + + # + # -h/--help + # + -h|--help) + help + exit 0 + shift # past argument with no value + ;; + + # + # Unknown option + # + *) + UNKNOWN_OPTION="${i#*=}" --- End diff -- To make sure we consume the option and value ---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1261#discussion_r236260621 --- Diff: metron-deployment/development/centos6_docker_build/build_and_run.sh --- @@ -0,0 +1,144 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +shopt -s nocasematch + +function help { + echo " " + echo "usage: ${0}" + echo "--skip-vagrant-up skip vagrant up" + echo "--force-docker-buildforce build docker machine" + echo "--skip-tags='tag,tag2,tag3' the ansible skip tags" + echo "-h/--help Usage information." + echo " " + echo "example: to skip vagrant up and force docker build with two tags" + echo " build_and_run.sh -skip-vagrant-up --force-docker-build --skip-tags='solr,sensors'" + echo " " +} + +SKIP_VAGRANT_UP=false +FORCE_DOCKER_BUILD=false +A_SKIP_TAGS="sensors,solr" + +# handle command line options +for i in "$@"; do + case $i in + # + # SKIP_VAGRANT_UP + # + # + --skip-vagrant-up) + SKIP_VAGRANT_UP=true + shift # past argument + ;; + + # + # FORCE_DOCKER_BUILD + # + # --force-docker-build + # + --force-docker-build) + FORCE_DOCKER_BUILD=true + shift # past argument + ;; + + # + # SKIP_TAGS + # + # --skip-tags='foo,bar' + # + --skip-tags=*) + A_SKIP_TAGS="${i#*=}" + shift # past argument=value + ;; + + # + # -h/--help + # + -h|--help) + help + exit 0 + shift # past argument with no value + ;; + + # + # Unknown option + # + *) + UNKNOWN_OPTION="${i#*=}" --- End diff -- This is kind of boilerplate ---
[GitHub] metron pull request #1279: METRON-1893 Syslog RFC-3164 parser
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1279#discussion_r236062925 --- Diff: metron-platform/metron-parsers/src/main/resources/META-INF/NOTICE --- @@ -37,4 +37,18 @@ Copyright 2006-2011 Google, Inc. Apache Software Foundation that were originally developed at iClick, Inc., software copyright (c) 1999. + (ASLv2) simple-syslog-5424 +The following NOTICE information applies: +simple-syslog-5424 +https://github.com/palindromicity/simple-syslog-5424 + +Copyright 2018 simple-syslog-5424 authors. + + (ASLv2) simple-syslog-5424 --- End diff -- nice catch, fixed. Thanks! ---
[GitHub] metron pull request #1279: METRON-1893 Syslog RFC-3164 parser
GitHub user ottobackwards opened a pull request: https://github.com/apache/metron/pull/1279 METRON-1893 Syslog RFC-3164 parser This PR adds a new parser for Syslog RFC-3164 This PR also moves from simple-syslog-5424 to simple-syslog, which has both RFC parsers in it. So the Syslog5424Parser is updated as well. The parsers are refactored to a common base since they share interfaces. ## Testing, throw some 3164 logs at it. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [-] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [-] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [-] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [-] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/metron syslog-3164 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1279.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1279 commit 568134979b348cc685e0a3f8fc66e66456dd08ce Author: Otto Fowler Date: 2018-11-21T14:49:43Z need to refactor, will not work with shading commit 02a856cc8eb75daf603c9be71e640681c9605ecc Author: Otto Fowler Date: 2018-11-24T17:11:05Z syslog 3164 parser ---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1261 @nickwallen Done ---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1261 @nickwallen, yeah, I did prompts as I went along debugging. I was thinking that folks may not like them. I'll parameterize things. ---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1261 If you create an issue for your vagrant base machine with our hadoop / ambari already in it, you can assign it to me. @justinleet ---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1261 It is possible to imagine a number of scenarios, including that, but also needing to build with new hadoop versions ( can't loose build from scratch ). There are a number of things we can do down the road. I think this work is going to help people enough in the near term to land it, while we discuss longer term refactoring and workflow. ---
[GitHub] metron pull request #1277: METRON-1890 disable audio in vm so it does not gr...
GitHub user ottobackwards opened a pull request: https://github.com/apache/metron/pull/1277 METRON-1890 disable audio in vm so it does not grab the mic and trigger security ⦠â¦software like microsnitch Run the vagrant ups, should be normal, but you won't get audio alert from [micro snitch](https://obdev.at/products/microsnitch/index.html) or [Oversight](https://objective-see.com/products/oversight.html). ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [-] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [-] Have you written or updated unit tests and or integration tests to verify your changes? - [-] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [-] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [-] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/metron disable-vm-audio Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1277.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1277 commit ae8af082b9c394e81f5d2c3c1956e4ec76fb38dc Author: Otto Fowler Date: 2018-11-20T21:19:38Z disable audio in vm so it does not grab the mic and trigger security software like microsnitch ---
[GitHub] metron issue #1249: METRON-1815: Separate metron-parsers into metron-parsers...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1249 wait, you don't have a plan??? lol. nm. just wondering. I have some old things around this, maybe I'll follow on. ---
[GitHub] metron issue #1249: METRON-1815: Separate metron-parsers into metron-parsers...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1249 I was going to suggest that we may want do it, but your use of _conflating_ has dissuaded me. Have you a list of 'post' things to add that to perhaps? ---
[GitHub] metron issue #1249: METRON-1815: Separate metron-parsers into metron-parsers...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1249 All the hard coded paths for test resources should be removed. Similar to how we build the test data paths for the integration tests. ---
[GitHub] metron-bro-plugin-kafka issue #18: Remove version from show-plugin btest out...
Github user ottobackwards commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/18 Thanks for the taking the time you did to submit the PR @dopheide-esnet. -1 Filling out the template to say "Yes reviewer, I tested, built, and check the lic etc " isn't a lot to ask and it helps the community, the reviewers a great deal. Having the courtesy to follow a project/communities policies when submitting is not a strange thing to expect. You can close this. Thanks again. ---
[GitHub] metron issue #1271: Updated org.apache.rat from 0.13-SNAPSHOT to 0.13
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1271 Hi, thanks for the contribution! Can you have a look at filling out the pr template? This this it is especially important to look at the part about saying you have tried: `mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ` Thanks! ---
[GitHub] metron issue #1268: METRON-1877: Nested IF ELSE statements can cause parse e...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1268 +1, great job ---
[GitHub] metron pull request #1268: METRON-1877: Nested IF ELSE statements can cause ...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1268#discussion_r234305980 --- Diff: metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/BasicStellarTest.java --- @@ -916,6 +916,53 @@ public void testShortCircuit_conditional() throws Exception { Assert.assertEquals("foo", run("if NOT('metron' in [ 'metron', 'metronicus'] ) then THROW('expression') else 'foo'", new HashMap<>())); } --- End diff -- yeah, that is the idea. I had to add more and more permutations ( I added tests for issues as @jjmeyer0 found them, or as @cestella found them ) If you have one mega case that works that is fine. I started out differently, because I had things that did not work at the beginning and I added later ( MAP support etc ) ---
[GitHub] metron pull request #1268: METRON-1877: Nested IF ELSE statements can cause ...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1268#discussion_r234293803 --- Diff: metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/BasicStellarTest.java --- @@ -916,6 +916,53 @@ public void testShortCircuit_conditional() throws Exception { Assert.assertEquals("foo", run("if NOT('metron' in [ 'metron', 'metronicus'] ) then THROW('expression') else 'foo'", new HashMap<>())); } --- End diff -- if you look at https://github.com/apache/metron/pull/814, which was when I did checkpoint work for match, I had a lot of fallout to account for. Maybe you can take a peak at the match tests. ---
[GitHub] metron pull request #1268: METRON-1877: Nested IF ELSE statements can cause ...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1268#discussion_r234290697 --- Diff: metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/BasicStellarTest.java --- @@ -916,6 +916,53 @@ public void testShortCircuit_conditional() throws Exception { Assert.assertEquals("foo", run("if NOT('metron' in [ 'metron', 'metronicus'] ) then THROW('expression') else 'foo'", new HashMap<>())); } --- End diff -- These tests are good, but these evaluations should be tested with other things in the things in the if s IF SOME_MORE_COMPLEX_FUNCTION THEN IF LAMBDA THEN MATCH ELSE B ELSE C To make sure we are not messing the stack up. ---
[GitHub] metron pull request #1268: METRON-1877: Nested IF ELSE statements can cause ...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1268#discussion_r234289745 --- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/StellarCompiler.java --- @@ -219,14 +220,23 @@ public Object apply(ExpressionState state) { //short circuit the if/then/else instanceDeque.pop(); if((Boolean)curr.getValue()) { -//choose then -skipElse = true; +//choose then. Need to make sure we're keeping track of nesting. +skipElseCount++; } else { //choose else +// Need to count in case we see another if-else, to avoid breaking on wrong else. +int innerIfCount = 0; while (it.hasNext()) { Token t = it.next(); + if (t.getUnderlyingType() == IfExpr.class) { +innerIfCount++; + } --- End diff -- wouldn't if, else if be more clear here? ---
[GitHub] metron issue #1265: METRON-1874 Create a Parser Debugger
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1265 Then maybe a function to evaluate stellar EVALUATE_TRANSFORM(config,PARSER_PARSE(parser, input) ---
[GitHub] metron issue #1265: METRON-1874 Create a Parser Debugger
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1265 maybe that can be a follow on, it would be much better to load from disk -> split lines than to open an editor and cut and past from your sample log. ---
[GitHub] metron pull request #1249: METRON-1815: Separate metron-parsers into metron-...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1249#discussion_r234230655 --- Diff: README.md --- @@ -132,7 +132,7 @@ component and Apache Kafka as the unified data bus. Some high level links to the relevant subparts of the architecture, for more information: --- End diff -- Fair enough, it just stuck out to me. Kafka is a part of the storm support, not the parsing ( we could have a udp bolt or something, or a nifi bolt ) with where we are going. ---
[GitHub] metron issue #1265: METRON-1874 Create a Parser Debugger
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1265 Can we test parser chains? ---
[GitHub] metron issue #1265: METRON-1874 Create a Parser Debugger
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1265 Can we load files from disk? It would be nice to not have had to setup kafka etc. ---
[GitHub] metron pull request #1249: METRON-1815: Separate metron-parsers into metron-...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1249#discussion_r233920015 --- Diff: README.md --- @@ -132,7 +132,7 @@ component and Apache Kafka as the unified data bus. Some high level links to the relevant subparts of the architecture, for more information: --- End diff -- Going forward in this effort, as it is more compassable, it may _not_ be kafka that is providing the data, should we explicitly name kafka here? ---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1261 anyone have any ideas of the best way to time these things? ---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1261 We could also use more tags, for example I may want to skip building the java, but not skip building the RPMs. Think of a dev flow -> I make my change, run my local tests and want to spin up full dev. It is already built, but needs the rpms, I should be able to make ansible skip the compile/package of java and still do the rpms/debs ---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1261 @nickwallen That is an option, but not something I would pick as the goal from the outset if you know what I mean. ---
[GitHub] metron issue #1249: METRON-1815: Separate metron-parsers into metron-parsers...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1249 @justinleet I'm fine with that ---
[GitHub] metron issue #1260: METRON-1868: metron-committer-common incorrectly checkin...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1260 +1, sorry I missed this ---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1261 The integration test failure has to do with the profiler tests and seem unrelated. ---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1261 @mmiklavc That is basically correct. Except that the ansible version is 2.5, since it only applies to this build, and allows for the yaml log formatting. Also, in the latest version, the ansible once again does the clean and build as opposed to the script. I had a lot of problems getting the c++ picked up from ansible and moved the build out of it for the time, but the idea was always to have ansible run the metron_build, and that has returned. The reasoning for the prompts to build the vagrant box and the docker -> - if you are using this during development, IE> we are working ON ansible or ON docker, you may fail in the docker or ansible stage without modifying the vm, and thus not need to vagrant up again. Likewise, you may not need to rebuild the docker machine if you have not made changes, _or_ you may in fact need to. I added these flags as I developed. ---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1261 @nickwallen I did not think of that. I was improving the process that stands today. I think in a world where the posted image exists, we would still want the ability to try the latest ( to verify a fix pre-release etc ). ---
[GitHub] metron issue #1260: METRON-1868: metron-committer-common incorrectly checkin...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1260 What did you seen before that is fix now? ---
[GitHub] metron issue #1249: METRON-1815: Separate metron-parsers into metron-parsers...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1249 I'm ok with changing parsing to something else, just not parsers. ---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
GitHub user ottobackwards opened a pull request: https://github.com/apache/metron/pull/1261 METRON-1860 [WIP] new developer option for ansible in docker to deploy to vagrant The goal of this PR is to provide a new "full_dev" option for new and old users that does not require as much setup and version matching to try Metron's full dev environment. Currently, the vagrant up command runs ansible locally, on the host machine, to build and deploy metron. This means that the user must not only have Vagrant, Virtual Box and Docker, but must also have all the tools necessary to build metron ( maven, java, c++ 11 etc ) and run ansible ( python and others ). It has been a common source of problems for new users to get started with Metron because of version or setup problems. This PR introduces a new metron-deployment/development option which tries to address this problem, and make it possible for the user to only have Vagrant, VirtualBox and Docker ( along with a local copy of the source ) to be able to run full dev. The new option starts the Vagrant VM, but does not run ansible in it. Instead it runs a docker container which contains all the tools/versions necessary, and that container is what runs ansible. This is a WIP, I am looking for initial feedback. ##Testing Have the correct versions of vagrant, virtual box and docker installed and running ``` cd $METRON_SRC_ROOT/metron-deployment/development/centos6_docker_build ./build_and_run.sh ``` Answer yes to building the vagrant box. Answer yes to building the docker machine Go grab a coffee. The end result should be full dev running in the vagrant instance. ### Differences - This does not support skip tags passed on the cli - This does not support provision ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [-] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [-] Have you written or updated unit tests and or integration tests to verify your changes? - [-] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [-] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/metron vagrant-docker Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1261.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1261 commit 1ac44c0d78b25113a5ebe807baccf160ef9af139 Author: Otto Fowler Date: 2018-11-09T03:51:56Z Initial commit - This provides an environment where only vagrant and docker ( and a copy of the metron codebase ) are required to run the Metron full-dev vm with it's default setup. This is the initial work, there will be refactorings commit f54cc493f8e045d5f0ca669258f56a5576b92ad1 Author: Otto Fowler Date: 2018-11-13T17:32:38Z refactored locations ---
[GitHub] metron issue #1171: METRON-1740 make parser support CONFIG and SYSTEM log ty...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1171 @nickwallen can we get a +1 on this? @JonZeolla you are a +1 but not explicitly? ---
[GitHub] metron pull request #1014: METRON-1563 : Base Stellar assign for feature bra...
Github user ottobackwards closed the pull request at: https://github.com/apache/metron/pull/1014 ---
[GitHub] metron issue #1014: METRON-1563 : Base Stellar assign for feature branch
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1014 This is in, not sure why it isn't closed ---
[GitHub] metron issue #1258: METRON-1864 fix issue where daylight savings breaks test...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1258 I think that the test logically tests what we are concerned about, that the format of the produced string matches the desired format. Testing that the value is explicitly x,y,z isn't testing the format, it is testing something else. I propose, with a mind to getting master and pr's building again, that I comment as requested but not change the test, and that any other ideas for improvement and or expansion of the tests be put to a jira for later work. ---
[GitHub] metron pull request #1258: METRON-1864 fix issue where daylight savings brea...
GitHub user ottobackwards opened a pull request: https://github.com/apache/metron/pull/1258 METRON-1864 fix issue where daylight savings breaks tests Thanks to @justinleet for the fix. I would have use a totally awesome but overdone regex. ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [-] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [-] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [-] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [-] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [-] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/metron stellar-datefunction-tz-fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1258.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1258 commit b89068d2a52e5e6ceb7c925d6ca3e14ee8fc15f6 Author: Otto Fowler Date: 2018-11-08T16:12:40Z fix issue where daylight savings breaks tests ---