[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on pull request #40: METRON-2309 Add a Kafka "metadata.broker.list" for each log writer filter.
JonZeolla commented on pull request #40: URL: https://github.com/apache/metron-bro-plugin-kafka/pull/40#issuecomment-621417174 Hi @mauropalumbo75 thanks again for the contribution. Can you rebase this on master and see if the e2e tests pass? It should include the `btest`s within Docker to take your local environment out of the equation. After the rebase you should be able to `cd Docker/; ./run_end_to_end.sh` and let it churn for a while to see if it works. It won't test multiple brokers, but that's something we are working towards with the move to `docker-compose` for e2e tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on pull request #43: METRON-2349: [BRO-PLUGIN-KAFKA] Fix hard coded topic_name in e2e tests
JonZeolla commented on pull request #43: URL: https://github.com/apache/metron-bro-plugin-kafka/pull/43#issuecomment-621396855 Force pushed to rebase on master without merge conflicts. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (METRON-2349) [BRO-PLUGIN-KAFKA] Fix hard coded topic_name in e2e tests
[ https://issues.apache.org/jira/browse/METRON-2349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17095791#comment-17095791 ] Jon Zeolla commented on METRON-2349: https://github.com/apache/metron-bro-plugin-kafka/pull/43 > [BRO-PLUGIN-KAFKA] Fix hard coded topic_name in e2e tests > - > > Key: METRON-2349 > URL: https://issues.apache.org/jira/browse/METRON-2349 > Project: Metron > Issue Type: Bug >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > https://github.com/apache/metron-bro-plugin-kafka/blob/1772b1fabb50e0710eafe93c14ec2d678c0a8bb9/docker/in_docker_scripts/configure_bro_plugin.sh#L32 > should not be hard coded to use a topic of bro in order to allow a custom > kafka topic like was meant to be accomplished in > https://github.com/apache/metron-bro-plugin-kafka/pull/27 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [metron] JonZeolla commented on pull request #1580: METRON-2348 fix prepare-commit
JonZeolla commented on pull request #1580: URL: https://github.com/apache/metron/pull/1580#issuecomment-621391223 FYI tested this to merge apache/metron-bro-plugin-kafka#42 successfully. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [metron-bro-plugin-kafka] JonZeolla opened a new pull request #43: METRON-2349: [BRO-PLUGIN-KAFKA] Fix hard coded topic_name in e2e tests
JonZeolla opened a new pull request #43: URL: https://github.com/apache/metron-bro-plugin-kafka/pull/43 ## Contributor Comments Fixes e2e tests when a custom kafka topic is specified. Previously everything would have completed without error, but the bro container would have sent to a hard coded `bro` topic. This is based off of #42 for the time being, until that is merged and I will rebase on `master`. ### Testing Run `./run_end_to_end.sh --kafka-topic=testing` and see that it works ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron's Bro kafka writer plugin. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [X] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [X] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [X] Have you ensured that the full suite of tests and checks have been executed via: ``` bro-pkg test $GITHUB_USERNAME/metron-bro-plugin-kafka --version $BRANCH ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Apache Metron's [Vagrant full-dev environment](https://github.com/apache/metron/tree/master/metron-deployment/development/centos6) or the equivalent? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [metron-bro-plugin-kafka] ottobackwards commented on pull request #42: METRON-2347: [BRO-PLUGIN-KAFKA] Use docker compose in end to end tests
ottobackwards commented on pull request #42: URL: https://github.com/apache/metron-bro-plugin-kafka/pull/42#issuecomment-621382174 +1 nice work This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on pull request #42: METRON-2347: [BRO-PLUGIN-KAFKA] Use docker compose in end to end tests
JonZeolla commented on pull request #42: URL: https://github.com/apache/metron-bro-plugin-kafka/pull/42#issuecomment-621380805 Ok, back to you @ottobackwards should have addressed all of your feedback now This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (METRON-2349) [BRO-PLUGIN-KAFKA] Fix hard coded topic_name in e2e tests
Jon Zeolla created METRON-2349: -- Summary: [BRO-PLUGIN-KAFKA] Fix hard coded topic_name in e2e tests Key: METRON-2349 URL: https://issues.apache.org/jira/browse/METRON-2349 Project: Metron Issue Type: Bug Reporter: Jon Zeolla Assignee: Jon Zeolla https://github.com/apache/metron-bro-plugin-kafka/blob/1772b1fabb50e0710eafe93c14ec2d678c0a8bb9/docker/in_docker_scripts/configure_bro_plugin.sh#L32 should not be hard coded to use a topic of bro in order to allow a custom kafka topic like was meant to be accomplished in https://github.com/apache/metron-bro-plugin-kafka/pull/27 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (METRON-2339) Can not match backslash in REGEXP_MATCH Stellar REPL
[
https://issues.apache.org/jira/browse/METRON-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17095530#comment-17095530
]
Otto Fowler commented on METRON-2339:
-
If you can confirm that '\\\' works for you then this is a dupe of METRON-2002,
it looks like you did 1 and 3.
> Can not match backslash in REGEXP_MATCH Stellar REPL
>
>
> Key: METRON-2339
> URL: https://issues.apache.org/jira/browse/METRON-2339
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.7.1
> Environment: centos 7
>Reporter: Dima Kovalyov
>Priority: Major
> Labels: stellar
>
> If I have a message with a backslash like C:\Windows
> How can I use REGEXP_MATCH to Profile this message?
> Here is where it does not work for me. I have a message:
> {code:java}
> {
> "line": "test"
> } {code}
> To keep it simple this is just text with no special characters.
> Using Stellar REPL I create the following profile:
> {code:java}
> {
> "profiles": [{
> "profile": "match-backslash",
> "foreach": "line",
> "onlyif": "REGEXP_MATCH(line, '\\')",
> "init": { "count": "0" },
> "update": { "count": "count + 1" },
> "result": "total"
> }
> ]
> } {code}
> Here is what happens when I apply any message:
> {code:java}
> [Stellar]>>> PROFILER_APPLY(msg, profile)
> 2020-01-09 18:24:26 ERROR DefaultMessageRouter:123 - error while executing
> profile; profile='match-backslash', error='Unable to parse:
> REGEXP_MATCH(line, '\') due to: Syntax error @ 1:26 no viable alternative at
> input 'REGEXP_MATCH(line,'''
> org.apache.metron.stellar.dsl.ParseException: Unable to parse:
> REGEXP_MATCH(line, '\') due to: Syntax error @ 1:26 no viable alternative at
> input 'REGEXP_MATCH(line,'' {code}
> If I escape more:
> {code:java}
> {
> "profiles": [{
> "profile": "match-backslash",
> "foreach": "line",
> "onlyif": "REGEXP_MATCH(line, '')",
> "init": { "count": "0" },
> "update": { "count": "count + 1" },
> "result": "total"
> }
> ]
> } {code}
> I'll get this:
> {code:java}
> [Stellar]>>> PROFILER_APPLY(msg, profile)
> 2020-01-09 18:11:31 ERROR DefaultMessageRouter:123 - error while executing
> profile; profile='profiler_test', error='Unable to parse: REGEXP_MATCH(line,
> '\\') due to: Unexpected internal error near index 1
> \ with relevant variables line=test'
> org.apache.metron.stellar.dsl.ParseException: Unable to parse:
> REGEXP_MATCH(line, '\\') due to: Unexpected internal error near index 1
> \ with relevant variables line=test {code}
>
> METRON-1922 - this does not affect me
> METRON-2002 - this might be relevant
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on pull request #42: METRON-2347: [BRO-PLUGIN-KAFKA] Use docker compose in end to end tests
JonZeolla commented on pull request #42: URL: https://github.com/apache/metron-bro-plugin-kafka/pull/42#issuecomment-621248256 Thanks - give me a bit to wrap up responses to your feedback, hopefully I'll get time in the next couple of days. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on a change in pull request #42: METRON-2347: [BRO-PLUGIN-KAFKA] Use docker compose in end to end tests
JonZeolla commented on a change in pull request #42:
URL:
https://github.com/apache/metron-bro-plugin-kafka/pull/42#discussion_r417359367
##
File path: docker/scripts/docker_run_create_topic_in_kafka.sh
##
@@ -80,7 +80,10 @@ echo "Running docker_run_create_topic_in_kafka with "
echo "NETWORK_NAME = $NETWORK_NAME"
echo "==="
-docker run --rm --network "${NETWORK_NAME}" ches/kafka \
+# TODO: Fix this
+sleep 2s
Review comment:
I have a solution for this which is `docker-compose` native, just need
to find some time to wrap it up.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
