[GitHub] nifi issue #858: NIFI-1971 - Introduce QueryWhois processor
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/858 @trixpan thanks for the changes, LGTM, full build with contrib-check, tested the processor with various configurations. Unless someone else wants to have a look, I'll merge it in master soon. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #858: NIFI-1971 - Introduce QueryWhois processor
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/858 @pvillard31 thanks for finding this. Could not believe... missing curly bracket after and else. :smile: --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #858: NIFI-1971 - Introduce QueryWhois processor
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/858 @trixpan Having an error totally makes sense, my concern was more about the exception being thrown on this line of code: session.transfer(flowFilesNotMatched, REL_NOT_FOUND); --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #858: NIFI-1971 - Introduce QueryWhois processor
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/858 @pvillard31 I just noticed the git woes I've noticed before were a bit more extensive that I initially realised. Some of the code changes made during code review were reverted when forced a push. Fixing this up and will let you know when done. Cheers --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #858: NIFI-1971 - Introduce QueryWhois processor
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/858 @pvillard31 Expected error as your regex string has no capture groups. I am not sure if I can test the regex contains at least one capture group but will try --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #858: NIFI-1971 - Introduce QueryWhois processor
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/858 @trixpan With: https://cloud.githubusercontent.com/assets/11541012/18610132/56f7abf0-7d14-11e6-8ac1-aea5e02b63a9.png";> I have: https://cloud.githubusercontent.com/assets/11541012/18610135/6f37dc4e-7d14-11e6-8595-cef902d93db0.png";> Could you have a look? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #858: NIFI-1971 - Introduce QueryWhois processor
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/858 Yes @trixpan it was with regex capture, next time I'll be more accurate in the regex to remove the newline. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #858: NIFI-1971 - Introduce QueryWhois processor
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/858 @pvillard31 can you confirm if this was generated by Split or Regex capture? Note that the regex is a multiline match, therefore user can remove the newline via regular expression. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #858: NIFI-1971 - Introduce QueryWhois processor
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/858 That's a detail, but regarding your last fix, I have now the following result: -- Standard FlowFile Attributes Key: 'entryDate' Value: 'Mon Sep 12 21:05:00 CEST 2016' Key: 'lineageStartDate' Value: 'Mon Sep 12 21:05:00 CEST 2016' Key: 'fileSize' Value: '0' FlowFile Attribute Map Content Key: 'enrich.whois.record0.group0' Value: '9394 | 123.69.0.0/16 | CTTNET | CN | chinatietong.com | China Tietong Telecommunications Corporation ' Key: 'filename' Value: '2645792035303317' Key: 'path' Value: './' Key: 'src.ip' Value: '123.69.123.40' Key: 'uuid' Value: '25d3744c-b5d4-4147-9cc5-47f4f47ff046' -- I'd try to remove the unnecessary carriage return at the end of the value. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #858: NIFI-1971 - Introduce QueryWhois processor
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/858 @pvillard31 feedback addressed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #858: NIFI-1971 - Introduce QueryWhois processor
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/858 @pvillard31 it is a bug caused by an extra newline. I missed due to some simplifications done within the jUnit. It is caused by the new line present in here: https://github.com/apache/nifi/pull/858/files#diff-5c34d310642cc536b1c8b2f6a87c7043R223 This newline should only be added if the processor is operating in batch mode. In summary the whois input should be: `origin 123.36.123.1` instead of ``` origin 123.36.123.1 ``` You can reproduce by doing a telnet to port 43 and sending the payloads above. Will fix and re-upload soon --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #858: NIFI-1971 - Introduce QueryWhois processor
Github user pvillard31 commented on the issue:
https://github.com/apache/nifi/pull/858
Hey @trixpan
Few remarks, regarding the ``customValidate``, I'd suggest the following
example:
java
results.add(new ValidationResult.Builder()
.input(validationContext.getProperty(BATCH_SIZE).getValue())
.subject(QUERY_PARSER.getDisplayName())
.explanation("NONE parser does not support batching.
Configure Batch Size to 1 or use another parser.")
.valid(false)
.build());
This way instead of:
'' validated against 'QUERY_PARSER' is invalid because NONE parser...
I have:
'500' validated against 'Results Parser' is invalid because NONE parser...
Thoughts?
Also, when I set the following configuration:
https://cloud.githubusercontent.com/assets/11541012/18432639/5e696a8e-78e3-11e6-932c-a128eea3cc55.png";>
I have the following result:
--
Standard FlowFile Attributes
Key: 'entryDate'
Value: 'Mon Sep 12 12:14:31 CEST 2016'
Key: 'lineageStartDate'
Value: 'Mon Sep 12 12:14:31 CEST 2016'
Key: 'fileSize'
Value: '0'
FlowFile Attribute Map Content
Key: 'enrich.whois.record0.group0'
Value: 'Request must be in the form of 'origin a.b.c.d' or 'peer
a.b.c.d' where a.b.c.d is a valid IPv4 address.
'
Key: 'filename'
Value: '2616257872984957'
Key: 'path'
Value: './'
Key: 'src.ip'
Value: '123.36.123.1'
Key: 'uuid'
Value: '0bc829ab-3423-4f8d-bed3-dd3c8ee7db23'
--
Is that expected? I'd assume (with the processor configuration) that the
result is not parsed in the same way but that the request is still performed
correctly, no?
Will continue reviewing later today.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #858: NIFI-1971 - Introduce QueryWhois processor
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/858 @pvillard31 any chance of completing this PR ? :smile: --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #858: NIFI-1971 - Introduce QueryWhois processor
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/858 @pvillard31 - reworded validators, unified the regex / split parses to use numeric capture groups/columns --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #858: NIFI-1971 - Introduce QueryWhois processor
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/858 As you can see the KEY group is documented within NiFi code but the UI lacks such explanation https://github.com/apache/nifi/pull/858/files#diff-d13cbfd36645869513f9d09b76636240R183 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #858: NIFI-1971 - Introduce QueryWhois processor
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/858 @pvillard31 The error is expected as the regex does not contain a capture group named KEY. It is a bug as I should detect KEY or whatever value was chosen. But your feedback highlights that instead of having named captures on regex vs numbered captures on split, we rather have a single approach (i.e. numbered captures) across both parsing strategies. I will fix it and also provide some better insight via validation messages --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #858: NIFI-1971 - Introduce QueryWhois processor
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/858 @trixpan Just made some tests with your template. I played with the parameters to check custom validation and the messages do not sound clear to me. For example: https://cloud.githubusercontent.com/assets/11541012/17750995/5cc41aa8-64c5-11e6-9935-32f5e327e8ae.png";> https://cloud.githubusercontent.com/assets/11541012/17751000/61378336-64c5-11e6-8a64-0d1189f15e34.png";> Then when coming back to the following configuration, I got this exception: https://cloud.githubusercontent.com/assets/11541012/17751026/86bff2dc-64c5-11e6-9ac8-2cead5777aaf.png";> https://cloud.githubusercontent.com/assets/11541012/17751028/8b868d9e-64c5-11e6-9ee7-ecdd3893d4d1.png";> Could you have a look? or let me know if this is expected. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #858: NIFI-1971 - Introduce QueryWhois processor
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/858 @pvillard31 given your experience reviewing QueryDNS, would you mind having a look on this one? Different interface, similar outcome. Cheers --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #858: NIFI-1971 - Introduce QueryWhois processor
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/858 A gist to help people going https://gist.github.com/trixpan/9c663866d2b02322650bfddde3eefb6e --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #858: NIFI-1971 - Introduce QueryWhois processor
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/858 # Note to reviewers: # This is a logging pipeline focused processor that should be capable of reaching to a Whois server and grabbing information that can be used to enrich data. The junit uses powermockito to prevent internet access dependency during build, however, real world testing can be done against the following Whois based APIs: 1. http://www.team-cymru.org/IP-ASN-mapping.html 2. https://www.shadowserver.org/wiki/pmwiki.php/Services/IP-BGP#toc4 3. http://www.pwhois.org/webquery.who Please shout in case you have questions. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
