[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-04-17 Thread JohannesDaniel
Github user JohannesDaniel commented on the issue: https://github.com/apache/nifi/pull/2517 @MikeThomsen - added provenance receive notifications - added tests for attributes - added upper limit of solr start parameter (1) - enhanced documentation for upper limit

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-04-17 Thread MikeThomsen
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2517 @JohannesDaniel Sounds good. Make sure it includes the request documentation as well. That's going to be a big deal in making this easy for new users to use. Thanks. ---

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-04-17 Thread JohannesDaniel
Github user JohannesDaniel commented on the issue: https://github.com/apache/nifi/pull/2517 (means in 5-6 hous) ---

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-04-17 Thread JohannesDaniel
Github user JohannesDaniel commented on the issue: https://github.com/apache/nifi/pull/2517 hi @MikeThomsen I will upload the changes this evening. ---

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-04-16 Thread MikeThomsen
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2517 Have you had a chance to add the missing provenance tracking? ---

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-04-13 Thread MikeThomsen
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2517 @JohannesDaniel I missed those tests for some reason. Basically I was just looking for something that goes into deep pagination behavior. Disregard for now. Thanks. ---

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-04-13 Thread JohannesDaniel
Github user JohannesDaniel commented on the issue: https://github.com/apache/nifi/pull/2517 @MikeThomsen There are two tests that test Solr result paging: ``` testRetrievalOfFullResults() testRetrievalOfFullResults2() ``` What do you mean exactly? A test that

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-04-13 Thread JohannesDaniel
Github user JohannesDaniel commented on the issue: https://github.com/apache/nifi/pull/2517 @ottobackwards you made my day! ---

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-04-13 Thread MikeThomsen
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2517 @ottobackwards > @JohannesDaniel If there are no results from a page request, does that mean there is no output? I ask because I have a PR for that use case with QueryElasticsearchHttp.

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-04-13 Thread ottobackwards
Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2517 how are you building? Like what command line? if you are using -Dmaven.test.skip=true you will have this problem if you want to skip tests use -DskipTests ---

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-04-13 Thread JohannesDaniel
Github user JohannesDaniel commented on the issue: https://github.com/apache/nifi/pull/2517 @ottobackwards When I try this I always run into the same problem: [ERROR] Failed to execute goal on project nifi-livy-processors: Could not resolve dependencies for project

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-04-13 Thread ottobackwards
Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2517 If you are building Nifi, and need to get things working after merging in the new version poms, you can do a mvn clean install -DskipTests and that should populate your local repo with the

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-04-13 Thread JohannesDaniel
Github user JohannesDaniel commented on the issue: https://github.com/apache/nifi/pull/2517 @MikeThomsen I'm still impeded, because my Maven does not download the NIFI-SNAPSHOTS. It downloads all dependencies except the SNAPSHOTS. When I follow the links (e. g.

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-04-13 Thread ottobackwards
Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2517 @JohannesDaniel If there are no results from a page request, does that mean there is no output? I ask because I have a PR for that use case with QueryElasticsearchHttp.

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-04-13 Thread MikeThomsen
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2517 @JohannesDaniel I didn't see a test that tests deep pagination. If you don't have one, please add one. You can just throw garbage data in there like random strings. Also, you should add at least

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-04-13 Thread MikeThomsen
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2517 Ok. Verified with a test run that it appears to run fine against a vanilla Solr installation. Docker: `docker run --name testsolr -p 8983:8983 solr:latest solr-demo`

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-04-13 Thread MikeThomsen
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2517 > should be an expectation that there is some commonality between them. Solr and ES are increasingly used for different use cases these days. Solr tends to be more commonly used for pure

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-04-12 Thread ottobackwards
Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2517 My comment is more about the way you work with the ES vs. the Solr processors and if there is or should be an expectation that there is some commonality between them. Like how this relates to

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-04-12 Thread JohannesDaniel
Github user JohannesDaniel commented on the issue: https://github.com/apache/nifi/pull/2517 @ottobackwards deep paging should be done with scrolling using cursor marks (as it is done in GetSolr). simple paging can be done by sucessively increasing the offset (start parameter).

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-04-12 Thread ottobackwards
Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2517 Are there best practices for handling paging? I think the ES processors handle this differently. Does it matter? ---

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-04-12 Thread JohannesDaniel
Github user JohannesDaniel commented on the issue: https://github.com/apache/nifi/pull/2517 @MikeThomsen Thank you for your help with Git!! @bbende - Solr core is a test dependency again (I actually had a reason to add this as a non-test dependency, but it was not a good

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-04-12 Thread MikeThomsen
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2517 Builds just fine for me locally. If you're still having build problems, try deleting everything under org/apache/nifi from .m2/repository in your home folder. ---

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-04-12 Thread MikeThomsen
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2517 Ok. I'll take a look and let you know sometime in a little while. Got some other things on my plate at the moment. ---

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-04-12 Thread JohannesDaniel
Github user JohannesDaniel commented on the issue: https://github.com/apache/nifi/pull/2517 @MikeThomsen done :) ---

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-04-12 Thread MikeThomsen
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2517 @JohannesDaniel also, you don't need to wait for a new build to push the changes. Overwriting everything will schedule the job to be killed and a new one started. ---

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-04-12 Thread MikeThomsen
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2517 @JohannesDaniel nope. If you did a rebase, those new commits would be behind your commits, not merged ahead of them. What a rebase does is it sets aside your commits, brings the base pointer for

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-04-12 Thread JohannesDaniel
Github user JohannesDaniel commented on the issue: https://github.com/apache/nifi/pull/2517 @MikeThomsen or is this a normal thing that the commits of others are shown here after the rebase? ---

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-04-12 Thread MikeThomsen
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2517 @JohannesDaniel see my earlier comment about rebasing. You **cannot** just do `git merge master` into another branch like that since master is the foundation of that branch. Don't worry, though.

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-04-12 Thread JohannesDaniel
Github user JohannesDaniel commented on the issue: https://github.com/apache/nifi/pull/2517 @MikeThomsen omg. now there are commits of others in the branch?? maybe I should simply close this PR and insert the code into a new branch. ---

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-04-12 Thread MikeThomsen
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2517 @JohannesDaniel go ahead and check in the change even if it's broken. I'll help take a look. ---

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-04-12 Thread ottobackwards
Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2517 If you are building and skipping the tests use -DskipTests=true. You can also do to do at least one mvn install run as well so that your local repo has everything. ---

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-04-11 Thread MikeThomsen
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2517 Do this: 1. git checkout master 2. git pull upstream master (whatever you call github.com/apache/nifi master) 3. git checkout NIFI-4516 4. git rebase master 5. git push

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-04-11 Thread JohannesDaniel
Github user JohannesDaniel commented on the issue: https://github.com/apache/nifi/pull/2517 @MikeThomsen rebased everything, build for solr processors works without any problems. however, when I try to build the whole application, I receive the following error: [ERROR] Failed to

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-04-11 Thread MikeThomsen
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2517 @JohannesDaniel just do a rebase against master. You shouldn't use `git merge master` for something like this. If you want to do a big squash event to make the rebase easier, go ahead. ---

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-04-11 Thread JohannesDaniel
Github user JohannesDaniel commented on the issue: https://github.com/apache/nifi/pull/2517 @MikeThomsen Yeah, this time I looked at this, but haven't had time to fix this. I think I have to merge master into the branch due to updates. Local build worked. But thanks for the ping :)

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-04-11 Thread MikeThomsen
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2517 @JohannesDaniel Build failed because of missing imports: > [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.6.0:compile (default-compile) on project

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-03-27 Thread JohannesDaniel
Github user JohannesDaniel commented on the issue: https://github.com/apache/nifi/pull/2517 Hi @MikeThomsen travic-ci test failed due to "The log length has exceeded the limit of 4 MB (this usually means that the test suite is raising the same exception over and over)." Is there

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-03-24 Thread MikeThomsen
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2517 > Is that what you meant with integration tests? What do you mean with FetchSolrIT? Do you have a link for an example? You would create a JUnit test called `FetchSolrIT` and follow the

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-03-24 Thread JohannesDaniel
Github user JohannesDaniel commented on the issue: https://github.com/apache/nifi/pull/2517 @MikeThomsen I will keep an eye on the CI tests in the future, thanks for the advice. Actually, I did not take them into account as they frequently appear to fail for no reason... I tested

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-03-24 Thread MikeThomsen
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2517 I have Solr Cloud in a docker session on the usual ports, so if you haven't done them yet some integration tests (ex: `FetchSolrIT`) should be added as well. ---

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-03-24 Thread MikeThomsen
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2517 BTW, that's the reason why your Travis CI build failed last time. Keep an eye on that because it'll slow down your code reviews. ---

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-03-24 Thread MikeThomsen
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2517 Believe me, I understand. I just checked out your branch and ran into this when doing a full build with `mvn clean install` from the root, can you verify/fix? > [ERROR] The project

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-03-24 Thread JohannesDaniel
Github user JohannesDaniel commented on the issue: https://github.com/apache/nifi/pull/2517 @MikeThomsen ok, sorry, I am too impatient :D ---

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-03-23 Thread MikeThomsen
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2517 Sorry, other stuff got in the way. I'll try to find some time soon. ---

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-03-23 Thread JohannesDaniel
Github user JohannesDaniel commented on the issue: https://github.com/apache/nifi/pull/2517 @MikeThomsen any news? ---

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-03-16 Thread JohannesDaniel
Github user JohannesDaniel commented on the issue: https://github.com/apache/nifi/pull/2517 Oh, sorry. Done. ---

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-03-16 Thread MikeThomsen
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2517 Sorry, haven't had time. There's a merge conflict now. Can you fix that? ---

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-03-16 Thread JohannesDaniel
Github user JohannesDaniel commented on the issue: https://github.com/apache/nifi/pull/2517 @MikeThomsen Any news? ---

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-03-10 Thread JohannesDaniel
Github user JohannesDaniel commented on the issue: https://github.com/apache/nifi/pull/2517 Refactored the treatment of flowfiles routed to relationship ORIGINAL. More attributes are added to flowfiles for better descriptions of requests / responses. Additionally, I adjusted some

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-03-09 Thread JohannesDaniel
Github user JohannesDaniel commented on the issue: https://github.com/apache/nifi/pull/2517 Tested the processor in a local build where it worked as expected. ---

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-03-08 Thread MikeThomsen
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2517 You should consider making the output format configurable. The Solr projects I've worked on in the past have used JSON instead of XML. ---

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

2018-03-06 Thread JohannesDaniel
Github user JohannesDaniel commented on the issue: https://github.com/apache/nifi/pull/2517 @ijokarumawak To be aligned with processor GetSolr I implemented two options for the data format of Solr results: Solr XML and record functions. However, facets and stats are written to