[GitHub] nifi issue #2561: NIFI-4035 Implement record-based Solr processors
Github user abhinavrohatgi30 commented on the issue: https://github.com/apache/nifi/pull/2561 @bbende @MikeThomsen Thanks for reviewing the pull request ---
[GitHub] nifi issue #2561: NIFI-4035 Implement record-based Solr processors
Github user bbende commented on the issue: https://github.com/apache/nifi/pull/2561 I was able to resolve the conflicts and everything looks good now, going to merge, thanks! ---
[GitHub] nifi issue #2561: NIFI-4035 Implement record-based Solr processors
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2561 @abhinavrohatgi30 While you were away, I merged another Solr-related commit and that's the reason you now have conflicts. ---
[GitHub] nifi issue #2561: NIFI-4035 Implement record-based Solr processors
Github user abhinavrohatgi30 commented on the issue: https://github.com/apache/nifi/pull/2561 I'm really sorry, it might take a while, I'm on a vacation and away from my workstation. I'll keep you updated as soon as I am back. ---
[GitHub] nifi issue #2561: NIFI-4035 Implement record-based Solr processors
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2561 @abhinavrohatgi30 You have a merge conflict in this branch. If you resolve it, I'll help @bbende finish the review. ---
[GitHub] nifi issue #2561: NIFI-4035 Implement record-based Solr processors
Github user abhinavrohatgi30 commented on the issue: https://github.com/apache/nifi/pull/2561 Hi, I've looked at the comments and I've made the following changes as part of the latest commit that cover all the comments : 1. Fixed the issue with Nested Records (The issue came up because of the change in field names in the previous commit) 2. Fixed the issue with Array of Records (It was generating an Object[] as opposed to a Record[] that I was expecting and as a result was storing the string representation of a Record) 3. Trimming field names individually 4. Adding Test cases for Nested Record, Array of Record and Record Parser failure 5. Using the getLogger() later in the code 6. Wrapping the Jsons in the additionalDetails.html in a tag I hope the processor now works as expected, let me know if any further changes are to be made Thanks ---
[GitHub] nifi issue #2561: NIFI-4035 Implement record-based Solr processors
Github user abhinavrohatgi30 commented on the issue: https://github.com/apache/nifi/pull/2561 @bbende I'll have a look at this and write test cases accordingly. ---
[GitHub] nifi issue #2561: NIFI-4035 Implement record-based Solr processors
Github user bbende commented on the issue:
https://github.com/apache/nifi/pull/2561
Nested records and arrays of nested records are not working correctly...
**Scenario 1 - Nested Record**
Schema:
```
{
"type": "record",
"name": "exams",
"fields" : [
{ "name": "first", "type": "string" },
{ "name": "last", "type": "string" },
{ "name": "grade", "type": "int" },
{
"name": "exam",
"type": {
"name" : "exam",
"type" : "record",
"fields" : [
{ "name": "subject", "type": "string" },
{ "name": "test", "type": "string" },
{ "name": "marks", "type": "int" }
]
}
}
]
}
```
Input:
```
{
"first": "Abhi",
"last": "R",
"grade": 8,
"exam": {
"subject": "Maths",
"test" : "term1",
"marks" : 90
}
}
```
Result:
```
java.util.NoSuchElementException: No value present
at java.util.Optional.get(Optional.java:135)
at
org.apache.nifi.processors.solr.SolrUtils.writeRecord(SolrUtils.java:313)
at
org.apache.nifi.processors.solr.SolrUtils.writeValue(SolrUtils.java:384)
at
org.apache.nifi.processors.solr.SolrUtils.writeRecord(SolrUtils.java:314)
at
org.apache.nifi.processors.solr.PutSolrRecord.onTrigger(PutSolrRecord.java:247)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
```
**Scenario 2 - Array of Records**
Schema:
```
{
"type": "record",
"name": "exams",
"fields" : [
{ "name": "first", "type": "string" },
{ "name": "last", "type": "string" },
{ "name": "grade", "type": "int" },
{
"name": "exams",
"type": {
"type" : "array",
"items" : {
"name" : "exam",
"type" : "record",
"fields" : [
{ "name": "subject", "type": "string" },
{ "name": "test", "type": "string" },
{ "name": "marks", "type": "int" }
]
}
}
}
]
}
```
Input:
```
{
"first": "Abhi",
"last": "R",
"grade": 8,
"exams": [
{
"subject": "Maths",
"test" : "term1",
"marks" : 90
},
{
"subject": "Physics",
"test" : "term1",
"marks" : 95
}
]
}
```
Result:
Solr Document with multi-valued field exams where the values are the
toString of a MapRecord:
```
"exams":["org.apache.nifi.serialization.record.MapRecord:MapRecord[{marks=90,
test=term1, subject=Maths}]",
"org.apache.nifi.serialization.record.MapRecord:MapRecord[{marks=95,
test=term1, subject=Physics}]"],
```
Should have created fields like exams_marks, exams_test, exams_subject.
Here is a full template for the two scenarios:
https://gist.githubusercontent.com/bbende/edc2e7d61db83b29533ac3fc520de30f/raw/8764d50ed5e14d876c53a0b84b3af5741d910b3b/PutSolrRecordTesting.xml
There needs to be unit tests that cover both these cases.
---
[GitHub] nifi issue #2561: NIFI-4035 Implement record-based Solr processors
Github user bbende commented on the issue: https://github.com/apache/nifi/pull/2561 Thanks, will try to take a look in a few days, unless someone gets to it first. ---
[GitHub] nifi issue #2561: NIFI-4035 Implement record-based Solr processors
Github user abhinavrohatgi30 commented on the issue: https://github.com/apache/nifi/pull/2561 Hi @bbende , I've brought it down to a single commit, can you have a look at it now? ---
[GitHub] nifi issue #2561: NIFI-4035 Implement record-based Solr processors
Github user bbende commented on the issue: https://github.com/apache/nifi/pull/2561 I tried to re-base this against master so I could squash it down to a single commit, but the re-base is encountering a lot of conflicts, which really shouldn't be happening because its conflicting with itself. Can you work through getting it down to a single commit? Normally it should just be: `git rebase -i upstream/master` Then in the list of commits you choose "s" for all the commits except the top-one, which squashes them all into the top one. Then force push. ---
[GitHub] nifi issue #2561: NIFI-4035 Implement record-based Solr processors
Github user abhinavrohatgi30 commented on the issue: https://github.com/apache/nifi/pull/2561 I'm done with the changes that @bbende and @MikeThomsen have suggested ---
[GitHub] nifi issue #2561: NIFI-4035 Implement record-based Solr processors
Github user bbende commented on the issue: https://github.com/apache/nifi/pull/2561 I would try doing a rebase against master to see what happens. In the worst case situation you would have to create another branch off latest master, and then individually cherry-pick your commits from this branch over to the new branch, to get rid of those other commits that are in between yours, but only do that if you can't get this branch straightened out. ---
[GitHub] nifi issue #2561: NIFI-4035 Implement record-based Solr processors
Github user abhinavrohatgi30 commented on the issue: https://github.com/apache/nifi/pull/2561 Sorry, instead of doing the force push i resolved conflicts and did a push, can i now do the rebase again on the current commit or will i have to add a new commit inorder to rebase from the master branch? ---
[GitHub] nifi issue #2561: NIFI-4035 Implement record-based Solr processors
Github user bbende commented on the issue: https://github.com/apache/nifi/pull/2561 Yea when you update your branch you should be doing something like the following... ``` git fetch upstream git rebase upstream/master ``` This assumes "upstream" points to either Apache NiFi git repo or Apache NiFi Github. Using rebase will apply all the incoming commits from upstream/master to your branch and then put your commits back on top of that so it looks like yours are always the latest. You then need to force push to your remote branch `git push origin your-branch --force` ---
[GitHub] nifi issue #2561: NIFI-4035 Implement record-based Solr processors
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2561 @abhinavrohatgi30 Looks like your latest push grabbed a bunch of other folks' commits. Unless @bbende disagrees, I think you're going to need to rebase and repush. ---
[GitHub] nifi issue #2561: NIFI-4035 Implement record-based Solr processors
Github user abhinavrohatgi30 commented on the issue: https://github.com/apache/nifi/pull/2561 NIFI-4035 Adding a PutSolrRecord Processor that reads NiFi Records and indexes them into Solr as SolrDocuments. ---
