Thanks Bryan and Thad for looking into the issue. On Tue, Jun 28, 2016 at 7:04 PM, Thad Guidry <[email protected]> wrote:
> I've added my additional comments. > > Looks good enough now. > > > Thad > +ThadGuidry <https://www.google.com/+ThadGuidry> > > On Tue, Jun 28, 2016 at 8:20 AM, Bryan Bende <[email protected]> wrote: > >> Thanks Thad. I created this JIRA: >> https://issues.apache.org/jira/browse/NIFI-2135 >> >> If I said anything wrong there please let me know, or add to the JIRA. >> >> >> >> >> On Tue, Jun 28, 2016 at 8:59 AM, Thad Guidry <[email protected]> >> wrote: >> >>> Yes, you want updateOne() >>> https://docs.mongodb.com/manual/reference/method/db.collection.updateOne/ >>> Not replaceOne() >>> https://docs.mongodb.com/manual/reference/method/db.collection.replaceOne/ >>> >>> However, also look at the upsert option itself as well which does a >>> replace if NO MATCH >>> https://api.mongodb.com/java/3.2/com/mongodb/client/model/UpdateOptions.html >>> which basically prescribes to this kind of handling >>> https://docs.mongodb.com/manual/reference/method/db.collection.update/ >>> >>> >>> Thad >>> +ThadGuidry <https://www.google.com/+ThadGuidry> >>> >>> On Tue, Jun 28, 2016 at 7:26 AM, Bryan Bende <[email protected]> wrote: >>> >>>> I'm not very familiar with MongoDB, but I can see that when using >>>> "update" mode PutMongo does this: >>>> >>>> else { >>>> // update >>>> final boolean upsert = context.getProperty(UPSERT).asBoolean(); >>>> final String updateKey = >>>> context.getProperty(UPDATE_QUERY_KEY).getValue(); >>>> final Document query = new Document(updateKey, doc.get(updateKey)); >>>> >>>> collection.replaceOne(query, doc, new >>>> UpdateOptions().upsert(upsert)); >>>> logger.info("updated {} into MongoDB", new Object[] { flowFile }); >>>> } >>>> >>>> I'm wondering if that collection.replaceOne() is the problem, I see >>>> there is collection.updateOne() which sounds more correct here. >>>> >>>> If someone with more MongoDB experience could verify this I would be >>>> happy to open a JIRA and get this changed. >>>> >>>> -Bryan >>>> >>>> On Tue, Jun 28, 2016 at 5:32 AM, Asanka Sanjaya Herath < >>>> [email protected]> wrote: >>>> >>>>> Hi Bryan, >>>>> >>>>> Your suggestion was worked fine. I have another question. This is not >>>>> related to the subject, but it is related to PutMongo processor. How can I >>>>> use put mongo processor to add a new key value pair to an existing >>>>> document? The flow file contains the document object Id. I have set 'mode' >>>>> property to 'update' and 'upsert' property to false and 'update query key' >>>>> property to '_id'. Flow file content is something like this. >>>>> >>>>> { >>>>> _id:ObjectId(577216f0154b943fe8068079) >>>>> expired:true >>>>> } >>>>> >>>>> Without inserting the 'expired:true', it replaces the whole document >>>>> with the given one. So is there a way to insert the new key value pair >>>>> to >>>>> collection without replacing the whole collection in MongoDB using >>>>> putmongo >>>>> processor? Your concern regarding this is highly appreciated. >>>>> >>>>> >>>>> >>>>> On Mon, Jun 27, 2016 at 6:43 PM, Asanka Sanjaya Herath < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi Bryan, >>>>>> Thank you for the input. That really helps. I'll try that. >>>>>> >>>>>> On Mon, Jun 27, 2016 at 6:31 PM, Bryan Bende <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> Right now AttributesToJson does treat everything as strings. I >>>>>>> remember a previous discussion about adding support for different types, >>>>>>> but I can't find a JIRA that references this. >>>>>>> >>>>>>> One option to work around this could be to use ReplaceText to >>>>>>> construct the JSON, instead of AttributesToJson. You could set the >>>>>>> Replacement Value property to something like: >>>>>>> >>>>>>> { >>>>>>> "dataSourceId" : "${datasource}", >>>>>>> "filename" : "${filename}", >>>>>>> "sent_date" : ${sent_date}, >>>>>>> "uuid" : "${uuid}", >>>>>>> "originalSource" : "${originalsource}" >>>>>>> } >>>>>>> >>>>>>> Of course using the appropriate attribute names. >>>>>>> >>>>>>> Another option is that in the upcoming 0.7.0 release, there is a new >>>>>>> processor to transform JSON using JOLT. With that processor you may be >>>>>>> able >>>>>>> to take the output of AttributesToJson and apply a transform that >>>>>>> converts >>>>>>> the date field to remove the quotes. >>>>>>> >>>>>>> Hope that helps. >>>>>>> >>>>>>> -Bryan >>>>>>> >>>>>>> On Mon, Jun 27, 2016 at 8:16 AM, Asanka Sanjaya Herath < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> I'm trying to insert a flow file to MongoDb which has a date record >>>>>>>> as an attribute. First I sent that flow file through an attribute to >>>>>>>> JSON >>>>>>>> processor, so that all attributes are now converted to a Json document >>>>>>>> in >>>>>>>> flow file body. When I insert that flow file to Mongodb using PutMongo >>>>>>>> processor, it saves the "sent_date" attribute as a String. I want this >>>>>>>> to >>>>>>>> be saved as an ISO date object. >>>>>>>> >>>>>>>> My flow file looked like this. >>>>>>>> >>>>>>>> { >>>>>>>> "dataSourceId" : "xxxx", >>>>>>>> "filename" : "979f7bc5-a395-4396-9625-69fdb2c806c6", >>>>>>>> "sent_date" : "Mon Jan 18 04:50:50 IST 2016", >>>>>>>> "uuid" : "77a5ef56-8b23-40ee-93b5-78c6323e0e1c", >>>>>>>> "originalSource" : "ImportedZip" >>>>>>>> } >>>>>>>> >>>>>>>> Then I prepend "ISODate" to "sent_date" attribute using another >>>>>>>> processor. So now my flow file content looks like this. >>>>>>>> { >>>>>>>> "dataSourceId" : "xxxx", >>>>>>>> "filename" : "979f7bc5-a395-4396-9625-69fdb2c806c6", >>>>>>>> "sent_date" : "ISODate('Mon Jan 18 04:50:50 IST 2016')", >>>>>>>> "uuid" : "77a5ef56-8b23-40ee-93b5-78c6323e0e1c", >>>>>>>> "originalSource" : "ImportedZip" >>>>>>>> } >>>>>>>> >>>>>>>> But still It is saved as a string in MongoDB, because of the double >>>>>>>> quotation marks. Is there a way to remove those double quotation marks >>>>>>>> when >>>>>>>> convert using AttributeToJson processor? >>>>>>>> >>>>>>>> Any help is appreciated. >>>>>>>> >>>>>>>> -- >>>>>>>> Thanks, >>>>>>>> Regards, >>>>>>>> ASH >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Thanks, >>>>>> Regards, >>>>>> ASH >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Thanks, >>>>> Regards, >>>>> ASH >>>>> >>>> >>>> >>> >> > -- Thanks, Regards, ASH
