Re: ReplaceText processor configuration help

2016-04-26 Thread Matt Burgess 
You can certainly use ExecuteScript with Groovy for JSON-to-JSON
conversion, I have a blog post about that here:
http://funnifi.blogspot.com/2016/02/executescript-json-to-json-conversion.html
 I am using UTF-8 in that example but you can use whatever you like,
or even let the user choose the value for a dynamic property like
"scriptCharset" or something, and use that value (scriptCharset.value)
in your Groovy script.

ExecuteScript can be very slow for high data loads as it executes 1
task at a time. I am looking into improving that, but if you need more
performance, you can use InvokeScriptedProcessor to do the same thing
(I can outline the differences if you like).

Regards,
Matt

On Tue, Apr 26, 2016 at 1:05 PM, Igor Kravzov <igork.ine...@gmail.com> wrote:
> Hi Matt,
>
> You described an interesting process. I will think about it.
> Initially I wanted to grab just some properties, like "entities" and "text",
> of original JSON and create anew one.
> ReplaceTexts works fine as long as "text" value does not have quotes inside
> the text. Once it has quotes and goes through the replace processor, JSON
> became broken.
>
> Also looking for some alterbatives like using Groovy for JSON-to-JSON
> conversion.  But not sure how StandardCharsets.UTF_8 will work with
> multi-byte languages.
>
>
> On Tue, Apr 26, 2016 at 12:11 PM, Matt Burgess <mattyb...@gmail.com> wrote:
>>
>> Yes, I think you'll be better off with Aldrin's suggestion of
>> ReplaceText. Then you can put the value of the attribute(s) directly
>> into the content.  For example, if you have two attributes "entities"
>> and "users", and you want a JSON doc with those two objects inside,
>> you can use ReplaceText with the following for replacement:
>>
>> {"entities": ${entities}, "users": ${users}}
>>
>> Note this "manually" transforms the JSON. Before we get the
>> TransformJSON processor, this is a decent workaround if you know what
>> the resulting JSON document should look like (and if you have
>> attributes containing the desired values).
>>
>> If you're doing this to insert into Elasticsearch, you might want to
>> handle entities and users separately and have "types" in ES for
>> "entities" and "users". In that case you could use EvaluateJsonPath to
>> get both attributes out, then wire the "success" relationship to two
>> different ReplaceTexts, one to store the entities and one for users.
>> Then you could add an attribute called "es.type" (for example), set to
>> "entities" and "users" respectively. Then you can send both forks to a
>> PutElasticsearch, setting the Type property to "${es.type}". That will
>> put the entities documents into the entities type and the same for
>> users. This will help with indexing versus one huge document.
>>
>> This process can be broken down into individual entities and users, if
>> you'd like a separate ES document for each. In that case you'd likely
>> need a SplitJson after the ReplaceText, pointing at the array of
>> entity/user objects. Then you'll get a flow file per entity/user,
>> meaning you'll get a separate ES doc for each entity and user,
>> stored/indexed/categorized by its type.
>>
>> Does this help solve your use case? If not please let me know, I'm
>> happy to help work through this :)
>>
>> Regards,
>> Matt
>>
>> On Tue, Apr 26, 2016 at 11:51 AM, Igor Kravzov <igork.ine...@gmail.com>
>> wrote:
>> > I see.
>> > But I think I found the problem. It's AttributesToJson escapes the
>> > result.
>> >
>> > On Apr 26, 2016 11:46 AM, "McDermott, Chris Kevin (MSDU -
>> > STaTS/StorefrontRemote)" <chris.mcderm...@hpe.com> wrote:
>> >>
>> >> Hi Igor,
>> >>
>> >> jsonPath will return JSON as an unescaped String.
>> >>
>> >> Chris
>> >>
>> >> From: Igor Kravzov
>> >> <igork.ine...@gmail.com<mailto:igork.ine...@gmail.com>>
>> >> Reply-To: "users@nifi.apache.org<mailto:users@nifi.apache.org>"
>> >> <users@nifi.apache.org<mailto:users@nifi.apache.org>>
>> >> Date: Monday, April 25, 2016 at 2:27 PM
>> >> To: "users@nifi.apache.org<mailto:users@nifi.apache.org>"
>> >> <users@nifi.apache.org<mailto:users@nifi.apache.org>>
>> >> Subject: Re: ReplaceText processor configuration help
>> >>
>> >> Hi Chris,
&g

Re: ReplaceText processor configuration help

2016-04-26 Thread Igor Kravzov 
Hi Matt,

You described an interesting process. I will think about it.
Initially I wanted to grab just some properties, like "entities" and
"text",  of original JSON and create anew one.
ReplaceTexts works fine as long as "text" value does not have quotes inside
the text. Once it has quotes and goes through the replace processor, JSON
became broken.

Also looking for some alterbatives like using Groovy for JSON-to-JSON
conversion.  But not sure how StandardCharsets.UTF_8 will work with
multi-byte languages.


On Tue, Apr 26, 2016 at 12:11 PM, Matt Burgess <mattyb...@gmail.com> wrote:

> Yes, I think you'll be better off with Aldrin's suggestion of
> ReplaceText. Then you can put the value of the attribute(s) directly
> into the content.  For example, if you have two attributes "entities"
> and "users", and you want a JSON doc with those two objects inside,
> you can use ReplaceText with the following for replacement:
>
> {"entities": ${entities}, "users": ${users}}
>
> Note this "manually" transforms the JSON. Before we get the
> TransformJSON processor, this is a decent workaround if you know what
> the resulting JSON document should look like (and if you have
> attributes containing the desired values).
>
> If you're doing this to insert into Elasticsearch, you might want to
> handle entities and users separately and have "types" in ES for
> "entities" and "users". In that case you could use EvaluateJsonPath to
> get both attributes out, then wire the "success" relationship to two
> different ReplaceTexts, one to store the entities and one for users.
> Then you could add an attribute called "es.type" (for example), set to
> "entities" and "users" respectively. Then you can send both forks to a
> PutElasticsearch, setting the Type property to "${es.type}". That will
> put the entities documents into the entities type and the same for
> users. This will help with indexing versus one huge document.
>
> This process can be broken down into individual entities and users, if
> you'd like a separate ES document for each. In that case you'd likely
> need a SplitJson after the ReplaceText, pointing at the array of
> entity/user objects. Then you'll get a flow file per entity/user,
> meaning you'll get a separate ES doc for each entity and user,
> stored/indexed/categorized by its type.
>
> Does this help solve your use case? If not please let me know, I'm
> happy to help work through this :)
>
> Regards,
> Matt
>
> On Tue, Apr 26, 2016 at 11:51 AM, Igor Kravzov <igork.ine...@gmail.com>
> wrote:
> > I see.
> > But I think I found the problem. It's AttributesToJson escapes the
> result.
> >
> > On Apr 26, 2016 11:46 AM, "McDermott, Chris Kevin (MSDU -
> > STaTS/StorefrontRemote)" <chris.mcderm...@hpe.com> wrote:
> >>
> >> Hi Igor,
> >>
> >> jsonPath will return JSON as an unescaped String.
> >>
> >> Chris
> >>
> >> From: Igor Kravzov <igork.ine...@gmail.com igork.ine...@gmail.com>>
> >> Reply-To: "users@nifi.apache.org<mailto:users@nifi.apache.org>"
> >> <users@nifi.apache.org<mailto:users@nifi.apache.org>>
> >> Date: Monday, April 25, 2016 at 2:27 PM
> >> To: "users@nifi.apache.org<mailto:users@nifi.apache.org>"
> >> <users@nifi.apache.org<mailto:users@nifi.apache.org>>
> >> Subject: Re: ReplaceText processor configuration help
> >>
> >> Hi Chris,
> >>
> >> How will it help in my situation?
> >>
> >> On Mon, Apr 25, 2016 at 1:50 PM, McDermott, Chris Kevin (MSDU -
> >> STaTS/StorefrontRemote)
> >> <chris.mcderm...@hpe.com<mailto:chris.mcderm...@hpe.com>> wrote:
> >> Igor,
> >>
> >> I think the jsonPath extension to the EL is going to be the ticket
> [1].  A
> >> patch is available if you are willing to build NiFi yourself to test it
> out.
> >>
> >> Cheers,
> >> Chris
> >>
> >> [1] https://issues.apache.org/jira/browse/NIFI-1660
> >>
> >>
> >> From: Igor Kravzov
> >> <igork.ine...@gmail.com<mailto:igork.ine...@gmail.com> igork.ine...@gmail.com<mailto:igork.ine...@gmail.com>>>
> >> Reply-To:
> >> "users@nifi.apache.org<mailto:users@nifi.apache.org> users@nifi.apache.org<mailto:users@nifi.apache.org>>"
> >> <users@nifi.apache.org<mailto:users@nifi.apache.org> users@nifi.apache.org<mailto:users@nifi.apach

Re: ReplaceText processor configuration help

2016-04-26 Thread Matt Burgess 
Yes, I think you'll be better off with Aldrin's suggestion of
ReplaceText. Then you can put the value of the attribute(s) directly
into the content.  For example, if you have two attributes "entities"
and "users", and you want a JSON doc with those two objects inside,
you can use ReplaceText with the following for replacement:

{"entities": ${entities}, "users": ${users}}

Note this "manually" transforms the JSON. Before we get the
TransformJSON processor, this is a decent workaround if you know what
the resulting JSON document should look like (and if you have
attributes containing the desired values).

If you're doing this to insert into Elasticsearch, you might want to
handle entities and users separately and have "types" in ES for
"entities" and "users". In that case you could use EvaluateJsonPath to
get both attributes out, then wire the "success" relationship to two
different ReplaceTexts, one to store the entities and one for users.
Then you could add an attribute called "es.type" (for example), set to
"entities" and "users" respectively. Then you can send both forks to a
PutElasticsearch, setting the Type property to "${es.type}". That will
put the entities documents into the entities type and the same for
users. This will help with indexing versus one huge document.

This process can be broken down into individual entities and users, if
you'd like a separate ES document for each. In that case you'd likely
need a SplitJson after the ReplaceText, pointing at the array of
entity/user objects. Then you'll get a flow file per entity/user,
meaning you'll get a separate ES doc for each entity and user,
stored/indexed/categorized by its type.

Does this help solve your use case? If not please let me know, I'm
happy to help work through this :)

Regards,
Matt

On Tue, Apr 26, 2016 at 11:51 AM, Igor Kravzov <igork.ine...@gmail.com> wrote:
> I see.
> But I think I found the problem. It's AttributesToJson escapes the result.
>
> On Apr 26, 2016 11:46 AM, "McDermott, Chris Kevin (MSDU -
> STaTS/StorefrontRemote)" <chris.mcderm...@hpe.com> wrote:
>>
>> Hi Igor,
>>
>> jsonPath will return JSON as an unescaped String.
>>
>> Chris
>>
>> From: Igor Kravzov <igork.ine...@gmail.com<mailto:igork.ine...@gmail.com>>
>> Reply-To: "users@nifi.apache.org<mailto:users@nifi.apache.org>"
>> <users@nifi.apache.org<mailto:users@nifi.apache.org>>
>> Date: Monday, April 25, 2016 at 2:27 PM
>> To: "users@nifi.apache.org<mailto:users@nifi.apache.org>"
>> <users@nifi.apache.org<mailto:users@nifi.apache.org>>
>> Subject: Re: ReplaceText processor configuration help
>>
>> Hi Chris,
>>
>> How will it help in my situation?
>>
>> On Mon, Apr 25, 2016 at 1:50 PM, McDermott, Chris Kevin (MSDU -
>> STaTS/StorefrontRemote)
>> <chris.mcderm...@hpe.com<mailto:chris.mcderm...@hpe.com>> wrote:
>> Igor,
>>
>> I think the jsonPath extension to the EL is going to be the ticket [1].  A
>> patch is available if you are willing to build NiFi yourself to test it out.
>>
>> Cheers,
>> Chris
>>
>> [1] https://issues.apache.org/jira/browse/NIFI-1660
>>
>>
>> From: Igor Kravzov
>> <igork.ine...@gmail.com<mailto:igork.ine...@gmail.com><mailto:igork.ine...@gmail.com<mailto:igork.ine...@gmail.com>>>
>> Reply-To:
>> "users@nifi.apache.org<mailto:users@nifi.apache.org><mailto:users@nifi.apache.org<mailto:users@nifi.apache.org>>"
>> <users@nifi.apache.org<mailto:users@nifi.apache.org><mailto:users@nifi.apache.org<mailto:users@nifi.apache.org>>>
>> Date: Monday, April 25, 2016 at 11:45 AM
>> To:
>> "users@nifi.apache.org<mailto:users@nifi.apache.org><mailto:users@nifi.apache.org<mailto:users@nifi.apache.org>>"
>> <users@nifi.apache.org<mailto:users@nifi.apache.org><mailto:users@nifi.apache.org<mailto:users@nifi.apache.org>>>
>> Subject: Re: ReplaceText processor configuration help
>>
>> Aldrin,
>>
>> The overall goal is to extract some subset of attributes from tweet's
>> JSON, create a new JSON and ingest it into Elasticsearch for indexing.
>> Hope this helps.
>>
>> On Mon, Apr 25, 2016 at 11:18 AM, Aldrin Piri
>> <aldrinp...@gmail.com<mailto:aldrinp...@gmail.com><mailto:aldrinp...@gmail.com<mailto:aldrinp...@gmail.com>>>
>> wrote:
>> Igor,
>>
>> Thanks for the template.  It looks like the trouble is with
>> AttributesToJSON

Re: ReplaceText processor configuration help

2016-04-26 Thread McDermott, Chris Kevin (MSDU - STaTS/StorefrontRemote) 
Hi Igor,

jsonPath will return JSON as an unescaped String.

Chris

From: Igor Kravzov <igork.ine...@gmail.com<mailto:igork.ine...@gmail.com>>
Reply-To: "users@nifi.apache.org<mailto:users@nifi.apache.org>" 
<users@nifi.apache.org<mailto:users@nifi.apache.org>>
Date: Monday, April 25, 2016 at 2:27 PM
To: "users@nifi.apache.org<mailto:users@nifi.apache.org>" 
<users@nifi.apache.org<mailto:users@nifi.apache.org>>
Subject: Re: ReplaceText processor configuration help

Hi Chris,

How will it help in my situation?

On Mon, Apr 25, 2016 at 1:50 PM, McDermott, Chris Kevin (MSDU - 
STaTS/StorefrontRemote) 
<chris.mcderm...@hpe.com<mailto:chris.mcderm...@hpe.com>> wrote:
Igor,

I think the jsonPath extension to the EL is going to be the ticket [1].  A 
patch is available if you are willing to build NiFi yourself to test it out.

Cheers,
Chris

[1] https://issues.apache.org/jira/browse/NIFI-1660


From: Igor Kravzov 
<igork.ine...@gmail.com<mailto:igork.ine...@gmail.com><mailto:igork.ine...@gmail.com<mailto:igork.ine...@gmail.com>>>
Reply-To: 
"users@nifi.apache.org<mailto:users@nifi.apache.org><mailto:users@nifi.apache.org<mailto:users@nifi.apache.org>>"
 
<users@nifi.apache.org<mailto:users@nifi.apache.org><mailto:users@nifi.apache.org<mailto:users@nifi.apache.org>>>
Date: Monday, April 25, 2016 at 11:45 AM
To: 
"users@nifi.apache.org<mailto:users@nifi.apache.org><mailto:users@nifi.apache.org<mailto:users@nifi.apache.org>>"
 
<users@nifi.apache.org<mailto:users@nifi.apache.org><mailto:users@nifi.apache.org<mailto:users@nifi.apache.org>>>
Subject: Re: ReplaceText processor configuration help

Aldrin,

The overall goal is to extract some subset of attributes from tweet's JSON, 
create a new JSON and ingest it into Elasticsearch for indexing.
Hope this helps.

On Mon, Apr 25, 2016 at 11:18 AM, Aldrin Piri 
<aldrinp...@gmail.com<mailto:aldrinp...@gmail.com><mailto:aldrinp...@gmail.com<mailto:aldrinp...@gmail.com>>>
 wrote:
Igor,

Thanks for the template.  It looks like the trouble is with AttributesToJSON 
converting the attribute, which in your case, is a JSON blob, into additional 
JSON and thus the escaping to ensure nothing is lost.  Are you just trying to 
get that entity body out to a file?  If so, the AttributesToJSON is likely not 
needed and you should be able to use something like ReplaceText to write the 
attribute to the FlowFile body.  Please let us know your overall goal and we 
can see if the right mix of components already exists or if we are running into 
a path that may need some additional functionality.

Thanks!
Aldrin



On Mon, Apr 25, 2016 at 10:33 AM, Igor Kravzov 
<igork.ine...@gmail.com<mailto:igork.ine...@gmail.com><mailto:igork.ine...@gmail.com<mailto:igork.ine...@gmail.com>>>
 wrote:
Hi Aldrin,


Attached please find the template.  In this workflow I want to pull "entities" 
and "user" entries for Twitter JSON as entire structure. I only can do it if I 
set Return Type as JSON.
Subsequently I use AttributesToJSON to create a new JSON file. But returning 
values for "entities" and "user" are escaped so I had to clean these before 
converting to JSON.

Hope this helps.

On Mon, Apr 25, 2016 at 10:15 AM, Aldrin Piri 
<aldrinp...@gmail.com<mailto:aldrinp...@gmail.com><mailto:aldrinp...@gmail.com<mailto:aldrinp...@gmail.com>>>
 wrote:
Hi Igor,

That should certainly be possible.  Would you mind opening up a ticket 
(https://issues.apache.org/jira/browse/NIFI) and providing a template of your 
flow that is causing the issue?

Thanks!

On Mon, Apr 25, 2016 at 10:09 AM, Igor Kravzov 
<igork.ine...@gmail.com<mailto:igork.ine...@gmail.com><mailto:igork.ine...@gmail.com<mailto:igork.ine...@gmail.com>>>
 wrote:
Thanks Pierre. It worked. Looks like I was doing something wrong inside my 
workflow.
Would not be it feasible to have an option for EvaluateJsonPath processor to 
have an option to return escaped or unescaped JSON result?

On Mon, Apr 25, 2016 at 7:20 AM, Pierre Villard 
<pierre.villard...@gmail.com<mailto:pierre.villard...@gmail.com><mailto:pierre.villard...@gmail.com<mailto:pierre.villard...@gmail.com>>>
 wrote:
Hi Igor,

Please use ReplaceText processors.

1.
Search value : \\
Replace value : Empty string set

2.
Search value : "\{
Replace value : \{

3.
Search value : \}"
Replace value : \}

Template example attached.

HTH
Pierre


2016-04-24 20:12 GMT+02:00 Igor Kravzov 
<igork.ine...@gmail.com<mailto:igork.ine...@gmail.com><mailto:igork.ine...@gmail.com<mailto:igork.ine...@gmail.com>>>:

I am not that good in regex. What would be the proper configuration to do the 
following;

  1.  Remove b

Re: ReplaceText processor configuration help

2016-04-25 Thread Igor Kravzov 
Hi Chris,

How will it help in my situation?

On Mon, Apr 25, 2016 at 1:50 PM, McDermott, Chris Kevin (MSDU -
STaTS/StorefrontRemote) <chris.mcderm...@hpe.com> wrote:

> Igor,
>
> I think the jsonPath extension to the EL is going to be the ticket [1].  A
> patch is available if you are willing to build NiFi yourself to test it out.
>
> Cheers,
> Chris
>
> [1] https://issues.apache.org/jira/browse/NIFI-1660
>
>
> From: Igor Kravzov <igork.ine...@gmail.com<mailto:igork.ine...@gmail.com>>
> Reply-To: "users@nifi.apache.org<mailto:users@nifi.apache.org>" <
> users@nifi.apache.org<mailto:users@nifi.apache.org>>
> Date: Monday, April 25, 2016 at 11:45 AM
> To: "users@nifi.apache.org<mailto:users@nifi.apache.org>" <
> users@nifi.apache.org<mailto:users@nifi.apache.org>>
> Subject: Re: ReplaceText processor configuration help
>
> Aldrin,
>
> The overall goal is to extract some subset of attributes from tweet's
> JSON, create a new JSON and ingest it into Elasticsearch for indexing.
> Hope this helps.
>
> On Mon, Apr 25, 2016 at 11:18 AM, Aldrin Piri <aldrinp...@gmail.com
> <mailto:aldrinp...@gmail.com>> wrote:
> Igor,
>
> Thanks for the template.  It looks like the trouble is with
> AttributesToJSON converting the attribute, which in your case, is a JSON
> blob, into additional JSON and thus the escaping to ensure nothing is
> lost.  Are you just trying to get that entity body out to a file?  If so,
> the AttributesToJSON is likely not needed and you should be able to use
> something like ReplaceText to write the attribute to the FlowFile body.
> Please let us know your overall goal and we can see if the right mix of
> components already exists or if we are running into a path that may need
> some additional functionality.
>
> Thanks!
> Aldrin
>
>
>
> On Mon, Apr 25, 2016 at 10:33 AM, Igor Kravzov <igork.ine...@gmail.com
> <mailto:igork.ine...@gmail.com>> wrote:
> Hi Aldrin,
>
>
> Attached please find the template.  In this workflow I want to pull
> "entities" and "user" entries for Twitter JSON as entire structure. I only
> can do it if I set Return Type as JSON.
> Subsequently I use AttributesToJSON to create a new JSON file. But
> returning values for "entities" and "user" are escaped so I had to clean
> these before converting to JSON.
>
> Hope this helps.
>
> On Mon, Apr 25, 2016 at 10:15 AM, Aldrin Piri <aldrinp...@gmail.com
> <mailto:aldrinp...@gmail.com>> wrote:
> Hi Igor,
>
> That should certainly be possible.  Would you mind opening up a ticket (
> https://issues.apache.org/jira/browse/NIFI) and providing a template of
> your flow that is causing the issue?
>
> Thanks!
>
> On Mon, Apr 25, 2016 at 10:09 AM, Igor Kravzov <igork.ine...@gmail.com
> <mailto:igork.ine...@gmail.com>> wrote:
> Thanks Pierre. It worked. Looks like I was doing something wrong inside my
> workflow.
> Would not be it feasible to have an option for EvaluateJsonPath processor
> to have an option to return escaped or unescaped JSON result?
>
> On Mon, Apr 25, 2016 at 7:20 AM, Pierre Villard <
> pierre.villard...@gmail.com<mailto:pierre.villard...@gmail.com>> wrote:
> Hi Igor,
>
> Please use ReplaceText processors.
>
> 1.
> Search value : \\
> Replace value : Empty string set
>
> 2.
> Search value : "\{
> Replace value : \{
>
> 3.
> Search value : \}"
> Replace value : \}
>
> Template example attached.
>
> HTH
> Pierre
>
>
> 2016-04-24 20:12 GMT+02:00 Igor Kravzov <igork.ine...@gmail.com igork.ine...@gmail.com>>:
>
> I am not that good in regex. What would be the proper configuration to do
> the following;
>
>   1.  Remove backslash from text.
>   2.  Replace "{ with {
>   3.  replace }" with }
>
> Basically I need to clean escaped JSON.
>
> Like before:
>
>
>  
> "{\"hashtags\":[{\"text\":\"Apple\",\"indices\":[45,51]}],\"urls\":[{\"url\":\"\",\"expanded_url\":\"\",\"display_url\":\"
> owler.us/abdLas\<http://owler.us/abdLas%5C
> >",\"indices\":[64,87]}],\"user_mentions\":[],\"symbols\":[{\"text\":\"AAPL\",\"indices\":[88,93]}]}",
>
> after:
>
>
> {"hashtags":[{"text":"Apple","indices":[45,51]}],"urls":[{"url":"","expanded_url":"","display_url":"
> owler.us/abdLas<http://owler.us/abdLas
> >","indices":[64,87]}],"user_mentions":[],"symbols":[{"text":"AAPL","indices":[88,93]}]},
>
> Thanks in advance.
>
>
>
>
>
>
>

Re: ReplaceText processor configuration help

2016-04-25 Thread McDermott, Chris Kevin (MSDU - STaTS/StorefrontRemote) 
All, 

I could also see that it would be useful additional configuration to 
AttributesToJson to defeat the escaping where the user knows the attribute 
contains literal JSON.



On 4/25/16, 1:50 PM, "McDermott, Chris Kevin (MSDU - STaTS/StorefrontRemote)" 
<chris.mcderm...@hpe.com> wrote:

>Igor,
>
>I think the jsonPath extension to the EL is going to be the ticket [1].  A 
>patch is available if you are willing to build NiFi yourself to test it out.
>
>Cheers,
>Chris
>
>[1] https://issues.apache.org/jira/browse/NIFI-1660
>
>
>From: Igor Kravzov <igork.ine...@gmail.com<mailto:igork.ine...@gmail.com>>
>Reply-To: "users@nifi.apache.org<mailto:users@nifi.apache.org>" 
><users@nifi.apache.org<mailto:users@nifi.apache.org>>
>Date: Monday, April 25, 2016 at 11:45 AM
>To: "users@nifi.apache.org<mailto:users@nifi.apache.org>" 
><users@nifi.apache.org<mailto:users@nifi.apache.org>>
>Subject: Re: ReplaceText processor configuration help
>
>Aldrin,
>
>The overall goal is to extract some subset of attributes from tweet's JSON, 
>create a new JSON and ingest it into Elasticsearch for indexing.
>Hope this helps.
>
>On Mon, Apr 25, 2016 at 11:18 AM, Aldrin Piri 
><aldrinp...@gmail.com<mailto:aldrinp...@gmail.com>> wrote:
>Igor,
>
>Thanks for the template.  It looks like the trouble is with AttributesToJSON 
>converting the attribute, which in your case, is a JSON blob, into additional 
>JSON and thus the escaping to ensure nothing is lost.  Are you just trying to 
>get that entity body out to a file?  If so, the AttributesToJSON is likely not 
>needed and you should be able to use something like ReplaceText to write the 
>attribute to the FlowFile body.  Please let us know your overall goal and we 
>can see if the right mix of components already exists or if we are running 
>into a path that may need some additional functionality.
>
>Thanks!
>Aldrin
>
>
>
>On Mon, Apr 25, 2016 at 10:33 AM, Igor Kravzov 
><igork.ine...@gmail.com<mailto:igork.ine...@gmail.com>> wrote:
>Hi Aldrin,
>
>
>Attached please find the template.  In this workflow I want to pull "entities" 
>and "user" entries for Twitter JSON as entire structure. I only can do it if I 
>set Return Type as JSON.
>Subsequently I use AttributesToJSON to create a new JSON file. But returning 
>values for "entities" and "user" are escaped so I had to clean these before 
>converting to JSON.
>
>Hope this helps.
>
>On Mon, Apr 25, 2016 at 10:15 AM, Aldrin Piri 
><aldrinp...@gmail.com<mailto:aldrinp...@gmail.com>> wrote:
>Hi Igor,
>
>That should certainly be possible.  Would you mind opening up a ticket 
>(https://issues.apache.org/jira/browse/NIFI) and providing a template of your 
>flow that is causing the issue?
>
>Thanks!
>
>On Mon, Apr 25, 2016 at 10:09 AM, Igor Kravzov 
><igork.ine...@gmail.com<mailto:igork.ine...@gmail.com>> wrote:
>Thanks Pierre. It worked. Looks like I was doing something wrong inside my 
>workflow.
>Would not be it feasible to have an option for EvaluateJsonPath processor to 
>have an option to return escaped or unescaped JSON result?
>
>On Mon, Apr 25, 2016 at 7:20 AM, Pierre Villard 
><pierre.villard...@gmail.com<mailto:pierre.villard...@gmail.com>> wrote:
>Hi Igor,
>
>Please use ReplaceText processors.
>
>1.
>Search value : \\
>Replace value : Empty string set
>
>2.
>Search value : "\{
>Replace value : \{
>
>3.
>Search value : \}"
>Replace value : \}
>
>Template example attached.
>
>HTH
>Pierre
>
>
>2016-04-24 20:12 GMT+02:00 Igor Kravzov 
><igork.ine...@gmail.com<mailto:igork.ine...@gmail.com>>:
>
>I am not that good in regex. What would be the proper configuration to do the 
>following;
>
>  1.  Remove backslash from text.
>  2.  Replace "{ with {
>  3.  replace }" with }
>
>Basically I need to clean escaped JSON.
>
>Like before:
>
> "{\"hashtags\":[{\"text\":\"Apple\",\"indices\":[45,51]}],\"urls\":[{\"url\":\"\",\"expanded_url\":\"\",\"display_url\":\"owler.us/abdLas\<http://owler.us/abdLas%5C>",\"indices\":[64,87]}],\"user_mentions\":[],\"symbols\":[{\"text\":\"AAPL\",\"indices\":[88,93]}]}",
>
>after:
>
>{"hashtags":[{"text":"Apple","indices":[45,51]}],"urls":[{"url":"","expanded_url":"","display_url":"owler.us/abdLas<http://owler.us/abdLas>","indices":[64,87]}],"user_mentions":[],"symbols":[{"text":"AAPL","indices":[88,93]}]},
>
>Thanks in advance.
>
>
>
>
>
>

Re: ReplaceText processor configuration help

2016-04-25 Thread McDermott, Chris Kevin (MSDU - STaTS/StorefrontRemote) 
Igor,

I think the jsonPath extension to the EL is going to be the ticket [1].  A 
patch is available if you are willing to build NiFi yourself to test it out.

Cheers,
Chris

[1] https://issues.apache.org/jira/browse/NIFI-1660


From: Igor Kravzov <igork.ine...@gmail.com<mailto:igork.ine...@gmail.com>>
Reply-To: "users@nifi.apache.org<mailto:users@nifi.apache.org>" 
<users@nifi.apache.org<mailto:users@nifi.apache.org>>
Date: Monday, April 25, 2016 at 11:45 AM
To: "users@nifi.apache.org<mailto:users@nifi.apache.org>" 
<users@nifi.apache.org<mailto:users@nifi.apache.org>>
Subject: Re: ReplaceText processor configuration help

Aldrin,

The overall goal is to extract some subset of attributes from tweet's JSON, 
create a new JSON and ingest it into Elasticsearch for indexing.
Hope this helps.

On Mon, Apr 25, 2016 at 11:18 AM, Aldrin Piri 
<aldrinp...@gmail.com<mailto:aldrinp...@gmail.com>> wrote:
Igor,

Thanks for the template.  It looks like the trouble is with AttributesToJSON 
converting the attribute, which in your case, is a JSON blob, into additional 
JSON and thus the escaping to ensure nothing is lost.  Are you just trying to 
get that entity body out to a file?  If so, the AttributesToJSON is likely not 
needed and you should be able to use something like ReplaceText to write the 
attribute to the FlowFile body.  Please let us know your overall goal and we 
can see if the right mix of components already exists or if we are running into 
a path that may need some additional functionality.

Thanks!
Aldrin



On Mon, Apr 25, 2016 at 10:33 AM, Igor Kravzov 
<igork.ine...@gmail.com<mailto:igork.ine...@gmail.com>> wrote:
Hi Aldrin,


Attached please find the template.  In this workflow I want to pull "entities" 
and "user" entries for Twitter JSON as entire structure. I only can do it if I 
set Return Type as JSON.
Subsequently I use AttributesToJSON to create a new JSON file. But returning 
values for "entities" and "user" are escaped so I had to clean these before 
converting to JSON.

Hope this helps.

On Mon, Apr 25, 2016 at 10:15 AM, Aldrin Piri 
<aldrinp...@gmail.com<mailto:aldrinp...@gmail.com>> wrote:
Hi Igor,

That should certainly be possible.  Would you mind opening up a ticket 
(https://issues.apache.org/jira/browse/NIFI) and providing a template of your 
flow that is causing the issue?

Thanks!

On Mon, Apr 25, 2016 at 10:09 AM, Igor Kravzov 
<igork.ine...@gmail.com<mailto:igork.ine...@gmail.com>> wrote:
Thanks Pierre. It worked. Looks like I was doing something wrong inside my 
workflow.
Would not be it feasible to have an option for EvaluateJsonPath processor to 
have an option to return escaped or unescaped JSON result?

On Mon, Apr 25, 2016 at 7:20 AM, Pierre Villard 
<pierre.villard...@gmail.com<mailto:pierre.villard...@gmail.com>> wrote:
Hi Igor,

Please use ReplaceText processors.

1.
Search value : \\
Replace value : Empty string set

2.
Search value : "\{
Replace value : \{

3.
Search value : \}"
Replace value : \}

Template example attached.

HTH
Pierre


2016-04-24 20:12 GMT+02:00 Igor Kravzov 
<igork.ine...@gmail.com<mailto:igork.ine...@gmail.com>>:

I am not that good in regex. What would be the proper configuration to do the 
following;

  1.  Remove backslash from text.
  2.  Replace "{ with {
  3.  replace }" with }

Basically I need to clean escaped JSON.

Like before:

 
"{\"hashtags\":[{\"text\":\"Apple\",\"indices\":[45,51]}],\"urls\":[{\"url\":\"\",\"expanded_url\":\"\",\"display_url\":\"owler.us/abdLas\<http://owler.us/abdLas%5C>",\"indices\":[64,87]}],\"user_mentions\":[],\"symbols\":[{\"text\":\"AAPL\",\"indices\":[88,93]}]}",

after:

{"hashtags":[{"text":"Apple","indices":[45,51]}],"urls":[{"url":"","expanded_url":"","display_url":"owler.us/abdLas<http://owler.us/abdLas>","indices":[64,87]}],"user_mentions":[],"symbols":[{"text":"AAPL","indices":[88,93]}]},

Thanks in advance.

Re: ReplaceText processor configuration help

2016-04-25 Thread Igor Kravzov 
Aldrin,

The overall goal is to extract some subset of attributes from tweet's JSON,
create a new JSON and ingest it into Elasticsearch for indexing.
Hope this helps.

On Mon, Apr 25, 2016 at 11:18 AM, Aldrin Piri  wrote:

> Igor,
>
> Thanks for the template.  It looks like the trouble is with
> AttributesToJSON converting the attribute, which in your case, is a JSON
> blob, into additional JSON and thus the escaping to ensure nothing is
> lost.  Are you just trying to get that entity body out to a file?  If so,
> the AttributesToJSON is likely not needed and you should be able to use
> something like ReplaceText to write the attribute to the FlowFile body.
> Please let us know your overall goal and we can see if the right mix of
> components already exists or if we are running into a path that may need
> some additional functionality.
>
> Thanks!
> Aldrin
>
>
>
> On Mon, Apr 25, 2016 at 10:33 AM, Igor Kravzov 
> wrote:
>
>> Hi Aldrin,
>>
>>
>> Attached please find the template.  In this workflow I want to pull
>> "entities" and "user" entries for Twitter JSON as entire structure. I only
>> can do it if I set Return Type as JSON.
>> Subsequently I use AttributesToJSON to create a new JSON file. But
>> returning values for "entities" and "user" are escaped so I had to clean
>> these before converting to JSON.
>>
>> Hope this helps.
>>
>> On Mon, Apr 25, 2016 at 10:15 AM, Aldrin Piri 
>> wrote:
>>
>>> Hi Igor,
>>>
>>> That should certainly be possible.  Would you mind opening up a ticket (
>>> https://issues.apache.org/jira/browse/NIFI) and providing a template of
>>> your flow that is causing the issue?
>>>
>>> Thanks!
>>>
>>> On Mon, Apr 25, 2016 at 10:09 AM, Igor Kravzov 
>>> wrote:
>>>
 Thanks Pierre. It worked. Looks like I was doing something wrong inside
 my workflow.
 Would not be it feasible to have an option for EvaluateJsonPath
 processor to have an option to return escaped or unescaped JSON result?

 On Mon, Apr 25, 2016 at 7:20 AM, Pierre Villard <
 pierre.villard...@gmail.com> wrote:

> Hi Igor,
>
> Please use ReplaceText processors.
>
> 1.
> Search value : \\
> Replace value : Empty string set
>
> 2.
> Search value : "\{
> Replace value : \{
>
> 3.
> Search value : \}"
> Replace value : \}
>
> Template example attached.
>
> HTH
> Pierre
>
>
> 2016-04-24 20:12 GMT+02:00 Igor Kravzov :
>
>> I am not that good in regex. What would be the proper configuration
>> to do the following;
>>
>>1. Remove backslash from text.
>>2. Replace "{ with {
>>3. replace }" with }
>>
>> Basically I need to clean escaped JSON.
>>
>> Like before:
>>
>>  
>> "{\"hashtags\":[{\"text\":\"Apple\",\"indices\":[45,51]}],\"urls\":[{\"url\":\"\",\"expanded_url\":\"\",\"display_url\":\"owler.us/abdLas\
>>  
>> ",\"indices\":[64,87]}],\"user_mentions\":[],\"symbols\":[{\"text\":\"AAPL\",\"indices\":[88,93]}]}",
>>
>> after:
>>
>> {"hashtags":[{"text":"Apple","indices":[45,51]}],"urls":[{"url":"","expanded_url":"","display_url":"owler.us/abdLas","indices":[64,87]}],"user_mentions":[],"symbols":[{"text":"AAPL","indices":[88,93]}]},
>>
>> Thanks in advance.
>>
>
>

>>>
>>
>

Re: ReplaceText processor configuration help

2016-04-25 Thread Igor Kravzov 
Hi Aldrin,


Attached please find the template.  In this workflow I want to pull
"entities" and "user" entries for Twitter JSON as entire structure. I only
can do it if I set Return Type as JSON.
Subsequently I use AttributesToJSON to create a new JSON file. But
returning values for "entities" and "user" are escaped so I had to clean
these before converting to JSON.

Hope this helps.

On Mon, Apr 25, 2016 at 10:15 AM, Aldrin Piri  wrote:

> Hi Igor,
>
> That should certainly be possible.  Would you mind opening up a ticket (
> https://issues.apache.org/jira/browse/NIFI) and providing a template of
> your flow that is causing the issue?
>
> Thanks!
>
> On Mon, Apr 25, 2016 at 10:09 AM, Igor Kravzov 
> wrote:
>
>> Thanks Pierre. It worked. Looks like I was doing something wrong inside
>> my workflow.
>> Would not be it feasible to have an option for EvaluateJsonPath processor
>> to have an option to return escaped or unescaped JSON result?
>>
>> On Mon, Apr 25, 2016 at 7:20 AM, Pierre Villard <
>> pierre.villard...@gmail.com> wrote:
>>
>>> Hi Igor,
>>>
>>> Please use ReplaceText processors.
>>>
>>> 1.
>>> Search value : \\
>>> Replace value : Empty string set
>>>
>>> 2.
>>> Search value : "\{
>>> Replace value : \{
>>>
>>> 3.
>>> Search value : \}"
>>> Replace value : \}
>>>
>>> Template example attached.
>>>
>>> HTH
>>> Pierre
>>>
>>>
>>> 2016-04-24 20:12 GMT+02:00 Igor Kravzov :
>>>
 I am not that good in regex. What would be the proper configuration to
 do the following;

1. Remove backslash from text.
2. Replace "{ with {
3. replace }" with }

 Basically I need to clean escaped JSON.

 Like before:

  
 "{\"hashtags\":[{\"text\":\"Apple\",\"indices\":[45,51]}],\"urls\":[{\"url\":\"\",\"expanded_url\":\"\",\"display_url\":\"owler.us/abdLas\
  
 ",\"indices\":[64,87]}],\"user_mentions\":[],\"symbols\":[{\"text\":\"AAPL\",\"indices\":[88,93]}]}",

 after:

 {"hashtags":[{"text":"Apple","indices":[45,51]}],"urls":[{"url":"","expanded_url":"","display_url":"owler.us/abdLas","indices":[64,87]}],"user_mentions":[],"symbols":[{"text":"AAPL","indices":[88,93]}]},

 Thanks in advance.

>>>
>>>
>>
>
Unescape JSON result6f00bc94-aea2-43e3-90d4-f88ac4bf50df009feada-f057-4ab1-9475-b35f06f54a400 MB0009feada-f057-4ab1-9475-b35f06f54a4008f8f008-68d9-4709-b13a-cf7f519e08ccPROCESSOR0 sec1success009feada-f057-4ab1-9475-b35f06f54a40686d94af-702a-4bab-8d1b-6bdc20c4a982PROCESSOR0b25b66ee-06ac-44af-b3ae-7bee6e125c0c009feada-f057-4ab1-9475-b35f06f54a400 MB0009feada-f057-4ab1-9475-b35f06f54a40691dccb3-a2d3-4a6a-bf0c-2e8c3597c732PROCESSOR0 sec1tweet009feada-f057-4ab1-9475-b35f06f54a40ae71492d-cc5f-4f01-8d14-a066fb7ee7f6PROCESSOR091ff7125-4273-4404-8075-cb9d8d7a346d009feada-f057-4ab1-9475-b35f06f54a400 MB0009feada-f057-4ab1-9475-b35f06f54a40ae71492d-cc5f-4f01-8d14-a066fb7ee7f6PROCESSOR0 sec1matched009feada-f057-4ab1-9475-b35f06f54a40d5e07702-98fa-46fb-8d75-b2c7f66d363dPROCESSOR038172799-c232-464a-ad9e-033eed9935da009feada-f057-4ab1-9475-b35f06f54a400 MB0009feada-f057-4ab1-9475-b35f06f54a40cdfea9f4-1552-4317-83d0-ff37ee1deb0fPROCESSOR0 sec1success009feada-f057-4ab1-9475-b35f06f54a4008f8f008-68d9-4709-b13a-cf7f519e08ccPROCESSOR0e2f6ff5b-46ee-4ab1-bf1c-89f1c46c5ec1009feada-f057-4ab1-9475-b35f06f54a400 MB0009feada-f057-4ab1-9475-b35f06f54a40e009ad50-6a9d-4117-b90e-dec358818593PROCESSOR0 sec1success009feada-f057-4ab1-9475-b35f06f54a40691dccb3-a2d3-4a6a-bf0c-2e8c3597c732PROCESSOR0650469ea-05ee-4e7a-bd67-6377bb0a5b01009feada-f057-4ab1-9475-b35f06f54a400 MB0009feada-f057-4ab1-9475-b35f06f54a40686d94af-702a-4bab-8d1b-6bdc20c4a982PROCESSOR0 sec1success009feada-f057-4ab1-9475-b35f06f54a40e009ad50-6a9d-4117-b90e-dec358818593PROCESSOR029a98373-d8c8-4017-af74-b7a9749e7a9b009feada-f057-4ab1-9475-b35f06f54a400 MB0009feada-f057-4ab1-9475-b35f06f54a402ac168df-b3ea-4d15-b607-8c750de3368ePROCESSOR0 sec1failure009feada-f057-4ab1-9475-b35f06f54a40691dccb3-a2d3-4a6a-bf0c-2e8c3597c732PROCESSOR0cee749aa-464c-4d88-8b4b-a79e6f4891d6009feada-f057-4ab1-9475-b35f06f54a400 MB0009feada-f057-4ab1-9475-b35f06f54a40cb6ba844-5a24-48f0-bb8d-cc3abacd8581PROCESSOR0 sec1tweet009feada-f057-4ab1-9475-b35f06f54a40ae71492d-cc5f-4f01-8d14-a066fb7ee7f6PROCESSOR031ef786f-8c51-4aa8-93f3-56200fdc1fbb009feada-f057-4ab1-9475-b35f06f54a400 MB0009feada-f057-4ab1-9475-b35f06f54a40d5e07702-98fa-46fb-8d75-b2c7f66d363dPROCESSOR0 sec1success009feada-f057-4ab1-9475-b35f06f54a401a9af885-6ea1-491a-ad0a-c400256d13dfPROCESSOR0691dccb3-a2d3-4a6a-bf0c-2e8c3597c732009feada-f057-4ab1-9475-b35f06f54a402253.6928540618933192.65883016737013WARN1TIMER_DRIVEN1EVENT_DRIVEN0CRON_DRIVEN1TIMER_DRIVEN0 secCRON_DRIVEN* * * * * ?Attributes ListComma separated list of attributes to be included in the resulting JSON. If this value is left empty then all existing Attributes will be included. This list of attributes is

Re: ReplaceText processor configuration help

2016-04-25 Thread Igor Kravzov 
HI Aldrin,

But I don't have JIRA account.

On Mon, Apr 25, 2016 at 10:15 AM, Aldrin Piri  wrote:

> Hi Igor,
>
> That should certainly be possible.  Would you mind opening up a ticket (
> https://issues.apache.org/jira/browse/NIFI) and providing a template of
> your flow that is causing the issue?
>
> Thanks!
>
> On Mon, Apr 25, 2016 at 10:09 AM, Igor Kravzov 
> wrote:
>
>> Thanks Pierre. It worked. Looks like I was doing something wrong inside
>> my workflow.
>> Would not be it feasible to have an option for EvaluateJsonPath processor
>> to have an option to return escaped or unescaped JSON result?
>>
>> On Mon, Apr 25, 2016 at 7:20 AM, Pierre Villard <
>> pierre.villard...@gmail.com> wrote:
>>
>>> Hi Igor,
>>>
>>> Please use ReplaceText processors.
>>>
>>> 1.
>>> Search value : \\
>>> Replace value : Empty string set
>>>
>>> 2.
>>> Search value : "\{
>>> Replace value : \{
>>>
>>> 3.
>>> Search value : \}"
>>> Replace value : \}
>>>
>>> Template example attached.
>>>
>>> HTH
>>> Pierre
>>>
>>>
>>> 2016-04-24 20:12 GMT+02:00 Igor Kravzov :
>>>
 I am not that good in regex. What would be the proper configuration to
 do the following;

1. Remove backslash from text.
2. Replace "{ with {
3. replace }" with }

 Basically I need to clean escaped JSON.

 Like before:

  
 "{\"hashtags\":[{\"text\":\"Apple\",\"indices\":[45,51]}],\"urls\":[{\"url\":\"\",\"expanded_url\":\"\",\"display_url\":\"owler.us/abdLas\
  
 ",\"indices\":[64,87]}],\"user_mentions\":[],\"symbols\":[{\"text\":\"AAPL\",\"indices\":[88,93]}]}",

 after:

 {"hashtags":[{"text":"Apple","indices":[45,51]}],"urls":[{"url":"","expanded_url":"","display_url":"owler.us/abdLas","indices":[64,87]}],"user_mentions":[],"symbols":[{"text":"AAPL","indices":[88,93]}]},

 Thanks in advance.

>>>
>>>
>>
>

Re: ReplaceText processor configuration help

2016-04-25 Thread Aldrin Piri 
Hi Igor,

That should certainly be possible.  Would you mind opening up a ticket (
https://issues.apache.org/jira/browse/NIFI) and providing a template of
your flow that is causing the issue?

Thanks!

On Mon, Apr 25, 2016 at 10:09 AM, Igor Kravzov 
wrote:

> Thanks Pierre. It worked. Looks like I was doing something wrong inside my
> workflow.
> Would not be it feasible to have an option for EvaluateJsonPath processor
> to have an option to return escaped or unescaped JSON result?
>
> On Mon, Apr 25, 2016 at 7:20 AM, Pierre Villard <
> pierre.villard...@gmail.com> wrote:
>
>> Hi Igor,
>>
>> Please use ReplaceText processors.
>>
>> 1.
>> Search value : \\
>> Replace value : Empty string set
>>
>> 2.
>> Search value : "\{
>> Replace value : \{
>>
>> 3.
>> Search value : \}"
>> Replace value : \}
>>
>> Template example attached.
>>
>> HTH
>> Pierre
>>
>>
>> 2016-04-24 20:12 GMT+02:00 Igor Kravzov :
>>
>>> I am not that good in regex. What would be the proper configuration to
>>> do the following;
>>>
>>>1. Remove backslash from text.
>>>2. Replace "{ with {
>>>3. replace }" with }
>>>
>>> Basically I need to clean escaped JSON.
>>>
>>> Like before:
>>>
>>>  
>>> "{\"hashtags\":[{\"text\":\"Apple\",\"indices\":[45,51]}],\"urls\":[{\"url\":\"\",\"expanded_url\":\"\",\"display_url\":\"owler.us/abdLas\
>>>  
>>> ",\"indices\":[64,87]}],\"user_mentions\":[],\"symbols\":[{\"text\":\"AAPL\",\"indices\":[88,93]}]}",
>>>
>>> after:
>>>
>>> {"hashtags":[{"text":"Apple","indices":[45,51]}],"urls":[{"url":"","expanded_url":"","display_url":"owler.us/abdLas","indices":[64,87]}],"user_mentions":[],"symbols":[{"text":"AAPL","indices":[88,93]}]},
>>>
>>> Thanks in advance.
>>>
>>
>>
>

Re: ReplaceText processor configuration help

2016-04-25 Thread Igor Kravzov 
Thanks Pierre. It worked. Looks like I was doing something wrong inside my
workflow.
Would not be it feasible to have an option for EvaluateJsonPath processor
to have an option to return escaped or unescaped JSON result?

On Mon, Apr 25, 2016 at 7:20 AM, Pierre Villard  wrote:

> Hi Igor,
>
> Please use ReplaceText processors.
>
> 1.
> Search value : \\
> Replace value : Empty string set
>
> 2.
> Search value : "\{
> Replace value : \{
>
> 3.
> Search value : \}"
> Replace value : \}
>
> Template example attached.
>
> HTH
> Pierre
>
>
> 2016-04-24 20:12 GMT+02:00 Igor Kravzov :
>
>> I am not that good in regex. What would be the proper configuration to do
>> the following;
>>
>>1. Remove backslash from text.
>>2. Replace "{ with {
>>3. replace }" with }
>>
>> Basically I need to clean escaped JSON.
>>
>> Like before:
>>
>>  
>> "{\"hashtags\":[{\"text\":\"Apple\",\"indices\":[45,51]}],\"urls\":[{\"url\":\"\",\"expanded_url\":\"\",\"display_url\":\"owler.us/abdLas\
>>  
>> ",\"indices\":[64,87]}],\"user_mentions\":[],\"symbols\":[{\"text\":\"AAPL\",\"indices\":[88,93]}]}",
>>
>> after:
>>
>> {"hashtags":[{"text":"Apple","indices":[45,51]}],"urls":[{"url":"","expanded_url":"","display_url":"owler.us/abdLas","indices":[64,87]}],"user_mentions":[],"symbols":[{"text":"AAPL","indices":[88,93]}]},
>>
>> Thanks in advance.
>>
>
>

ReplaceText processor configuration help

2016-04-24 Thread Igor Kravzov 
I am not that good in regex. What would be the proper configuration to do
the following;

   1. Remove backslash from text.
   2. Replace "{ with {
   3. replace }" with }

Basically I need to clean escaped JSON.

Like before:

 
"{\"hashtags\":[{\"text\":\"Apple\",\"indices\":[45,51]}],\"urls\":[{\"url\":\"\",\"expanded_url\":\"\",\"display_url\":\"owler.us/abdLas\",\"indices\":[64,87]}],\"user_mentions\":[],\"symbols\":[{\"text\":\"AAPL\",\"indices\":[88,93]}]}",

after:

{"hashtags":[{"text":"Apple","indices":[45,51]}],"urls":[{"url":"","expanded_url":"","display_url":"owler.us/abdLas","indices":[64,87]}],"user_mentions":[],"symbols":[{"text":"AAPL","indices":[88,93]}]},

Thanks in advance.