I was able to construct the Json with right data type output from
ExecuteScript and sending to Kafka directly. The problem i am seeing is if
i send one record to kafka, kafka processor is writing the message again
and again and not ending the loop. How can i send exactly once message? Any
help.
Here is what i am doing in my script:
import simplejson as json
from org.apache.nifi.processor.io import StreamCallback
from user_agents import parse
def num(s):
try:
return int(s)
except ValueError:
try:
return float(s)
except ValueError:
raise ValueError('argument is not a string of number')
class PyStreamCallback(StreamCallback):
def __init__(self):
pass
def process(self, inputStream, outputStream):
obj = {'browser':
str(parse(flowFile.getAttribute('useragent')).browser.family),
'browser_version':
str(parse(flowFile.getAttribute('useragent')).browser.version_string),
'os':
str(parse(flowFile.getAttribute('useragent')).os.family),
'os_version':
str(parse(flowFile.getAttribute('useragent')).os.version_string),
'client_ip': flowFile.getAttribute('clientip')}
if flowFile.getAttribute('http.param.t_resp') and
flowFile.getAttribute('http.param.t_page') and
flowFile.getAttribute('http.param.t_done'):
obj['rt_firstbyte'] =
num(flowFile.getAttribute('http.param.t_resp'))
obj['rt_lastbyte'] =
num(flowFile.getAttribute('http.param.t_page'))
obj['rt_loadtime'] =
num(flowFile.getAttribute('http.param.t_done'))
outputStream.write(bytearray(json.dumps(obj,
indent=4).encode('utf-8')))
flowFile = session.get()
if (flowFile != None):
flowFile = session.write(flowFile, PyStreamCallback())
session.transfer(flowFile, REL_SUCCESS)
Thanks
On Tue, Mar 29, 2016 at 2:30 AM, Conrad Crampton <
[email protected]> wrote:
> Hi,
> Depending on the final destination of the data (json) you could use the
> JsonToAvro -> ConvertAvroSchema -> AvroToJson, with the ConvertAvroSchema
> doing the type conversion. I had to do this as I came across this behaviour
> previously. I use the Avro directly (after the conversion) as that was my
> final data format requirement, but I don’t see any reason if you want Json
> back that this wouldn’t work. I haven’t tried this by the way, but the type
> conversion certainly works for the final attributes in the Avro documents.
> Conrad
>
> From: Madhukar Thota <[email protected]>
> Reply-To: "[email protected]" <[email protected]>
> Date: Friday, 25 March 2016 at 14:01
> To: "[email protected]" <[email protected]>
> Subject: Re: String conversion to Int, float double
>
> Any Other ways to achieve this?
>
> On Thu, Mar 24, 2016 at 4:48 PM, Bryan Bende <[email protected]> wrote:
>
>> I think the problem is that all attributes are actually Strings
>> internally, even after calling toNumber() that is only temporary while the
>> expression language is executing.
>>
>> So by the time it gets to AttributesToJson it doesn't have any
>> information about the type of each attribute and they all end up as
>> Strings. I think we would have to come up with a way to pass some type
>> information along to AttributesToJson in order to get something other than
>> Strings.
>>
>> -Bryan
>>
>>
>> On Thu, Mar 24, 2016 at 3:30 PM, Madhukar Thota <[email protected]
>> > wrote:
>>
>>> Hi i am trying to convert string value to integer in UpdateAtrributes
>>> using toNumber like this
>>>
>>>
>>> ${http.param.t_resp:toNumber()} where http.param.t_resp = "132"
>>>
>>> but when the fileattribute pushed to Attributetojson processor , i am
>>> stilling seeing it as string. Am i am doing something wrong? and also how
>>> can i convert string to float?
>>>
>>>
>>>
>>>
>>>
>>
>
>
> ***This email originated outside SecureData***
>
> Click here <https://www.mailcontrol.com/sr/MZbqvYs5QwJvpeaetUwhCQ==> to
> report this email as spam.
>
>
> SecureData, combating cyber threats
>
> ------------------------------
>
> The information contained in this message or any of its attachments may be
> privileged and confidential and intended for the exclusive use of the
> intended recipient. If you are not the intended recipient any disclosure,
> reproduction, distribution or other dissemination or use of this
> communications is strictly prohibited. The views expressed in this email
> are those of the individual and not necessarily of SecureData Europe Ltd.
> Any prices quoted are only valid if followed up by a formal written quote.
>
> SecureData Europe Limited. Registered in England & Wales 04365896.
> Registered Address: SecureData House, Hermitage Court, Hermitage Lane,
> Maidstone, Kent, ME16 9NT
>
