Re: Getting a notification once a request is finished

[KhajaAsmath Mohammed]

Hi Vibhath,

After a lot of research, I did this way.

Let’s say, you have 50 tables that needs to be processed. I will have status as 
loading, success or fail foe each file processing. If anyone of the flow file 
is failed, then process is considered as failed . I will keep checking the 
status using db query for these tables and Mark as complete based on db query 
status

Thanks,
Asmath

> On May 19, 2021, at 6:32 AM, Vibhath Ileperuma  
> wrote:
> 
> 
> Hi all,
> 
> I'm developing a NIFI flow to extract data in a database and write into 
> files.This flow starts with a ListenHttp processor. A python script sends a 
> list of table names required to be extracted to the port ListenHttp processor 
> is listening to. NIFI flow has been developed to iterate through that list 
> and extract the data in given tables.
> 
> I would like to know whether there is a way to get a notification to the 
> python script once all the tables are extracted. 
> If it is not possible, is it possible to log a message informing all the 
> tables have been extracted.
> 
> Thank You.
> Best Regards,
> Vibhath.

NIFI help on getting JSON data

[KhajaAsmath Mohammed]

Hi,

I have a below flow but I will be losing the properties that I need. ANy
help on how to get this?

Generate flow flow -- Has JSON data that I will be using as properties

GenerateFlowFile --> ExecuteSQLRecord(JSON Record Writer) (JSON of Generate
flow file is lost )--> Retreive BatchNumber from JSON(ExecuteSQLRecord
JSON) -->  Have to Get JSON of Generate flow file

How to get this?

Thanks,
Khaja

Nifi dependency management- check if all db tables in batch are loaded

[KhajaAsmath Mohammed]

Hi,

I have a use case where I need to update the source system with success or fail 
status after processing in nifi. Any suggestions. I have one in my mind by 
maintaining audit table of processed tables and continuously check if that 
table has any error or loading status. I don’t like this approach but checking 
if there is any other alternative

Let’s say, I read source database which has 300+ tables and it keeps changing. 
Make sure that all 300 + tables data is loaded based on batch number by nifi 
and return the status back to source system so that nifi can pick up next batch 
based on previous state of record( success - process , fail - don’t process ). 



Thanks,
Khaja

Re: Multiple OR conditions route on attribute

[KhajaAsmath Mohammed]

Thanks Chris.

Thanks,
Khaja

> On Apr 24, 2021, at 12:47 PM, Chris Sampson  wrote:
> 
> 
> Yes, you can chain together multiple Expression Language OR conditions[1] in 
> the same processor property.
> 
> For example 
> "${filename:toLower():equals(${filename}):or(${filename:length():equals(5)}):or(${filename:endsWith('.txt')})}"
> 
> https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#or
> 
> Cheers,
> 
> Chris Sampson
> 
>> On Sat, 24 Apr 2021, 17:12 KhajaAsmath Mohammed,  
>> wrote:
>> Hi,
>> 
>> I have a use case where I need to route based on multiple or conditions.
>> 
>> Instead of writing multiple separate conditions in route on attribute, I 
>> want to add all those attributes with or conditions . Is this possible ?
>> 
>> Thanks,
>> Khaja

Multiple OR conditions route on attribute

[KhajaAsmath Mohammed]

Hi,

I have a use case where I need to route based on multiple or conditions.

Instead of writing multiple separate conditions in route on attribute, I want 
to add all those attributes with or conditions . Is this possible ?

Thanks,
Khaja

Re: ExecuteSQL - Ignore zero records file

[KhajaAsmath Mohammed]

Thanks Chris.

On Fri, Apr 23, 2021 at 4:21 PM Chris Sampson 
wrote:

> You could use a RouteOnAttribute[1] processor to look for FlowFiles with
> an executesql.row.count attribute with a value of 0, then remove them
> from the flow.
>
> [1]
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.RouteOnAttribute/index.html
>
>
> ---
> *Chris Sampson*
> IT Consultant
> chris.samp...@naimuri.com
> <https://www.naimuri.com/>
>
>
> On Fri, 23 Apr 2021 at 22:16, KhajaAsmath Mohammed <
> mdkhajaasm...@gmail.com> wrote:
>
>> Hi,
>>
>> I have a requirement where I need to ignore the flow files that have zero
>> records from executesql processor.
>>
>> Process only the files which have results from executesql.  any
>> suggestions please?
>>
>> Thanks,
>> Khaja
>>
>

ExecuteSQL - Ignore zero records file

[KhajaAsmath Mohammed]

Hi,

I have a requirement where I need to ignore the flow files that have zero
records from executesql processor.

Process only the files which have results from executesql.  any suggestions
please?

Thanks,
Khaja

Avro writer - execute sql

[KhajaAsmath Mohammed]

Hi,

I am having an issue with avro writer that gets automatically populated with 
different data types. 

In my case, big decimal in oracle is converted to double with avro writer and 
logic data type is causing issue when loading data back to other database . 
Values get changed .

In order to resolve this, I want to change all data types to String except time 
stamp with avro reader. Any suggestions ?

Thanks,
Khaja

Sent from my iPhone

Upsert logic with SQLServer - NIFI Putdatabase

[KhajaAsmath Mohammed]

Hi,

I am looking for some help on how to deal with upserts/updates on SQL
Server with NIFI.

I get a flow file where the records are updated. Primary key are already
defined on the table.

I don't want to try the logic of delete and insert, is there a way to
handle upsert automatically with this approach? This can be present for any
table and should be dynamic.

Thanks,
Asmath

AvroWriter - Executesql - Not retaining the datatype

[KhajaAsmath Mohammed]

Hi,

I have an issue with oracle driver where the executesql processor is
converting all the datatypes to String. AvroWriter used internally by
executesql is writing all the fields as string type.

I want to limit the timestamp field from oracle to 23 characters.

2021-02-02 09:01:24.847756 to 2021-02-02 09:01:24.847 ?

Any suggestions on how to achieve this?

Column names can be dynamic so I don't want to add a custom schema.

Thanks,
Asmath

Re: CSVRecordSetWriter - Date format

[KhajaAsmath Mohammed]

Hi Peter,

It resolved the timestamp issue but I am still getting more characters in
timestamp. I only need to get maximum of 23 characters. any suggetsions on
how to trim this data from  2021-02-02 09:01:24.847756 to  2021-02-02
09:01:24.8 ? I have yyy-MM-dd hh:mm:ss.S in timestamp field for CSVwriter.

EMPLID|EMPL_RCD|EFFDT|EFFSEQ|PER_ORG|DEPTID|JOBCODE|POSITION_NBR|SUPERVISOR_ID|HR_STATUS|POSITION_OVERRIDE|POSN_CHANGE_RECORD|EMPL_STATUS|ACTION|ACTION_DT|ACTION_REASON|LOCATION|TAX_LOCATION_CD|JOB_ENTRY_DT|DEPT_ENTRY_DT|POSITION_ENTRY_DT|SHIFT|REG_TEMP|FULL_PART_TIME|COMPANY|PAYGROUP|BAS_GROUP_ID|ELIG_CONFIG1|ELIG_CONFIG2|ELIG_CONFIG3|ELIG_CONFIG4|ELIG_CONFIG5|ELIG_CONFIG6|ELIG_CONFIG7|ELIG_CONFIG8|ELIG_CONFIG9|BEN_STATUS|BAS_ACTION|COBRA_ACTION|EMPL_TYPE|HOLIDAY_SCHEDULE|STD_HOURS|STD_HRS_FREQUENCY|OFFICER_CD|EMPL_CLASS|SAL_ADMIN_PLAN|GRADE|GRADE_ENTRY_DT|STEP|STEP_ENTRY_DT|EARNS_DIST_TYPE|COMP_FREQUENCY|COMPRATE|CHANGE_AMT|CHANGE_PCT|ANNUAL_RT|MONTHLY_RT|DAILY_RT|HOURLY_RT|ANNL_BENEF_BASE_RT|SHIFT_RT|SHIFT_FACTOR|CURRENCY_CD|BUSINESS_UNIT|SETID_DEPT|SETID_JOBCODE|SETID_LOCATION|SETID_SALARY|SETID_EMPL_CLASS|REG_REGION|DIRECTLY_TIPPED|FLSA_STATUS|EEO_CLASS|UNION_CD|BARG_UNIT|UNION_SENIORITY_DT|GP_PAYGROUP|GP_DFLT_ELIG_GRP|GP_ELIG_GRP|GP_DFLT_CURRTTYP|CUR_RT_TYPE|GP_DFLT_EXRTDT|GP_ASOF_DT_EXG_RT|CLASS_INDC|ENCUMB_OVERRIDE|FICA_STATUS_EE|FTE|PRORATE_CNT_AMT|PAY_SYSTEM_FLG|LUMP_SUM_PAY|CONTRACT_NUM|JOB_INDICATOR|BENEFIT_SYSTEM|WORK_DAY_HOURS|REPORTS_TO|JOB_DATA_SRC_CD|ESTABID|SUPV_LVL_ID|SETID_SUPV_LVL|ABSENCE_SYSTEM_CD|POI_TYPE|HIRE_DT|LAST_HIRE_DT|TERMINATION_DT|ASGN_START_DT|LST_ASGN_START_DT|ASGN_END_DT|LDW_OVR|LAST_DATE_WORKED|EXPECTED_RETURN_DT|EXPECTED_END_DATE|AUTO_END_FLG|LASTUPDDTTM|LASTUPDOPRID|CR_BT_DTM|CR_BT_NBR|UPD_BT_DTM|UPD_BT_NBR|ODS_VRSN_NBR|DML_IND
10474406|0|2020-11-01 00:00:00.0|0|CWR|000286|CWR015|" "|"
"|A|N|N|A|ADD|2021-02-02 00:00:00.0|ADD|VCAA|CA-DEFAULT|2020-11-01
00:00:00.0|2020-11-01 00:00:00.0||N|R|F|UCS|CWR|" "|" "|" "|" "|" "|" "|"
"|" "|" "|" "|A|" "|" "|S|" "|40|W|N|13|" "|"
"||0||N|M|0|0|0|0|0|0|0|0|0|0|USD|SDCMP|SDCMP|UCSHR|SDCMP|"
"|STD|USA|N|X|N|" "|" "||" "|Y|" "|Y|" "|Y|" "|C|N|N|0|N|OT|N|" "|P|OT|0|"
"|" "|UCSD|" "|" "|OT|" "|2020-11-01 00:00:00.0|2020-11-01
00:00:00.0||2020-11-01 00:00:00.0|2020-11-01 00:00:00.0||N|||2021-10-31
00:00:00.0|N|*2021-02-02 09:01:24.847756*|UC_BATCH|2021-02-02
09:01:25.0|2625|2021-02-02 09:01:25.0|2625|211|I

Thanks,
Asmath

On Tue, Mar 2, 2021 at 10:52 AM KhajaAsmath Mohammed <
mdkhajaasm...@gmail.com> wrote:

> Thanks Peter, this worked. I have now issue with csv reader and loading
> that into SQL server. Will send out seperate email for that.
>
> On Tue, Mar 2, 2021 at 9:37 AM Peter Turcsanyi 
> wrote:
>
>> Hi Asmath,
>>
>> If I understand your flow correctly, you have ExecuteSQLRecord
>> (or QueryDatabaseTableRecord) with CSVRecordSetWriter and the expected
>> output format for Oracle DATE type would be DD-MM- in the CSV.
>>
>> To achieve this, you'll need to configure CSVRecordSetWriter's "Timestamp
>> Format" property as DD-MM- and also set the processor's "Use Avro
>> Logical Types" property to true. The output format of your timestamp
>> columns will not be affected by this.
>>
>> Explanation / technical background info on date/timestamp handling in
>> NiFi in case of Oracle DB:
>> For DATE data type, the Oracle driver returns a java.sql.Timestamp object
>> by default. That's why the user needs to configure "Timestamp Format"
>> instead of "Date Format".
>> For TIMESTAMP data type, the Oracle driver returns an
>> oracle.sql.TIMESTAMP object by default. It is not a regular java timestamp
>> and no formatting will be applied to it. It will be added in its default
>> format in the CSV (eg.: 2021-03-02 16:30:41.0).
>> Other RDBMSs (typically) return DATE/TIMESTAMP values as java.sql.Date
>> and java.sql.Timestamp respectively and the proper format (from "Date
>> Format" vs "Timestamp Format" properties) can be applied to them in the
>> NiFi flow.
>>
>> Hope it helps. Let us know if you are able to configure your flow to
>> produce the date format you want.
>>
>> Best,
>> Peter
>>
>> On Mon, Mar 1, 2021 at 7:35 PM KhajaAsmath Mohammed <
>> mdkhajaasm...@gmail.com> wrote:
>>
>>> Hi Peter,
>>>
>>> This issue is occuring for the DATE field on oracle. There are other
>>> fields which are timestampe in oracle

CSVReader - Putdatabase - Timestamp issue with sqlserver

[KhajaAsmath Mohammed]

Hi.

Is there a way to convert timestamp column automatically to 21 characters
for any columns, Columns can be dynamic, name can change frequently.

  2020-05-29 23:08:44.541744 should be changed to  2020-05-29 23:08:44.5
for sql server . Sql server can have maximum of 23 characters. any
suggestions please ?

[image: image.png]


EMPLID|EMPL_RCD|EFFDT|EFFSEQ|EMPL_CLASS|JOBCODE|LOCATION|UC_RETMT_SFTY_CD|UC_SUMMER_SAL|EXPECTED_END_DATE|UNION_CD|BUSINESS_UNIT|ACTION|ACTION_REASON|UC_DAYS_INACTV|UC_ROLL_12MONHRS|UC_FTE_BEGIN_DT_1|UC_FTE_BEGIN_DT_05|UC_UCRP_FTE_DT_1|UC_UCRP_FTE_DT_05|UC_TOT_FTE|UC_TOT_FTE_UCRP|UC_AVG_PAID_TIMEHR|UC_175HR_ELIG_DT|UC_175HRS_1_M_FLG|UC_175HRS_2_M_FLG|UC_175HRS_1MTH_END|UC_175HRS_2MTH_END|ELIG_BGN_DT|UC_EBEUC|UC_BN_PRI_JOB_OVRR|UC_BN_PRJB_OVR_EDT|ELIG_CONFIG1|ELIG_CONFIG2|ELIG_CONFIG3|ELIG_CONFIG4|ELIG_CONFIG5|ELIG_CONFIG6|ELIG_CONFIG7|ELIG_CONFIG9|UC_STS_QUAL_CD|UC_WELFR_OVR_ENDDT|UC_HD_OVR_ENDDT|ELIG_DATE|UC_BEN_ELIGIBLE|UC_ELIG_RSN|SETID|UC_APPT_DURATION|UC_APPT_DUR_UCRP|UC_ELIG_REASON|OVERRIDE_DT|UC_ABBR_PROC_IND|UC_UCRP_ELIG|UC_UCRP_ELIG_DT|UC_UCRP_ELIG_LSTDT|UC_ELECT_ELG_DT|UC_UCRP_OVR_DT|UC_OVR_FACULTY|UC_OVR_RET_ELIG|FLAG3|PROCESSED_FLAG|UC_BN_CLUSTER|UC_BN_GROUP|ELIG_CONFIG8|UC_FTE|UC_LAST_PR_ELIG_DT|UC_BBE_STAB_END_DT|UC_ACA_STAB_END_DT|UC_OVR_HEALTH|UC_OVR_WELFARE|UC_OVR_SUM_SAL|UC_OVR_EBEUC|UC_OVR_RET_REH|UC_OVR_EED|UC_BN_BED_FRC|UC_BN_AWHS_VAL|UC_BN_DUR_FTE|LASTUPDOPRID|LASTUPDDTTM|CR_BT_DTM|CR_BT_NBR|UPD_BT_DTM|UPD_BT_NBR|ODS_VRSN_NBR|DML_IND
10363485|0|2020-05-01 00:00:00.0|2|24|002732|VCHSS|" "|" "|2021-06-30
00:00:00.0|99|SDCMP|POS|RTC|0|0|0|0|0||" "|" "|||2016-06-23
00:00:00.0|99|0||N|N|" "|" "|99C|" "|" "|DCPSH|" "|||2016-06-23
00:00:00.0|N|CNV|UCSHR|0|0|" "||" "|" "|N|N|N|" "|" "|" "|" "|0"
"|" "|" "|" "|" "|" "|" "|0|0|10002576|*2020-05-29 23:08:44.541744*|*2020-05-29
23:08:44.541744*|2623|2021-02-01 03:30:00.0|2623|211|I


Thanks,
Asmath

Re: CSVRecordSetWriter - Date format

[KhajaAsmath Mohammed]

Thanks Peter, this worked. I have now issue with csv reader and loading
that into SQL server. Will send out seperate email for that.

On Tue, Mar 2, 2021 at 9:37 AM Peter Turcsanyi  wrote:

> Hi Asmath,
>
> If I understand your flow correctly, you have ExecuteSQLRecord
> (or QueryDatabaseTableRecord) with CSVRecordSetWriter and the expected
> output format for Oracle DATE type would be DD-MM- in the CSV.
>
> To achieve this, you'll need to configure CSVRecordSetWriter's "Timestamp
> Format" property as DD-MM- and also set the processor's "Use Avro
> Logical Types" property to true. The output format of your timestamp
> columns will not be affected by this.
>
> Explanation / technical background info on date/timestamp handling in NiFi
> in case of Oracle DB:
> For DATE data type, the Oracle driver returns a java.sql.Timestamp object
> by default. That's why the user needs to configure "Timestamp Format"
> instead of "Date Format".
> For TIMESTAMP data type, the Oracle driver returns an oracle.sql.TIMESTAMP
> object by default. It is not a regular java timestamp and no formatting
> will be applied to it. It will be added in its default format in the CSV
> (eg.: 2021-03-02 16:30:41.0).
> Other RDBMSs (typically) return DATE/TIMESTAMP values as java.sql.Date
> and java.sql.Timestamp respectively and the proper format (from "Date
> Format" vs "Timestamp Format" properties) can be applied to them in the
> NiFi flow.
>
> Hope it helps. Let us know if you are able to configure your flow to
> produce the date format you want.
>
> Best,
> Peter
>
> On Mon, Mar 1, 2021 at 7:35 PM KhajaAsmath Mohammed <
> mdkhajaasm...@gmail.com> wrote:
>
>> Hi Peter,
>>
>> This issue is occuring for the DATE field on oracle. There are other
>> fields which are timestampe in oracle for the same table.
>>
>> Sample Data in Oracle: Date Field type --> 01-NOV-20
>>
>> May I know the field name and format that I need to use in
>> csvrecordwriter now?
>>
>>
>> Thanks,
>> Asmath
>>
>> On Mon, Mar 1, 2021 at 12:20 PM Peter Turcsanyi 
>> wrote:
>>
>>> Hi Asmath,
>>>
>>> I would try to specify the format in the Timestamp Format property.
>>> Oracle's DATE type can contain HH:mm:ss part and I believe it is
>>> converted to a Timestamp and the Timestamp Format is applied.
>>>
>>> Regards,
>>> Peter
>>>
>>> On Mon, Mar 1, 2021 at 6:42 PM KhajaAsmath Mohammed <
>>> mdkhajaasm...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I have an issue where the csvrecordwriter is automatically converting
>>>> data from date to number. how to resolve this?
>>>>
>>>> any suggestions to change this?
>>>>
>>>> [image: image.png]
>>>>
>>>> Source : Oracle with Date format
>>>>
>>>> [image: image.png]
>>>>
>>>> Target: Sql-server into Date format
>>>>
>>>>
>>>> Thanks,
>>>> Asmath
>>>>
>>>

Re: CSVRecordSetWriter - Date format

[KhajaAsmath Mohammed]

Hi Peter,

This issue is occuring for the DATE field on oracle. There are other fields
which are timestampe in oracle for the same table.

Sample Data in Oracle: Date Field type --> 01-NOV-20

May I know the field name and format that I need to use in csvrecordwriter
now?


Thanks,
Asmath

On Mon, Mar 1, 2021 at 12:20 PM Peter Turcsanyi 
wrote:

> Hi Asmath,
>
> I would try to specify the format in the Timestamp Format property.
> Oracle's DATE type can contain HH:mm:ss part and I believe it is converted
> to a Timestamp and the Timestamp Format is applied.
>
> Regards,
> Peter
>
> On Mon, Mar 1, 2021 at 6:42 PM KhajaAsmath Mohammed <
> mdkhajaasm...@gmail.com> wrote:
>
>> Hi,
>>
>> I have an issue where the csvrecordwriter is automatically converting
>> data from date to number. how to resolve this?
>>
>> any suggestions to change this?
>>
>> [image: image.png]
>>
>> Source : Oracle with Date format
>>
>> [image: image.png]
>>
>> Target: Sql-server into Date format
>>
>>
>> Thanks,
>> Asmath
>>
>

CSVRecordSetWriter - Date format

[KhajaAsmath Mohammed]

Hi,

I have an issue where the csvrecordwriter is automatically converting data
from date to number. how to resolve this?

any suggestions to change this?

[image: image.png]

Source : Oracle with Date format

[image: image.png]

Target: Sql-server into Date format


Thanks,
Asmath

Incremental Fetch in NIFI

[KhajaAsmath Mohammed]

Hi,

I have a use case where I need to do incremental fetch on the oracle
tables. Is there a easy way to do this? I saw some posts about
querydatabase table. want to check if there is any efficient way to do this?

Thanks,
Khaja

Re: NIFI - Topic pattern names

[KhajaAsmath Mohammed]

Please ignore I was able to figure out . Was missing . before *

On Tue, Feb 23, 2021 at 4:19 PM KhajaAsmath Mohammed <
mdkhajaasm...@gmail.com> wrote:

> Hi Joe,
>
> I have used the below but it is not reading data from topics.  I want to
> read data from all the topics which starts with LOCATION_RI
>
> [image: image.png]
>
> On Tue, Feb 23, 2021 at 2:58 PM Joe Witt  wrote:
>
>> See
>> http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-kafka-2-6-nar/1.13.0/org.apache.nifi.processors.kafka.pubsub.ConsumeKafka_2_6/index.html
>> Look at 'Topic Name(s)' and 'Topic Name Format' with a provided naming
>> pattern.
>>
>> Thanks
>>
>> On Tue, Feb 23, 2021 at 1:55 PM KhajaAsmath Mohammed
>>  wrote:
>> >
>> > Hi,
>> >
>> > I am planning to consume multiple topics from kafka that status with a
>> particular word. May I know how to use this?
>> >
>> > I dont want to enter all the names.
>> >
>> > Thanks,
>> > Asmath
>>
>

Re: NIFI - Topic pattern names

[KhajaAsmath Mohammed]

Hi Joe,

I have used the below but it is not reading data from topics.  I want to
read data from all the topics which starts with LOCATION_RI

[image: image.png]

On Tue, Feb 23, 2021 at 2:58 PM Joe Witt  wrote:

> See
> http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-kafka-2-6-nar/1.13.0/org.apache.nifi.processors.kafka.pubsub.ConsumeKafka_2_6/index.html
> Look at 'Topic Name(s)' and 'Topic Name Format' with a provided naming
> pattern.
>
> Thanks
>
> On Tue, Feb 23, 2021 at 1:55 PM KhajaAsmath Mohammed
>  wrote:
> >
> > Hi,
> >
> > I am planning to consume multiple topics from kafka that status with a
> particular word. May I know how to use this?
> >
> > I dont want to enter all the names.
> >
> > Thanks,
> > Asmath
>

NIFI - Topic pattern names

[KhajaAsmath Mohammed]

Hi,

I am planning to consume multiple topics from kafka that status with a
particular word. May I know how to use this?

I dont want to enter all the names.

Thanks,
Asmath

NIFI - Performance issues

[KhajaAsmath Mohammed]

Hi,

I am looking for some information on how to improve performance of our
flows. Any suggestions?

1. how to troubleshoot which processor in the nifi is causing an issue?
2. What is the best way to increase the threads or tasks for a particular
processor?
3. How to debug the flow when there is an issue on a particular processor
group?

Thanks,
Asmath

Re: Week of year - query record processor

[KhajaAsmath Mohammed]

This didn't work. Am I missing something?

EVENT_TIMESTAMP
2020-07-10 23:46:14.805


${EVENT_TIMESTAMP.value:toDate('-MM-dd HH:mm:ss'):format('w')}


Thanks,
Mohd

On Tue, Feb 2, 2021 at 8:29 AM Mark Payne  wrote:

> You should be able to use something like:
>
> ${field.value:toDate(‘-MM-dd HH:mm:ss’):format(‘w’)}
>
> Thanks
> -Mark
>
> > On Feb 1, 2021, at 9:13 PM, KhajaAsmath Mohammed <
> mdkhajaasm...@gmail.com> wrote:
> >
> > Hi,
> >
> > Is there a way to get weekofyear from the query record processor. Any
> syntax would really help.
> >
> > Time stamp: 2021-02-01 12:12:12
> >
> > Thanks,
> > Khaja
> >
> > Sent from my iPhone
>
>

Week of year - query record processor

[KhajaAsmath Mohammed]

Hi,

Is there a way to get weekofyear from the query record processor. Any syntax 
would really help.

Time stamp: 2021-02-01 12:12:12

Thanks,
Khaja

Sent from my iPhone

Re: Special Characters - Table name - Putddatabase record

[KhajaAsmath Mohammed]

Creating a synonym on the table name in the database and calling it with
put database helped it.

Thanks
Asmath

On Wed, Jan 20, 2021 at 1:38 PM KhajaAsmath Mohammed <
mdkhajaasm...@gmail.com> wrote:

> Hi,
>
> Yes, it is case sensitive in SAP HANA. Table name is the same as it shows
> in the error message. Should have another alternative but nothing works in
> this case for me.
>
> Thanks,
> Asmath
>
> On Wed, Jan 20, 2021 at 9:53 AM Peter Turcsanyi 
> wrote:
>
>> Hi Asmath,
>>
>> Scenarios 2 and 3 should work.
>> When using quotes, the table / columns names are typically case sensitive
>> in the database (though I don't know SAP).
>> Are you sure your table exists in the database with the exact same name
>> (in terms of lower/upper case)?
>>
>> Regards,
>> Peter
>>
>

Re: Special Characters - Table name - Putddatabase record

[KhajaAsmath Mohammed]

Hi,

Yes, it is case sensitive in SAP HANA. Table name is the same as it shows
in the error message. Should have another alternative but nothing works in
this case for me.

Thanks,
Asmath

On Wed, Jan 20, 2021 at 9:53 AM Peter Turcsanyi 
wrote:

> Hi Asmath,
>
> Scenarios 2 and 3 should work.
> When using quotes, the table / columns names are typically case sensitive
> in the database (though I don't know SAP).
> Are you sure your table exists in the database with the exact same name
> (in terms of lower/upper case)?
>
> Regards,
> Peter
>

Special Characters - Table name - Putddatabase record

[KhajaAsmath Mohammed]

Hi,

I am having hard time inserting data into the table which has special
characters in the table name. I am able to test the same connection without
special characters in the table name. Any suggestions on how to resolve
this?

I tried quotes too for the table name but it didn't help me.  Any
suggestions please?
[image: image.png]

Thanks,
Asmath

NIFI - Regex replace Help

[KhajaAsmath Mohammed]

Hi,

I have e replacetext processor with below settings. I want to replace them
with only one processor and use or condition in the search query. any help
on how to do this?

[image: image.png]

[image: image.png]

Thanks,
Asmath

Re: NIFI Groovy Script - Filter file names and get count

[KhajaAsmath Mohammed]

Thanks Mike. 

Sent from my iPhone

> On Oct 30, 2020, at 5:10 AM, Mike Thomsen  wrote:
> 
> You need to use the listFiles() that has a FilenameFilter interface in it:
> 
> https://docs.oracle.com/javase/8/docs/api/java/io/File.html#listFiles-java.io.FilenameFilter-
> 
>> On Thu, Oct 29, 2020 at 5:12 PM KhajaAsmath Mohammed
>>  wrote:
>> 
>> Hi,
>> 
>> I have a requirement where I need to get the file count from the path using 
>> the groovy script.
>> 
>> I came up with the below but unable to filter and count only txt files . Any 
>> suggestions please?
>> 
>> import org.apache.commons.io.IOUtils
>> import java.nio.charset.*;
>> import java.io.*;
>> 
>> def flowFile = session.get()
>> if(!flowFile) return
>> 
>> 
>> def eahpath = flowFile.getAttribute("eahpath")
>> def count = new File(eahpath).listFiles().size();// I need to filter 
>> only txt files and get count
>> flowFile=session.putAttribute(flowFile,"eahfilecount",count+"");
>> def fail = false
>> if(fail){
>> session.transfer(flowFile, REL_FAILURE)
>> fail = false
>> } else {
>> session.transfer(flowFile, REL_SUCCESS)
>> }
>> 
>> Thanks,
>> Asmath

NIFI Groovy Script - Filter file names and get count

[KhajaAsmath Mohammed]

Hi,

I have a requirement where I need to get the file count from the path using
the groovy script.

I came up with the below but unable to filter and count only txt files .
Any suggestions please?

import org.apache.commons.io.IOUtils
import java.nio.charset.*;
import java.io.*;

def flowFile = session.get()
if(!flowFile) return


def eahpath = flowFile.getAttribute("eahpath")
*def count = new File(eahpath).listFiles().size();// I need to filter
only txt files and get count*
flowFile=session.putAttribute(flowFile,"eahfilecount",count+"");
def fail = false
if(fail){
session.transfer(flowFile, REL_FAILURE)
fail = false
} else {
session.transfer(flowFile, REL_SUCCESS)
}

Thanks,
Asmath

GetFile with putsql/executesql

[KhajaAsmath Mohammed]

Hi,

I have a scenario where I need to get value from the database and pass it
as an attribute for getfile in subsequent processors.

GetFile >> Execute SQL/PUTSQL >> Get value from the output of SQL and
assign it to Attribute >>  pass attribute value to GetFIle .

Any help please ?

Thanks,
Asmath

Re: NIFI - Wait before merging the files

[KhajaAsmath Mohammed]

Yes Dave, ours is streaming data . We do ETL on the live data before loading 
into target.  We parse the json data, since the volume is high and don’t know 
how many events we get, I am looking to load the data by merging instead of 
doing individual insert. This will avoid more threads on database and does the 
bulk update. 

Merge content works when I stop the prior professor so I was looking to 
implement wait strategy. 

Thanks,
Asmath 

Sent from my iPhone

> On Oct 26, 2020, at 10:33 PM, David Early  wrote:
> 
> 
> Can you expand on your use case?  Do you need to wait for a specific amount 
> of time after receiving the first flow file?  Why would a scheduled run every 
> 2 min not work?  Is the issue that you need all related flowfiles merged 
> together?
> 
> Dave
> 
>> On Mon, Oct 26, 2020, 9:25 PM KhajaAsmath Mohammed  
>> wrote:
>> Thanks David, I have the same issue. Was never able to get it work with 
>> merge record directly . Since my flow has stream of data, I can’t implement 
>> cron in between the job. I might have to see if groovy script has any option 
>> to wait for specified amount of time before merge operation 
>> 
>> Sent from my iPhone
>> 
>>>> On Oct 26, 2020, at 9:22 PM, David Early  
>>>> wrote:
>>>> 
>>> 
>>> I have a case where I have a single stream of data items that need merged 
>>> into a single file.
>>> 
>>> I do this by setting the number of bins in merge to 1 and using the cron 
>>> scheduler to run the merge every 15 min.  I never got the bin age to work 
>>> the way I wanted.  
>>> 
>>> I set the number of flowfiles to include in the output to a value much 
>>> greater than the expected number of ff that will appear in the input queue.
>>> 
>>> This creates a single output flowfile on a timed schedule.
>>> 
>>> In my case, I follow this with a record based query that deduplicates the 
>>> data.
>>> 
>>> Dave
>>> 
>>>> On Sun, Oct 25, 2020, 12:00 PM KhajaAsmath Mohammed 
>>>>  wrote:
>>>> Hi,
>>>> 
>>>> I am looking for a use case to wait for processor/flow files for 2 minutes 
>>>> and later merge them using a merge processor. Is there a processor or 
>>>> script to achieve this?
>>>> 
>>>> Thanks,
>>>> Asmath

Re: NIFI - Wait before merging the files

[KhajaAsmath Mohammed]

Thanks David, I have the same issue. Was never able to get it work with merge 
record directly . Since my flow has stream of data, I can’t implement cron in 
between the job. I might have to see if groovy script has any option to wait 
for specified amount of time before merge operation 

Sent from my iPhone

> On Oct 26, 2020, at 9:22 PM, David Early  wrote:
> 
> 
> I have a case where I have a single stream of data items that need merged 
> into a single file.
> 
> I do this by setting the number of bins in merge to 1 and using the cron 
> scheduler to run the merge every 15 min.  I never got the bin age to work the 
> way I wanted.  
> 
> I set the number of flowfiles to include in the output to a value much 
> greater than the expected number of ff that will appear in the input queue.
> 
> This creates a single output flowfile on a timed schedule.
> 
> In my case, I follow this with a record based query that deduplicates the 
> data.
> 
> Dave
> 
>> On Sun, Oct 25, 2020, 12:00 PM KhajaAsmath Mohammed 
>>  wrote:
>> Hi,
>> 
>> I am looking for a use case to wait for processor/flow files for 2 minutes 
>> and later merge them using a merge processor. Is there a processor or script 
>> to achieve this?
>> 
>> Thanks,
>> Asmath

Re: Putdatabase Record - Value too large for column

[KhajaAsmath Mohammed]

Hi Mike,

I do the same during the failures but it is hard to know which Column it is. We 
have around 90 columns and this error can occur to any of those fields. If the 
error message mentions about the column that would be really helpful to expand 
our size.

Thanks,
Asmath

Sent from my iPhone

> On Oct 26, 2020, at 7:32 AM, Mike Thomsen  wrote:
> 
> One way you might be able to get there would be to add a SplitRecord
> on the failure relationship of the processor and have it loop its
> output back to the processor so you can narrow down which record was
> failing.
> 
>> On Sun, Oct 25, 2020 at 1:58 PM KhajaAsmath Mohammed
>>  wrote:
>> 
>> Hi,
>> 
>> I am using putddabase record with json tree reader to insert data into 
>> database. This works great but is there a possibility to get column name in 
>> the error?  I need to open the file and see the text to find out the column.
>> 
>> putdatabaserecord.error
>> SAP DBTech JDBC: Value too large for column:
>> 
>> Displaying column name will be really helpful. Any suggestions?
>> 
>> Thanks,
>> Asmath
>> 

NIFI - Wait before merging the files

[KhajaAsmath Mohammed]

Hi,

I am looking for a use case to wait for processor/flow files for 2 minutes
and later merge them using a merge processor. Is there a processor or
script to achieve this?

Thanks,
Asmath

Putdatabase Record - Value too large for column

[KhajaAsmath Mohammed]

Hi,

I am using putddabase record with json tree reader to insert data into
database. This works great but is there a possibility to get column name in
the error?  I need to open the file and see the text to find out the column.

putdatabaserecord.error
SAP DBTech JDBC: Value too large for column:

Displaying column name will be really helpful. Any suggestions?

Thanks,
Asmath

Putdatabase- non closeable input stream- json tree reader

[KhajaAsmath Mohammed]

Hi,

I am seeing lot of issues with our data when using putdatabase with
jsontree reader. JSON is valid when I checked actual data but NIFI is
complaining about non closeable input stream. May I know how to resolve
this?

Here is sample json and error message

error = 'Unrecognized character escape '>' (code 62)  at [Source:
org.apache.nifi.stream.io.NonCloseableInputStream@44fe2a37; line: 1,
column: 7162]'



JSON DATA:

{"sensor":"http://canvas..edu/
","sendTime":"2020-07-11T00:33:43.719Z","dataVersion":"
http://purl.imsglobal.org/ctx/caliper/v1p1","context":;
http://purl.imsglobal.org/ctx/caliper/v1p1","id":"urn:uuid:71965033-0e7d-4a76-b9f8-a19c2723b574","type":"AssignableEvent","actor_id":"urn:instructure:canvas:user:1317153817","actor_type":"Person","actor_extensions_user_login":"ckho","actor_extensions_user_sis_id":"A16061867","actor_extensions_root_account_id":"131711","actor_extensions_root_account_lti_guid":"YYzs3up2ikmNIyNSel3biohOBqVtaOJCALBEKdzq:canvas-lms","actor_extensions_root_account_uuid":"YYzs3up2ikmNIyNSel3biohOBqVtaOJCALBEKdzq","actor_extensions_entity_id":"1317153817","action":"Submitted","object_id":"urn:instructure:canvas:submission:13171009353003","object_type":"Attempt","object_dateCreated":"2020-07-11T00:33:42.000Z","object_extensions_submission_type":"online_text_entry","object_extensions_entity_id":"13171009353003","object_assignee_id":"urn:instructure:canvas:user:1317153817","object_assignee_type":"Person","object_assignable_id":"urn:instructure:canvas:assignment:13171000149907","object_assignable_type":"AssignableDigitalResource","object_count":1,"object_body":;Tuesday’s discussion centered on the topic of
phonemes, and how they connect to various writing systems that are found
across numerous languages. These relationships between phonemes and writing
systems were bounded by the various types and classifications of writing
systems, such as abjads, abugidas, alphabets, syllabaries, and logographic
systems. In addition, the writing systems were also placed on a spectrum of
mapping between letters and sounds, with one end of the scale being
opaque/inconsistent and the other being
transparent/consistent. \r\nThe first round of
discussions created several examples of different types of writing systems
along with their characteristics. For example, romantic languages such as
Italian and Spanish were found to be generally transparent, whereas a
language like Japanese was relatively more opaque.\r\nStudents also discussed how the teaching of
writing systems influenced language learning, particularly within opaque
examples of language such as English. For example, the English plural
marker, s, is phonologically found to be expressed as /z/. However, the s
continues to be the prevailing marker of plurality within English. So, when
learning a language and being exposed to its writing system early on, such
as a second or third language, these inconsistencies can make it more
difficult to obtain a proper understanding of the word within a lexicon.
Going the other way around, the understanding of phonemes in one language
could affect the learning of another language’s writing system, which could
represent the same phoneme in different ways. Tania gives the example of
English to Korean, where Hangul represents what would be the same phonemes
in English separately. \r\nFor me, the relationship between phonemes of a particular language
and writing systems represents the absolute complexity found in studying
human language. Through this brief discussion, students were able to
generate ideas spanning across the globe that all inherently dealt with the
same concepts of language. The discussion also generated a specific
difficulty in learning the English language, which is something that I have
experienced growing up as the child of immigrants, and seeing family
members continue to improve their English. \r\nI also think that the different types of
writing systems create different emphasis on individual words throughout
different cultures. For example, the logographic systems of language
generate meaning through a single symbol, which has carried over to be a
popular concept within America through logos and
tattoos. \r\nFinally, one
particular oddity that I noticed was how the Korean alphabet contrasts with
the systems of neighboring countries such as Japan and China. For other
Asian countries such as Vietnam, I know that their alphabet comes from the
French, so I am now more curious to research how the Korean alphabet came
to be.
","EVENT_ID":"f30c6a2c-7df9-49a4-b6c2-5c6b9ead6683","SOURCE_SYSTEM_MODULE":"CANVAS","SOURCE_SYSTEM":"CANVAS","INTEGRATION_TIMESTAMP":"","NIFI_TIMESTAMP":"","NIFI_EVENT_ID":"51bb8d09-a493-4983-9c4d-ab5386c210f2","UNIQUE_ID":"d81fa382-1271-426c-8cfe-3949ea2f0896","EVENT_TYPE":"AssignableEvent","EVENT_TIMESTAMP":"2020-07-11
00:33:42.677","eventTime":"2020-07-11 00:33:42.677","referrer":"

ExtractText resulting multiple attributes

[KhajaAsmath Mohammed]

Hi,

I am extracting text using the below expression in NIFI and it results in 3
attributes.

Is there a way to get only one attribute?
event_bag
*event_bag.0*
*event_bag.1*

[image: image.png]

Thanks,
Asmath

Putdatabase record - changing schema and text of value to CLOB

[KhajaAsmath Mohammed]

Hi,

I am trying to insert json data using the putdatabase record.  All the
columns of json are inserted as expected but one of the column which has
nested JSON data is inserted as CLOB. how to resolve this and force
putdatabase to insert only string.

[image: image.png]

I need to insert and read JSON string which is readable. any suggestions
please?

Thanks,
Asmath

NIFI - Flatten JOSN and entire JSON record to database

[KhajaAsmath Mohammed]

Hi,

I am stuck with implementation where I need to load the entire JSON data
into database using NIFI. I was able to flatten json and load it but at the
same time I need to load the entire text of JSON into other field.

Here is the  approach I tried.
Load entire json into attribute using extractext processor
FLatten json
JOLT transformation on flatten json and add attribute of entire json.

Error: Unmarshal error

Here is the JSON that i am looking for . any suggestions pelase?

{
   "sensor": "http://canvas.wic.edu/;,
   "sendTime": "2020-10-02T17:58:51.747Z",
   "dataVersion": "http://purl.imsglobal.org/ctx/caliper/v1p1;,
   "data": [
  {
 "@context": "http://purl.imsglobal.org/ctx/caliper/v1p1;,
 "id": "urn:uuid:0972f27d-4b87-4e8e-90a0-40f10968820f",
 "type": "NavigationEvent",
 "actor": {
"id": "urn:instructure:canvas:user:1317164254",
"type": "Person",
"extensions": {
   "com.instructure.canvas": {
  "user_login": "jstachow",
  "user_sis_id": "A15397698",
  "root_account_id": "131711",
  "root_account_lti_guid":
"YYzs3up2ikmNIyNSel3biohOBqVtaOJCALBEKdzq:canvas-lms",
  "root_account_uuid":
"YYzs3up2ikmNIyNSel3biohOBqVtaOJCALBEKdzq",
  "entity_id": "1317164254"
   }
}
 },
 "action": "NavigatedTo",
 "object": {
"id": "urn:instructure:canvas:wikiPage:1317184157",
"type": "Page",
"extensions": {
   "com.instructure.canvas": {
  "asset_name": "Zoom",
  "asset_type": "wiki_page",
  "entity_id": "1317184157",
  "context_account_id": "1317100291",
  "http_method": "GET"
   }
}
 },
 "eventTime": "2020-10-02T17:58:50.700Z",
 "referrer": "https://canvas.wic.edu/courses/19278/modules;,
 "edApp": {
"id": "http://canvas.wic.edu/;,
"type": "SoftwareApplication"
 },
 "group": {
"id": "urn:instructure:canvas:course:1317119278",
"type": "CourseOffering",
"extensions": {
   "com.instructure.canvas": {
  "context_type": "Course",
  "entity_id": "1317119278"
   }
}
 },
 "membership": {
"id":
"urn:instructure:canvas:course:1317119278:Learner:1317164254",
"type": "Membership",
"member": {
   "id": "urn:instructure:canvas:user:1317164254",
   "type": "Person"
},
"organization": {
   "id": "urn:instructure:canvas:course:1317119278",
   "type": "CourseOffering"
},
"roles": [
   "Learner"
]
 },
 "session": {
"id":
"urn:instructure:canvas:session:4afdd42062c3e4b4ef3be57ef4e6745d",
"type": "Session"
 },
 "extensions": {
"com.instructure.canvas": {
   "hostname": "canvas.wic.edu",
   "request_id": "ae0dc534-19a4-49ce-b10b-adf4e7ee1602",
   "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36",
   "client_ip": "68.252.124.155",
   "request_url": "
https://canvas.wic.edu/courses/19278/pages/zoom?module_item_id=526508;,
   "version": "1.0.0"
}
 }
  }
   ]
}

Thanks,
Asmath

Read only txt files - Getfile

[KhajaAsmath Mohammed]

Hi,

I have folder where I need to pick up only the files that end with .txt
with getfile. I can read everything and do the route on attribute but we
will loose the files from the folder.

Thanks,
Asmath

Query Record processor

[KhajaAsmath Mohammed]

Hi,

I am looking for some information on how to check datatypes of the data and
load transform them accordingly. I am okay to use any other processor to.

My req:

Check if column is Integer, if integer then load to _INT column else null
value
Check if column length is > 256, if more than 256 load to _Text column else
load to varchar column.

I am assuming we can use case statements and length in query record but not
able to get the syntax. Any help is appreciated

Thanks,
Asmath

Re: Help - CSV Parser - Invalid char

[KhajaAsmath Mohammed]

Hello Mark,

Removing " manually from the string resolved the issue. is there a easy way
by parser? somehow quote character is being ignored in parser.

Thanks,
Asmath

<http://www.avg.com/email-signature?utm_medium=email_source=link_campaign=sig-email_content=webmail>
Virus-free.
www.avg.com
<http://www.avg.com/email-signature?utm_medium=email_source=link_campaign=sig-email_content=webmail>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

On Wed, Sep 2, 2020 at 9:38 AM Mark Payne  wrote:

> Asmanth,
>
> Can you check the logs and provide the full stack trace?
>
> On Sep 2, 2020, at 10:35 AM, KhajaAsmath Mohammed 
> wrote:
>
> Here is the error that I am getting.
>
> 
>
>
> <http://www.avg.com/email-signature?utm_medium=email_source=link_campaign=sig-email_content=webmail>
>  Virus-free.
> www.avg.com
> <http://www.avg.com/email-signature?utm_medium=email_source=link_campaign=sig-email_content=webmail>
>
> On Wed, Sep 2, 2020 at 9:33 AM Mark Payne  wrote:
>
>> Asmanth,
>>
>> What is the exception that you’re seeing?
>>
>> On Sep 2, 2020, at 10:28 AM, KhajaAsmath Mohammed <
>> mdkhajaasm...@gmail.com> wrote:
>>
>> Hi,
>>
>> I am having an issue using the Tab delimited CSV parser in NIFI. I need
>> to convert this csv to json before loading into queue.  I am getting an
>> exception in this data. any help on how to resolve this?
>>
>> *Sample Data:*
>>
>> SUBMISSION_ID ASSIGNMENT_ID COURSE_ID ENROLLMENT_TERM_ID USER_ID
>> GRADER_ID COURSE_ACCOUNT_ID ENROLLMENT_ROLLUP_ID SCORE PUBLISHED_SCORE
>> WHAT_IF_SCORE SUBMISSION_COMMENTS_COUNT ACCOUNT_ID ASSIGNMENT_GROUP_ID
>> GROUP_ID QUIZ_ID QUIZ_SUBMISSION_ID WIKI_ID
>> 13171007638523 7638523 "I will take this exam with integrity and
>> without using any help or materials, according to the expectations of the
>> Instructor and of UCSD's guidelines for academic integrity." Tushita Tandon
>> complete 2020-04-30 18:18:54.235 online_text_entry graded 2020-04-30
>> 18:00:22.073 2020-05-11 17:03:41.972 false true complete 2020-05-05
>> 02:26:16.152 1 13171000124204 regular_submission not_graded_anonymously
>> 470563165038122618 -480988430103600455 human_graded 2020-05-11 17:03:41.972
>>
>> *Tab Delimited CSV reader with header as schema:*
>>
>> 
>> 
>>
>> Thanks,
>> Asmath
>>
>>
>>
>>
>>
>> <http://www.avg.com/email-signature?utm_medium=email_source=link_campaign=sig-email_content=webmail>
>>  Virus-free.
>> www.avg.com
>> <http://www.avg.com/email-signature?utm_medium=email_source=link_campaign=sig-email_content=webmail>
>>
>>
>>
>

Re: Help - CSV Parser - Invalid char

[KhajaAsmath Mohammed]

Here is the error that I am getting.

[image: image.png]

<http://www.avg.com/email-signature?utm_medium=email_source=link_campaign=sig-email_content=webmail>
Virus-free.
www.avg.com
<http://www.avg.com/email-signature?utm_medium=email_source=link_campaign=sig-email_content=webmail>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

On Wed, Sep 2, 2020 at 9:33 AM Mark Payne  wrote:

> Asmanth,
>
> What is the exception that you’re seeing?
>
> On Sep 2, 2020, at 10:28 AM, KhajaAsmath Mohammed 
> wrote:
>
> Hi,
>
> I am having an issue using the Tab delimited CSV parser in NIFI. I need to
> convert this csv to json before loading into queue.  I am getting an
> exception in this data. any help on how to resolve this?
>
> *Sample Data:*
>
> SUBMISSION_ID ASSIGNMENT_ID COURSE_ID ENROLLMENT_TERM_ID USER_ID GRADER_ID
> COURSE_ACCOUNT_ID ENROLLMENT_ROLLUP_ID SCORE PUBLISHED_SCORE WHAT_IF_SCORE
> SUBMISSION_COMMENTS_COUNT ACCOUNT_ID ASSIGNMENT_GROUP_ID GROUP_ID QUIZ_ID
> QUIZ_SUBMISSION_ID WIKI_ID
> 13171007638523 7638523 "I will take this exam with integrity and
> without using any help or materials, according to the expectations of the
> Instructor and of UCSD's guidelines for academic integrity." Tushita Tandon
> complete 2020-04-30 18:18:54.235 online_text_entry graded 2020-04-30
> 18:00:22.073 2020-05-11 17:03:41.972 false true complete 2020-05-05
> 02:26:16.152 1 13171000124204 regular_submission not_graded_anonymously
> 470563165038122618 -480988430103600455 human_graded 2020-05-11 17:03:41.972
>
> *Tab Delimited CSV reader with header as schema:*
>
> 
> 
>
> Thanks,
> Asmath
>
>
>
>
>
> <http://www.avg.com/email-signature?utm_medium=email_source=link_campaign=sig-email_content=webmail>
>  Virus-free.
> www.avg.com
> <http://www.avg.com/email-signature?utm_medium=email_source=link_campaign=sig-email_content=webmail>
>
>
>

Help - CSV Parser - Invalid char

[KhajaAsmath Mohammed]

Hi,

I am having an issue using the Tab delimited CSV parser in NIFI. I need to
convert this csv to json before loading into queue.  I am getting an
exception in this data. any help on how to resolve this?

*Sample Data:*

SUBMISSION_ID ASSIGNMENT_ID COURSE_ID ENROLLMENT_TERM_ID USER_ID GRADER_ID
COURSE_ACCOUNT_ID ENROLLMENT_ROLLUP_ID SCORE PUBLISHED_SCORE WHAT_IF_SCORE
SUBMISSION_COMMENTS_COUNT ACCOUNT_ID ASSIGNMENT_GROUP_ID GROUP_ID QUIZ_ID
QUIZ_SUBMISSION_ID WIKI_ID
13171007638523 7638523 "I will take this exam with integrity and
without using any help or materials, according to the expectations of the
Instructor and of UCSD's guidelines for academic integrity." Tushita Tandon
complete 2020-04-30 18:18:54.235 online_text_entry graded 2020-04-30
18:00:22.073 2020-05-11 17:03:41.972 false true complete 2020-05-05
02:26:16.152 1 13171000124204 regular_submission not_graded_anonymously
470563165038122618 -480988430103600455 human_graded 2020-05-11 17:03:41.972

*Tab Delimited CSV reader with header as schema:*

[image: image.png]
[image: image.png]

Thanks,
Asmath





Virus-free.
www.avg.com

<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

Wait and Notify is very slow

[KhajaAsmath Mohammed]

Hi,

I have seen an issue where wait and notify processor performance is very
slow when the number of files is greater than 1000. It runs faster if it is
less than 1000. Is there a way setting on map cache client service that can
speed up ?

[image: image.png]


Thanks,
Asmath

ReplaceText - Out of memory - Requested array size exceeds VM limit

[KhajaAsmath Mohammed]

Hi ,

I have a file that is throwing an error when looking for particular string
and replace it with other string.

Requested array size exceeds VM limit

Any suggestions for this? File is around 800 MB.

Thanks,
Asmath

Re: NIFI /Kafka - Data loss possibility with node failures

[KhajaAsmath Mohammed]

Thanks Joe. This is really helpful.

On Tue, Aug 11, 2020 at 9:33 AM Joe Witt  wrote:

> Asmath
>
> In a traditional installation, regardless of how a NiFi cluster obtains
> data (kafka, ftp, HTTP calls, TCP listening, etc, ) once it is
> responsible for the data it has ack'd its receipt to the source(s).
>
> If that NiFi node were to become offline the data it owns is delayed. If
> that node becomes unrecoverably offline the data is likely to be lost.
>
> If you're going to run in environments where there are more powerful
> storage alignment options like in many Kubernetes based deployments then
> there are definitely options to solve the possibility of loss case to a
> very high degree and to ensure there is only minimal data delay in the
> worst case.
>
> In a Hadoop style environment though the traditional model I describe
> works very well, leverages appropriate RAID, and is proven highly reliable
> and durable.
>
> Thanks
>
> On Tue, Aug 11, 2020 at 7:26 AM KhajaAsmath Mohammed <
> mdkhajaasm...@gmail.com> wrote:
>
>> Hi,
>>
>> [image: image.png]
>>
>> we have 3 node NIFI clusters and due to some reasons NODE 2 and NODE 3
>> were disconnected when the flow was running . Consume kafka was reading
>> data from all node settings and loading the data into the database.
>>
>> In the above scenario, is there a possibility of loss of data?
>> Distributed processing in terms of hadoop will handle it automatically and
>> assign the task to other active nodes. Will it be the same case with the
>> NIFI cluster?
>>
>> Thanks,
>> Asmath
>>
>

NIFI /Kafka - Data loss possibility with node failures

[KhajaAsmath Mohammed]

Hi,

[image: image.png]

we have 3 node NIFI clusters and due to some reasons NODE 2 and NODE 3 were
disconnected when the flow was running . Consume kafka was reading data
from all node settings and loading the data into the database.

In the above scenario, is there a possibility of loss of data?  Distributed
processing in terms of hadoop will handle it automatically and assign the
task to other active nodes. Will it be the same case with the NIFI cluster?

Thanks,
Asmath

Re: PUT SQL error attribute

[KhajaAsmath Mohammed]

Yes errors are being written to log but it would be useful if part of error
was sent as attribute in failure queue. That will help us to decide on what
processor should be called instead of checking logs.

Thanks,
Asmath

On Mon, Aug 3, 2020 at 3:24 PM Russell Bateman 
wrote:

> Pardon me, I meant:  *${NIFI_ROOT}**/logs/**nifi-app.log .*
>
> On 8/3/20 2:23 PM, Russell Bateman wrote:
>
> Mohammed,
>
> I didn't write *PutSQL* and I haven't dug around to look at its source
> code, but I wouldn't be surprised to learn that those errors were already
> written by it to *${NIFI_ROOT}**/**nifi-app.log* too.
>
> Russ
>
> On 8/3/20 8:56 AM, KhajaAsmath Mohammed wrote:
>
> Hi,
>
> I generally get errors on putsql on the top right for this processor. Is
> there a way to get errors inside the attribute so that I can use it and log
> in our audit table.
>
> Any other alternative? Put Database gives this attribute but it is not
> possible to get with PUTSQL.
>
> Thanks,
> Asmath
>
>
>
>

PUT SQL error attribute

[KhajaAsmath Mohammed]

Hi,

I generally get errors on putsql on the top right for this processor. Is
there a way to get errors inside the attribute so that I can use it and log
in our audit table.

Any other alternative? Put Database gives this attribute but it is not
possible to get with PUTSQL.

Thanks,
Asmath

Re: Retry logic for rest api - NIFI

[KhajaAsmath Mohammed]

Thanks Matt. I got it. We implemented a similar approach to  retry in
database failures.  I will try that now.

On Thu, Jul 30, 2020 at 3:31 PM Matt Burgess  wrote:

> Asmath,
>
> InvokeHttp routes the original flowfile to a number of different
> relationships based on things like the status code. For example if
> you're looking for a 2xx code but want to retry on that for some
> reason, you'd use the "Original" relationship. If you want a retryable
> code (5xx) you can use the "Retry" relationship, and so on. If you
> need a specific code, you can use RouteOnAttribute to match the
> "invokehttp.status.code" attribute to the code(s) you want, and all
> that match can be sent back to the original InvokeHttp processor.
>
> That's a simpler pattern but they can get more complex. For example
> you can start the flowfile with an attribute "retries" set to 5, and
> before you send the flow file back to InvokeHttp, you'd decrement the
> counter with UpdateAttribute and then perhaps drop the flowfile or
> send it to some other mechanism after retries becomes zero.  Also you
> could put a delay in the flow so you're not retrying the same thing as
> fast as possible (that could constitute a Denial-Of-Service attack on
> the HTTP endpoint you're trying to reach). I can't remember which
> relationships from InvokeHttp penalize the flowfile, so there's a
> chance that the processor will handle the delay for you (see the
> User's Guide section on penalized flowfiles).
>
> Regards,
> Matt
>
> On Thu, Jul 30, 2020 at 3:38 PM KhajaAsmath Mohammed
>  wrote:
> >
> > can you please let me know how to use this in NIFI.
> >
> > On Thu, Jul 30, 2020 at 11:19 AM Otto Fowler 
> wrote:
> >>
> >> nipyapi does something like that:
> https://github.com/Chaffelson/nipyapi/blob/164351ee2d92f8c4a75989310662bbad0f7bafc4/nipyapi/utils.py#L210
> >>
> >>
> >>
> >>
> >> On July 30, 2020 at 11:22:29, KhajaAsmath Mohammed (
> mdkhajaasm...@gmail.com) wrote:
> >>
> >> Hi,
> >>
> >> I am looking for some information on how to do retry logic on restapi
> until we get specific status code. Please let me know if you have any
> approach/templates for this
> >>
> >> Thanks,
> >> Asmath
>

Re: Retry logic for rest api - NIFI

[KhajaAsmath Mohammed]

can you please let me know how to use this in NIFI.

On Thu, Jul 30, 2020 at 11:19 AM Otto Fowler 
wrote:

> nipyapi does something like that:
> https://github.com/Chaffelson/nipyapi/blob/164351ee2d92f8c4a75989310662bbad0f7bafc4/nipyapi/utils.py#L210
>
>
>
>
> On July 30, 2020 at 11:22:29, KhajaAsmath Mohammed (
> mdkhajaasm...@gmail.com) wrote:
>
> Hi,
>
> I am looking for some information on how to do retry logic on
> restapi until we get specific status code. Please let me know if you have
> any approach/templates for this
>
> Thanks,
> Asmath
>
>

Re: Route Attribute - Database down

[KhajaAsmath Mohammed]

Yes , you are right Luis. It comes in executesql processor. 

Sent from my iPhone

> On Jun 11, 2020, at 11:41 PM, Luis  wrote:
> 
> 
> 
> HI,
> 
> I have done that with PostgreSQL and MSSQL, and it comes in the answer read 
> by ExecuteSQL.
> 
> Maybe I am not understanding your question. If you like please send more 
> details.
> 
> LC
> 
> 
> 
>> El jue, 11-06-2020 a las 12:11 -0500, KhajaAsmath Mohammed escribió:
>> Hi,
>> 
>> How can I capture the execution error on executesql and route it to 
>> different queue when there is failure on the database. 
>> 
>> executesql.error.message = 'com.sap.db.jdbc.exceptions.JDBCDriverException: 
>> SAP DBTech JDBC: Cannot connect to jdbc:sap://hana-xx:30041 [Cannot 
>> connect to host hanaxxx:30041 [Connection refused (Connection refused)], 
>> -813.].' 
>>
>> Thanks,
>> Asmath

Route Attribute - Database down

[KhajaAsmath Mohammed]

Hi,

How can I capture the execution error on executesql and route it to
different queue when there is failure on the database.

executesql.error.message = 'com.sap.db.jdbc.exceptions.JDBCDriverException:
SAP DBTech JDBC: Cannot connect to jdbc:sap://hana-xx:30041 [Cannot
connect to host hanaxxx:30041 [Connection refused (Connection refused)],
-813.].'

Thanks,
Asmath

Getfile Count

[KhajaAsmath Mohammed]

Hi,

I am looking for below approach, any ideas on this please

Getfile --> CountOfFiles(GetFileFolder) -->Pass Count as attribute to all
flow files.

Thanks,
Asmath

Batch Dependency in NIFI - GetFile

[KhajaAsmath Mohammed]

Hi,

I have use case where the data is read using Getfile from file location and
loads that data into database. I would like to have trigger once the
database load is successful for all the files.

I tried approach of Wait/Notify but still it does not work as it works for
individual files. Lets say, I have 1300 files and I should alert trigger
after completion of 1300 files.

This count changes and it is dynamic in nature. Any suggestions on this
approach please.

Thanks,
Asmath

Consume Kafka record - Key and offset from kafka

[KhajaAsmath Mohammed]

Hi,

Is there a way to get key,timestamp and offset in consume kafka record?

Offset: 380050   Key:
urn:uuid:eb3b516b-9a31-4365-8ccb-e8c009073f63|+|2020-05-05
  Timestamp: 2020-05-05 03:23:54.210 Headers: empty



{"ID":"urn:uuid:eb3b516b-9a31-4365-8ccb-e8c009073f63","KSQL_COL_1":2}

NIFI - Connect processors properly

[KhajaAsmath Mohammed]

Hi,

How do I connect two processors properly as shown below?

[image: image.png]

when I connect, they doesnt look good.

Thanks,
Asmath

Re: List file and Ftoetch File - wait until all files are read

[KhajaAsmath Mohammed]

Thanks Mark. This is really helpful. Will try this option. 

Sent from my iPhone

> On Apr 29, 2020, at 1:00 AM, Mark Payne  wrote:
> 
> Asmath,
> 
> NiFi is not job oriented, and it doesn’t really have any notion of being 
> “done” - it monitors the directory continuously, as you noted.
> 
> What might be of use to you is the MonitorActivity processor and the 
> Wait/Notify Processors. I recommend that you read up on their documentation. 
> But you could potentially do something along the lines of:
> 
> 
> ListFile — success —> MonitorActivity — success —> FetchFile —> Wait
>— inactive —> 
> Notify
> 
> So it should pull the data in, and then wait.
> The MonitorActivity will detect that no more data has come in for some amount 
> of time (say 5 minutes) and as a result determine that the “job is done” and 
> send a FlowFile to Notify to unblock the waiting FlowFiles.
> 
> Hope this helps
> -Mark
> 
> 
>> On Apr 28, 2020, at 3:07 AM, KhajaAsmath Mohammed  
>> wrote:
>> 
>> Hi,
>> 
>> We are using List file and Fetch file to continuously monitor shared 
>> location to read the files and process the next level of jobs. Issue we are 
>> facing is to wait for all the files to be read before running the next jobs.
>> 
>> Currently list file has 5 files but during the fetch process , it reads file 
>> one by one. Next processor after fetch will work on the files but we have to 
>> wait until all the files are read before the next processor runs. How to 
>> acheive this.
>> 
>> Number of files keeps changing.
>> 
>> Thanks,
>> Asmath
> 

Re: Nifi -Incremental load from database

[KhajaAsmath Mohammed]

Hi ,

I am looking for both solutions actually.

Thanks,
Asmath

Sent from my iPhone

> On Feb 21, 2020, at 1:47 PM, Pierre Villard  
> wrote:
> 
> 
> Hi,
> 
> When you mean incremental load, do you mean only newly inserted lines in the 
> source database? or do you also want to capture deletes and updates?
> 
> Thanks,
> Pierre
> 
>> Le ven. 21 févr. 2020 à 05:16, KhajaAsmath Mohammed 
>>  a écrit :
>> Hi,
>> 
>> I am looking for solution to do incremental load from database once the 
>> initial load is done. Incremental should happen automatically with the flow 
>> . Any suggestions 
>> 
>> Does capturechangesql help in this case ? 
>> 
>> Thanks,
>> Asmath
>> 
>> Sent from my iPhone

Nifi -Incremental load from database

[KhajaAsmath Mohammed]

Hi,

I am looking for solution to do incremental load from database once the initial 
load is done. Incremental should happen automatically with the flow . Any 
suggestions 

Does capturechangesql help in this case ? 

Thanks,
Asmath

Sent from my iPhone

Re: NIFI Bug with Convert Record - 999999.99 changed to 1000000.0

[KhajaAsmath Mohammed]

Thanks Mark, this is helpful. I found out some more issues. Let me circle back 
with test data.

Btw, this works with Jackson parser

Sent from my iPhone

> On Feb 19, 2020, at 3:52 PM, Mark Payne  wrote:
> 
>  I presume you're using schema inference to determine the column type in the 
> reader?
> 
> Interestingly, the following code:
> System.out.println("As a double: " + Double.parseDouble("99.99"));
> System.out.println("As a float: " + Float.parseFloat("99.99"));
> Produces the following output:
> As a double: 99.99
> As a float: 100.0
> 
> The CSV inference logic has the following code:
> if (value.contains(".")) {
> try {
> final double doubleValue = Double.parseDouble(value);
> if (doubleValue > Float.MAX_VALUE || doubleValue < Float.MIN_VALUE) {
> return RecordFieldType.DOUBLE.getDataType();
> }
> 
> return RecordFieldType.FLOAT.getDataType();
> } catch (final NumberFormatException nfe) {
> return RecordFieldType.STRING.getDataType();
> }
> }
> 
> So it determines that there's a decimal point. And the value easily fits 
> within a Float, so it considers the value to be a Float. We should probably 
> improve our inference logic to do a better job determining whether a value is 
> a Float or Double - or else always use a Double instead of a Float.
> 
> Thanks
> -Mark
> 
> 
>> On Feb 19, 2020, at 4:42 PM, Shawn Weeks  wrote:
>> 
>> How are you defining the schema and what data type are setting for that 
>> column?
>>  
>> Thanks
>> Shawn
>>  
>> From: KhajaAsmath Mohammed 
>> Reply-To: "users@nifi.apache.org" 
>> Date: Wednesday, February 19, 2020 at 3:32 PM
>> To: "users@nifi.apache.org" 
>> Subject: NIFI Bug with Convert Record - 99.99 changed to 100.0
>>  
>> Hi,
>>  
>> I am using below text to convert CSV to JSON using convert record processor. 
>> Value of 99.99 changed to 100.0 , May I know how is this possible? 
>>  
>>  
>> OPERATING_UNIT|TEST_34_2|TEST_34_15|TEST_34_5
>> "141516"|"1.1"|"1.1"|"1.1"
>> "141517"|"1.11"|"1.11"|"1.11"
>> "141518"|"1101.11"|"99.99"|"1101.11"
>>  
>>  
>> 
>>  
>> Is this happening with CSV reader or JSON record set writer? 
>>  
>> Thanks,
>> Asmath
> 

NIFI Bug with Convert Record - 999999.99 changed to 1000000.0

[KhajaAsmath Mohammed]

Hi,

I am using below text to convert CSV to JSON using convert record
processor. Value of 99.99 changed to 100.0 , May I know how is this
possible?


OPERATING_UNIT|TEST_34_2|TEST_34_15|TEST_34_5
"141516"|"1.1"|"1.1"|"1.1"
"141517"|"1.11"|"1.11"|"1.11"
"141518"|"1101.11"|"99.99"|"1101.11"


[image: image.png]

Is this happening with CSV reader or JSON record set writer?

Thanks,
Asmath

Kakfa Consume Record with offsets

[KhajaAsmath Mohammed]

Hi Community,

I am looking on some information on how to access kafka offsets, keys and
entire value of kafka message using consume record?

Is there any other processor to consume messages from particular offset?

Also, what happens if there is error after consuming messages. Assume there
is database failure and we will loose those messages. Potential data loss
in this case.

Thanks,
Asmath

Re: COnsumekakarecord error messages

[KhajaAsmath Mohammed]

I am using json record reader and json record set writer in consume kafka. Do I 
need to change these record readers just to confirm about the corrupt data? Can 
you help me what readers should I use in that case ?

Sent from my iPhone

> On Feb 5, 2020, at 3:47 PM, Pierre Villard  
> wrote:
> 
> 
> Hi,
> 
> It seems like you have a message that cannot be parsed correctly by your 
> record processor. It is probably not matching your schema or containing some 
> corrupted data? You could use a the non record ConsumeKafka processor to get 
> the message without doing any parsing and see what the message looks like.
> 
> HTH
> Pierre
> 
>> Le mer. 5 févr. 2020 à 16:27, KhajaAsmath Mohammed  
>> a écrit :
>> Hi,
>> 
>> I was able to consume messages from consumekafkarecord processor. Suddenly I 
>> started getting below error. any solutions for this please?
>> 
>> 
>> 
>> 
>> Thanks,
>> Asmath

Re: Add filename to JSON attributes

[KhajaAsmath Mohammed]

I was able to get it through JOLT using below. Need to add attribute
extracted from JSON instead of test

[
  {
"operation": "shift",
"spec": {
  "*": "test_&"
    }
  }

]

On Thu, Jan 30, 2020 at 6:05 PM KhajaAsmath Mohammed <
mdkhajaasm...@gmail.com> wrote:

> Hi,,
>
> I have requirement where I need to add filename to attributes so that it
> would be easy for us to load into table by matching attribute name to
> columns in database.
>
> In this case we have 70 + json files having different attributes.
>
> "session": {
> "filename":"course",
> "id": "
> https://example.edu/sessions/1f6442a482de72ea6ad134943812bff564a76259;,
> "type": "Session",
> "user": "https://example.edu/users/554433;,
> "dateCreated": "2018-11-15T10:00:00.000Z",
> "startedAtTime": "2018-11-15T10:00:00.000Z"
>   }
>
> "course_filename":"course",
> " course_id": "
> https://example.edu/sessions/1f6442a482de72ea6ad134943812bff564a76259;,
> " course_type": "Session",
> " course_user": "https://example.edu/users/554433;,
> " course_dateCreated": "2018-11-15T10:00:00.000Z",
> " course_startedAtTime": "2018-11-15T10:00:00.000Z"
>
> I am assuming it can be achieved through JOLT, would be really helpful if
> anyone has an idea on this.
>
> Thanks,
> Asmath
>

Add filename to JSON attributes

[KhajaAsmath Mohammed]

Hi,,

I have requirement where I need to add filename to attributes so that it
would be easy for us to load into table by matching attribute name to
columns in database.

In this case we have 70 + json files having different attributes.

"session": {
"filename":"course",
"id": "
https://example.edu/sessions/1f6442a482de72ea6ad134943812bff564a76259;,
"type": "Session",
"user": "https://example.edu/users/554433;,
"dateCreated": "2018-11-15T10:00:00.000Z",
"startedAtTime": "2018-11-15T10:00:00.000Z"
  }

"course_filename":"course",
" course_id": "
https://example.edu/sessions/1f6442a482de72ea6ad134943812bff564a76259;,
" course_type": "Session",
" course_user": "https://example.edu/users/554433;,
" course_dateCreated": "2018-11-15T10:00:00.000Z",
" course_startedAtTime": "2018-11-15T10:00:00.000Z"

I am assuming it can be achieved through JOLT, would be really helpful if
anyone has an idea on this.

Thanks,
Asmath

Re: CRON issue in NIFI

[KhajaAsmath Mohammed]

I have used the below expression to match PST and it didn't run. I am not
sure what is making wrong.
0 10 11 ? * *

Thanks,
Asmath

On Mon, Jan 13, 2020 at 8:20 PM Boris Tyukin  wrote:

> note this is quartz cron syntax which is different from standard cron.
> This is a handy quartz cron calculator
>
> https://www.freeformatter.com/cron-expression-generator-quartz.html
>
> On Mon, Jan 13, 2020 at 7:25 PM Shawn Weeks 
> wrote:
>
>> I usually use one of the cron calculators online as it can be tricky. Try
>> this one as it should run every minute or so.
>>
>> https://crontab.guru/#1-59_*_*_*_*
>>
>> Sent from my iPhone
>>
>> On Jan 13, 2020, at 6:06 PM, KhajaAsmath Mohammed <
>> mdkhajaasm...@gmail.com> wrote:
>>
>> 
>> Hi,
>>
>> I am trying to schedule job in NIFI through CRON and it never triggers.
>> Tried all possible options and it never worked. does the time showing on
>> the UI is the time that is set to? should we use same format?
>>
>> 
>>
>>
>> Thanks,
>> Asmath
>>
>>

Stored procedure in NIFI

[KhajaAsmath Mohammed]

Hi,

How to call stored procedure in NIFI that accepts parameters? I have seen
solution to use executeprocess or python scripts but is there  a way to do
with processors?

Thanks,
Asmath

150 + if else on route on attributes

[KhajaAsmath Mohammed]

Hi,

I have requirement where I need to do develop and parse json based on attribute 
value. This will eventually end up writing 150+ route on attributes. Is there a 
good approach to deal with it?

Thanks,
Asmath

Sent from my iPhone

NIFI expression language

[KhajaAsmath Mohammed]

Hi,

I have existing flow where it replace value in the json document. I
understood what is does but i would like to know what does ?s .*? does and
how to learn more about all these characters from expression language. Any
guidance would really help me.

(?s)("eventTime"\s*:\s*)("(.*?)") with $1\"${SEND_TIME}\"

[image: image.png]

Thanks,
Asmath

Re: Global Variable in NIFI

[KhajaAsmath Mohammed]

Thanks. You were right . 

Sent from my iPhone

> On Nov 20, 2019, at 6:10 PM, Bimal Mehta  wrote:
> 
> Click on the Process Group and open Variable Registry. Most likely it's 
> defined as Process Group variable inside Variable Registry
> 
>> On Wed, Nov 20, 2019, 1:24 PM KhajaAsmath Mohammed  
>> wrote:
>> Hi,
>> 
>> I have a flow in NIFI where I am not able to find the value of attribute 
>> that is being read from. I checked all the processors but didnt find this 
>> attribute. Looks like it is being read somewhere from globally. May I know 
>> how to do this?
>> 
>> 
>> 
>> 
>> 
>> Any idea where this property can be located?
>> 
>> Thanks,
>> Asmath

NIFI - Double quotes jackson parser CSV reader

[KhajaAsmath Mohammed]

Hi,

I am using the jackson parser inside csv reader before loading that into
database. I have weird behavior where the double quotes are removed for
below string and works for other strings. May I know what would be reason.

"2160 University Ave (""Chulo"")"  --> Converted to 2160 University Ave
("Chulo")
"""Biology for Voters"""  --> Biology for Voters . Double quotes is removed
completely.

May I know what is the issue with String """Biology for Voters""".

Thanks,
Asmath

NIFI- Delete file after putdatabase record

[KhajaAsmath Mohammed]

Hi,

I have use case where I need to remove the files only after put database is
successful.

Currently I have , GetFile(With KeepSource=true) --> PutDatase -->
ExecuteStreamCommand to remove the successfile from NIFI fileserver.

May I know how to achieve this without using executestreamcommand
shellscript?

Thanks,
Asmath

NIFI - Fetchfile - Execute SQL - Put Database Record

[KhajaAsmath Mohammed]

HI,

I have requirement of read a file from fileserver and delete all records
from Database if filename is present in database. Next step is to load data
from this file.

This is more like delete/insert. I cannot do upserts because the new file
can have more/less records after correction from the source.

Does anyone have idea to this?

I tried Fetchfile ->ExecuteSQL- PutDatabse.
I will loose the file in this case after executeSQL.

Have idea of placing funnel in between and do fork but delete might take
time before file starts processing.

Thanks,
Asmath

NIFI - KSQL table from kafka

[KhajaAsmath Mohammed]

Hi,

Could anyone please let me know your experience on reading KSQL table in
NIFI. We use ksql table in conflunet version of kafka to manage updates and
I would like to view the data from it using NIFI. Any suggestions?

Thanks,
Asmath

NIFI - Kafka & confluent Schema registry example

[KhajaAsmath Mohammed]

Hi,

I am looking for an example where I need to push data to kafka topic and at
the same time use schema registry of confluent to use the schema for this
topic.

can anyone please share examples/templates if anyone has it. That would be
really helpful.

Thanks,
Asmath

Re: NIFI - PUTSQL - sql.args.1.type

[KhajaAsmath Mohammed]

This link doesn’t have any value to integer or other types.  It has comparison 
of data types between sql and java.  

Sent from my iPhone

> On Sep 23, 2019, at 8:20 PM, Juan Pablo Gardella 
>  wrote:
> 
> The values of the constants defined at 
> https://docs.oracle.com/javase/8/docs/api/java/sql/Types.html
> 
>> On Mon, Sep 23, 2019, 8:10 PM Wesley C. Dias de Oliveira 
>>  wrote:
>> Hi, KhajaAsmath.
>> 
>> I've search it too but found nothing.
>> 
>> I think it's more easy to observe the type based on your query. "mapping" 
>> the arguments with the query fields.
>> 
>>> Em seg, 23 de set de 2019 às 20:02, KhajaAsmath Mohammed 
>>>  escreveu:
>>> Hi,
>>> 
>>> I have existing flow and trying to understand what is sql.args.1.type ?
>>> 
>>> sql.args.1.type =11
>>> sql.args.1.type=13
>>> 
>>> I understood that they are matched to column data type for sql.args.1.value.
>>> 
>>> what is data type for 11 and 13 ? May I know what are all the options 
>>> availabe for String, Integer, double ,date etc
>>> 
>>> Thanks,
>>> Asmath 
>> 
>> 
>> -- 
>> Grato,
>> Wesley C. Dias de Oliveira.
>> 
>> Linux User nº 576838.

NIFI - PUTSQL - sql.args.1.type

[KhajaAsmath Mohammed]

Hi,

I have existing flow and trying to understand what is sql.args.1.type ?

sql.args.1.type =11
sql.args.1.type=13

I understood that they are matched to column data type for sql.args.1.value.

what is data type for 11 and 13 ? May I know what are all the options
availabe for String, Integer, double ,date etc

Thanks,
Asmath

Re: NIFI - " character in CSV Reader

[KhajaAsmath Mohammed]

Thanks. May I know the controller service for this. I dont see that option
in CSV reader.

On Tue, Sep 17, 2019 at 5:51 PM Eric Ladner  wrote:

> Try changing the CSV Format to Excel and put the escape character back to
> \ (backslash).
>
> I believe the CSV format definition defines the handling of double quotes
> separately from an escape character.  I know Excel handles double-double
> quotes by default, tho.
>
> Fyi - """Teahous"" Beijing People's Art" after the removal of surrounding
> quotes and special handling of double-double quotes should end up as "Teahous"
> Beijing People's Art
>
> On Tue, Sep 17, 2019 at 5:35 PM KhajaAsmath Mohammed <
> mdkhajaasm...@gmail.com> wrote:
>
>> Hi,
>>
>> I am using the CSVReader to read the data from the csv files.
>>
>> [image: image.png]
>>
>> Here is the sample data I have it.
>>
>> "A"|"""Teahous"" Beijing People's Art"|"""Teahous"""|"
>>
>> With the above CSV reader, " are completely removed. My expected output
>> should be *""Teahous"" Beijing People's Art* for the value  *"""Teahous""
>> Beijing People's Art" . *
>>
>> any suggestions please?
>>
>> Thanks,
>> Asmath
>>
>
>
> --
> Eric Ladner
>

NIFI - " character in CSV Reader

[KhajaAsmath Mohammed]

Hi,

I am using the CSVReader to read the data from the csv files.

[image: image.png]

Here is the sample data I have it.

"A"|"""Teahous"" Beijing People's Art"|"""Teahous"""|"

With the above CSV reader, " are completely removed. My expected output
should be *""Teahous"" Beijing People's Art* for the value  *"""Teahous""
Beijing People's Art" . *

any suggestions please?

Thanks,
Asmath

Re: NIFI Questions

[KhajaAsmath Mohammed]

Thanks Kei, I got most of the information. I have one question about the
history run.

Lets say I have a flow of 10 processors, I would like to see the history of
this flow run. will Data provenance show us combine flow or individual
processor levels?


On Wed, Aug 28, 2019 at 11:14 AM Kei Schaff  wrote:

> *Version control*: Keeping the development modular through the use of
> process groups helps. We use NiFi Registries to track changes in both
> development and production environments. We also separately create
> templates
> of process groups that we version control with git. Although, the templates
> are XML files, viewing differences and doing merges are usually too
> difficult to be practicable.
>
> *Prod Migration*: When promoting a process group from development to
> production we normally import the git-managed templates of process groups.
> It is also possible to get the process groups directly from the development
> environment's NiFi registry. In either case, there are always manual steps
> to take, such as setting sensitive parameters and configuring controllers.
>
> *Job Monitoring*: We normally only monitor failures by having the sent (via
> HttpInvoke) to an external alerting system. We have also used the NiFi REST
> API to create data points that are sent to AWS CloudWatch to see the
> historical data volumes of various process groups (usually representing
> distinct data sets or providers). We have integrated NiFi with Cloudera
> (before the Horton Works merger) to get server status.
>
> *External Schedulers*: There are so many ways to integrate with external
> schedulers. In addition to a scheduler pushing an event via a NiFi REST API
> call, NiFi can listen for external events through processors that read a
> (local or remote) file system, monitor queues, perform REST calls, etc.
>
> *History Run Log*: NiFi's provenance is likely what you are looking for
> here. Also, any part of a NiFi flow can be instrumented to log information
> externally, as well.
>
> Although, your environment is likely different from ours. I hope these
> ideas
> help.
>
> On Wed, Aug 28, 2019 at 11:16 AM KhajaAsmath Mohammed <
> mdkhajaasm...@gmail.com> wrote:
>
>> Hi,
>>
>> I have few questions on nifi as we are implementing it in our production
>> environment.
>>
>>
>>- How to perform version control among multiple developers
>>- What’s the strategy to do QA/Prod migration
>>- Job monitoring in NiFi
>>- Can the NiFi flows be integrated with external schedulers ( maybe
>>it needs to be through API?)
>>- Is there a history run log, for example, is it possible to check
>>the status or error message for the run 3 days ago?
>>
>>
>> Thanks,
>> Mohd
>>
>

NIFI Questions

[KhajaAsmath Mohammed]

Hi,

I have few questions on nifi as we are implementing it in our production
environment.


   - How to perform version control among multiple developers
   - What’s the strategy to do QA/Prod migration
   - Job monitoring in NiFi
   - Can the NiFi flows be integrated with external schedulers ( maybe it
   needs to be through API?)
   - Is there a history run log, for example, is it possible to check the
   status or error message for the run 3 days ago?


Thanks,
Mohd