date:20220118

[ANNOUNCE] Apache NiFi 1.15.3 release

2022-01-18 Thread Joe Witt

Hello

The Apache NiFi team would like to announce the release of Apache NiFi 1.15.3.

This is a bug fix and security focused release.

Apache NiFi is an easy to use, powerful, and reliable system to
process and distribute
data.  Apache NiFi was made for dataflow.  It supports highly
configurable directed graphs
of data routing, transformation, and system mediation logic.

More details on Apache NiFi can be found here:
https://nifi.apache.org/

The release artifacts can be downloaded from here:
https://nifi.apache.org/download.html

Maven artifacts have been made available and mirrored as per normal
ASF artifact processes.

Issues closed/resolved for this list can be found here:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12351203

Release note highlights can be found here:
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.15.3

Thank you
The Apache NiFi team

Re: PutSQL in combination with ConvertJSONToSQL gives java.sql.SQLException: Invalid column type for Orcale DataType BINARY_DOUBLE

2022-01-18 Thread Mark Payne

Hi Sven,

That is interesting. Looking at the JDBC Types class, there is no BINARY_DOUBLE 
type - and no constant with a value of 101. So it appears to be a non-standard 
type.

So the script that you have in place is certainly one alternative.

It is worth noting, though, that ConvertJsonToSql and PutSQL are older 
implementations an PutDatabaseRecord should be preferred. This allows you to 
send your JSON directly to the database without that intermediate step of 
converting the JSON into SQL. Recommend you give that a try and see if it works 
for you.

Thanks
-Mark


On Jan 18, 2022, at 9:12 AM, Sven Ritter 
mailto:sven.rit...@sedapta.com>> wrote:

Hi all,

It’s my first post here and I hope I give all needed information to help me 😊

We are using PutSQL (1.15.2) in combination with ConvertJSONToSQL to perform 
inserts/updates into Oracle 19 DB.
It seems that Nifi has problems to handle the BINARY_DOUBLE data type.

The ConvertJSONToSQL processor produces the attributes sql.args.N.type = ‘101’ 
and sql.args.N.value = ‘0.32’.
The PutSQL processor fails with java.sql.SQLException: Invalid column type

When I pass the sql statement directly as SQL Statement in PutSQL it works fine.

As a workaround I change by groovy script all sql.args.N.type = ‘101’ 
(BINARY_DOUBLE) to sql.args.N.type = ‘6’ (FLOAT), which is working for now.

When I read the source code correctly, it seems that there is no case for 
BINARY_DOUBE and it jumps to the default (at 
org.apache.nifi.util.db.JdbcCommon.setParameter(JdbcCommon.java:863))

Did I forget something essential or is the data type BINARY_DOUBLE not 
supported?

Here the log entry:

2022-01-18 13:10:38,802 ERROR [Timer-Driven Process Thread-5] 
o.apache.nifi.processors.standard.PutSQL 
PutSQL[id=40bb12a1-cf65-3999-6d1d-d98c308f52fb] Failed to update database for 
StandardFlowFileRecord[uuid=7fd2db89-c559-44ce-b934-d47a8b6195a9,claim=StandardContentClaim
 [resourceClaim=StandardResourceClaim[id=1642429584073-1, container=default, 
section=1], offset=599401, 
length=761],offset=0,name=2beadf83-626f-4ec6-aa32-e1232db34a68,size=761] due to 
java.sql.SQLException: Invalid column type; it is possible that retrying the 
operation will succeed, so routing to retry: java.sql.SQLException: Invalid 
column type
java.sql.SQLException: Invalid column type
at 
oracle.jdbc.driver.OraclePreparedStatement.setObjectCritical(OraclePreparedStatement.java:8148)
at 
oracle.jdbc.driver.OraclePreparedStatement.setObjectInternal(OraclePreparedStatement.java:7639)
at 
oracle.jdbc.driver.OraclePreparedStatement.setObject(OraclePreparedStatement.java:8213)
at 
oracle.jdbc.driver.OraclePreparedStatementWrapper.setObject(OraclePreparedStatementWrapper.java:227)
at 
org.apache.commons.dbcp2.DelegatingPreparedStatement.setObject(DelegatingPreparedStatement.java:529)
at 
org.apache.commons.dbcp2.DelegatingPreparedStatement.setObject(DelegatingPreparedStatement.java:529)
at sun.reflect.GeneratedMethodAccessor919.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254)
at 
org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.access$100(StandardControllerServiceInvocationHandler.java:38)
at 
org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler$ProxiedReturnObjectInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:240)
at com.sun.proxy.$Proxy192.setObject(Unknown Source)
at 
org.apache.nifi.util.db.JdbcCommon.setParameter(JdbcCommon.java:863)
at 
org.apache.nifi.util.db.JdbcCommon.setParameters(JdbcCommon.java:696)
at 
org.apache.nifi.processors.standard.PutSQL.lambda$null$4(PutSQL.java:344)
at 
org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:127)
at 
org.apache.nifi.processors.standard.PutSQL.lambda$new$5(PutSQL.java:342)
at 
org.apache.nifi.processors.standard.PutSQL.lambda$new$7(PutSQL.java:387)
at 
org.apache.nifi.processor.util.pattern.PutGroup.putFlowFiles(PutGroup.java:91)
at 
org.apache.nifi.processor.util.pattern.Put.onTrigger(Put.java:103)
at 
org.apache.nifi.processors.standard.PutSQL.lambda$onTrigger$19(PutSQL.java:635)
at 
org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:120)
at 
org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184)
at 
org.apache.nifi.processors.standar

PutSQL in combination with ConvertJSONToSQL gives java.sql.SQLException: Invalid column type for Orcale DataType BINARY_DOUBLE

2022-01-18 Thread Sven Ritter

Hi all,

 

It’s my first post here and I hope I give all needed information to help me 😊

 

We are using PutSQL (1.15.2) in combination with ConvertJSONToSQL to perform 
inserts/updates into Oracle 19 DB.

It seems that Nifi has problems to handle the BINARY_DOUBLE data type.

 

The ConvertJSONToSQL processor produces the attributes sql.args.N.type = ‘101’ 
and sql.args.N.value = ‘0.32’.

The PutSQL processor fails with java.sql.SQLException: Invalid column type

 

When I pass the sql statement directly as SQL Statement in PutSQL it works fine.

 

As a workaround I change by groovy script all sql.args.N.type = ‘101’ 
(BINARY_DOUBLE) to sql.args.N.type = ‘6’ (FLOAT), which is working for now.

 

When I read the source code correctly, it seems that there is no case for 
BINARY_DOUBE and it jumps to the default (at 
org.apache.nifi.util.db.JdbcCommon.setParameter(JdbcCommon.java:863))

 

Did I forget something essential or is the data type BINARY_DOUBLE not 
supported?

 

Here the log entry:

 

2022-01-18 13:10:38,802 ERROR [Timer-Driven Process Thread-5] 
o.apache.nifi.processors.standard.PutSQL 
PutSQL[id=40bb12a1-cf65-3999-6d1d-d98c308f52fb] Failed to update database for 
StandardFlowFileRecord[uuid=7fd2db89-c559-44ce-b934-d47a8b6195a9,claim=StandardContentClaim
 [resourceClaim=StandardResourceClaim[id=1642429584073-1, container=default, 
section=1], offset=599401, 
length=761],offset=0,name=2beadf83-626f-4ec6-aa32-e1232db34a68,size=761] due to 
java.sql.SQLException: Invalid column type; it is possible that retrying the 
operation will succeed, so routing to retry: java.sql.SQLException: Invalid 
column type

java.sql.SQLException: Invalid column type

at 
oracle.jdbc.driver.OraclePreparedStatement.setObjectCritical(OraclePreparedStatement.java:8148)

at 
oracle.jdbc.driver.OraclePreparedStatement.setObjectInternal(OraclePreparedStatement.java:7639)

at 
oracle.jdbc.driver.OraclePreparedStatement.setObject(OraclePreparedStatement.java:8213)

at 
oracle.jdbc.driver.OraclePreparedStatementWrapper.setObject(OraclePreparedStatementWrapper.java:227)

at 
org.apache.commons.dbcp2.DelegatingPreparedStatement.setObject(DelegatingPreparedStatement.java:529)

at 
org.apache.commons.dbcp2.DelegatingPreparedStatement.setObject(DelegatingPreparedStatement.java:529)

at sun.reflect.GeneratedMethodAccessor919.invoke(Unknown Source)

at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at 
org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254)

at 
org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.access$100(StandardControllerServiceInvocationHandler.java:38)

at 
org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler$ProxiedReturnObjectInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:240)

at com.sun.proxy.$Proxy192.setObject(Unknown Source)

at 
org.apache.nifi.util.db.JdbcCommon.setParameter(JdbcCommon.java:863)

at 
org.apache.nifi.util.db.JdbcCommon.setParameters(JdbcCommon.java:696)

at 
org.apache.nifi.processors.standard.PutSQL.lambda$null$4(PutSQL.java:344)

at 
org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:127)

at 
org.apache.nifi.processors.standard.PutSQL.lambda$new$5(PutSQL.java:342)

at 
org.apache.nifi.processors.standard.PutSQL.lambda$new$7(PutSQL.java:387)

at 
org.apache.nifi.processor.util.pattern.PutGroup.putFlowFiles(PutGroup.java:91)

at 
org.apache.nifi.processor.util.pattern.Put.onTrigger(Put.java:103)

at 
org.apache.nifi.processors.standard.PutSQL.lambda$onTrigger$19(PutSQL.java:635)

at 
org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:120)

at 
org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184)

at 
org.apache.nifi.processors.standard.PutSQL.onTrigger(PutSQL.java:635)

at 
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273)

at 
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)

at 
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:103)

at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)

at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at 
java.util.concurrent.FutureTa

RE: nifi-app log rotation

2022-01-18 Thread QUEVILLON EMMANUEL - EXT-SAFRAN ENGINEERING SERVICES (SAFRAN)

Hi Isha,

You’re right, I wasn’t aware of this rollingPolicy and the supported patterns
for it.
After reading logback documentation, it looks %i is mandatory with %d when
using SizeAndTimeBaseRollingPolicy.

Thanks for pointing me there!
Regards

Emmanuel
C2 - Restricted

De : Isha Lamboo
Envoyé : mardi 18 janvier 2022 11:40
À : users@nifi.apache.org
Objet : RE: nifi-app log rotation

Hi Emmanuel,

The filename pattern needs to match with the rolling policy. You omitted that
line in your snippet, but by default, the nifi-app looks like this:

SizeAndTimeBased means you have to use both %d (time) and %i (size). If you
only want to use one of those, change the policy to TimeBased or SizeBased.

In your example, you left out the %i, so it will be invalid for the default
policy. Change the policy to TimeBasedRollingPolicy (like the user file has) to
make it work.
You can also add “debug=true” to the tag, but that only takes effect if you submit a valid configuration,
so you may need to revert to the original one first. With debug on, you’ll find
more feedback in the logfile.

Regards,

Isha

Van: QUEVILLON EMMANUEL - EXT-SAFRAN ENGINEERING SERVICES (SAFRAN)
mailto:emmanuel.quevillon.e...@safrangroup.com>>
Verzonden: dinsdag 18 januari 2022 10:54
Aan: users@nifi.apache.org
Onderwerp: nifi-app log rotation

Hi guys,

I’m trying to configure the nifi-app.log rotation process by decreasing the
rotation time and compressing the final rotated file.
To do this, I’ve updated ‘conf/logback.xml’ file by this default configuration:

…

${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{-MM-dd_HH}.%i.log
…

To this one

…

${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d.log.gz
…

However, since I’ve updated `conf/logback.xml`, nifi-appl.log is not updated
anymore and not new file is created.
I’ve tried different patterns and the following are working ok (log append to
nifi-app.log):

1. 'nifi-app_%d.%i.log.gz'
2. 'nifi-app_%d{-MM-dd_HH}.%i.log.gz'

But either 'nifi-app_%d{-MM-dd_HH}.log.gz' nor nifi-app_%d.log.gz are
working.

It there a reason for that? Sounds like the ‘%i’ in the fileNamePattern is
required, whereas it is not the case for ‘nifi-user.log’ for example, this
configuration ‘nifi-app_%d.log.gz’ is working ok, logs are appended into
‘nifi-user.log’

Thanks for any help

Emmanuel
C2 - Restricted

#
" Ce courriel et les documents qui lui sont joints peuvent contenir des
informations confidentielles, être soumis aux règlementations relatives au
contrôle des exportations ou ayant un caractère privé. S'ils ne vous sont pas
destinés, nous vous signalons qu'il est strictement interdit de les divulguer,
de les reproduire ou d'en utiliser de quelque manière que ce soit le contenu.
Toute exportation ou réexportation non autorisée est interdite Si ce message
vous a été transmis par erreur, merci d'en informer l'expéditeur et de
supprimer immédiatement de votre système informatique ce courriel ainsi que
tous les documents qui y sont attachés."
**
" This e-mail and any attached documents may contain confidential or
proprietary information and may be subject to export control laws and
regulations. If you are not the intended recipient, you are notified that any
dissemination, copying of this e-mail and any attachments thereto or use of
their contents by any means whatsoever is strictly prohibited. Unauthorized
export or re-export is prohibited. If you have received this e-mail in error,
please advise the sender immediately and delete this e-mail and all attached
documents from your computer system."
#

RE: nifi-app log rotation

2022-01-18 Thread QUEVILLON EMMANUEL - EXT-SAFRAN ENGINEERING SERVICES (SAFRAN)

Hi Juan,

Thanks for your quick rely.
The partition where the logs are stored is not full, and as soon as I change 
‘fileNamePattern’ to a working on (default or 'nifi-app_%d.%i.log.gz' for 
example), I see lines appending to nifi-app.log file (tail –f nifi-app.log)
So there’s something I don’t get.

Emmanuel
C2 - Restricted

De : Juan Pablo Gardella 
Envoyé : mardi 18 janvier 2022 11:00
À : users@nifi.apache.org
Objet : Re: nifi-app log rotation

Hi Emmanuel,

That is not related to Nifi itself, I suggest to chek logback manual or, if the 
disk partition where are you storing logs are not full.

Juan

On Tue, 18 Jan 2022 at 06:54, QUEVILLON EMMANUEL - EXT-SAFRAN ENGINEERING 
SERVICES (SAFRAN) 
mailto:emmanuel.quevillon.e...@safrangroup.com>>
 wrote:
Hi guys,

I’m trying to configure the nifi-app.log rotation process by decreasing the 
rotation time and compressing the final rotated file.
To do this, I’ve updated ‘conf/logback.xml’ file by this default configuration:


   …

${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{-MM-dd_HH}.%i.log
…


To this one


…

${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d.log.gz
   …


However, since I’ve updated `conf/logback.xml`, nifi-appl.log is not updated 
anymore and not new file is created.
I’ve tried different patterns and the following are working ok (log append to 
nifi-app.log):

1. 'nifi-app_%d.%i.log.gz'
2. 'nifi-app_%d{-MM-dd_HH}.%i.log.gz'

But either 'nifi-app_%d{-MM-dd_HH}.log.gz' nor nifi-app_%d.log.gz are 
working.

It there a reason for that? Sounds like the ‘%i’ in the fileNamePattern is 
required, whereas it is not the case for ‘nifi-user.log’ for example, this 
configuration ‘nifi-app_%d.log.gz’ is working ok, logs are appended into 
‘nifi-user.log’

Thanks for any help

Emmanuel
C2 - Restricted

#
" Ce courriel et les documents qui lui sont joints peuvent contenir des 
informations confidentielles, être soumis aux règlementations relatives au 
contrôle des exportations ou ayant un caractère privé. S'ils ne vous sont pas 
destinés, nous vous signalons qu'il est strictement interdit de les divulguer, 
de les reproduire ou d'en utiliser de quelque manière que ce soit le contenu. 
Toute exportation ou réexportation non autorisée est interdite Si ce message 
vous a été transmis par erreur, merci d'en informer l'expéditeur et de 
supprimer immédiatement de votre système informatique ce courriel ainsi que 
tous les documents qui y sont attachés."
**
" This e-mail and any attached documents may contain confidential or 
proprietary information and may be subject to export control laws and 
regulations. If you are not the intended recipient, you are notified that any 
dissemination, copying of this e-mail and any attachments thereto or use of 
their contents by any means whatsoever is strictly prohibited. Unauthorized 
export or re-export is prohibited. If you have received this e-mail in error, 
please advise the sender immediately and delete this e-mail and all attached 
documents from your computer system."
#

#
" Ce courriel et les documents qui lui sont joints peuvent contenir des 
informations confidentielles, être soumis aux règlementations relatives au 
contrôle des exportations ou ayant un caractère privé. S'ils ne vous sont pas 
destinés, nous vous signalons qu'il est strictement interdit de les divulguer, 
de les reproduire ou d'en utiliser de quelque manière que ce soit le contenu. 
Toute exportation ou réexportation non autorisée est interdite Si ce message 
vous a été transmis par erreur, merci d'en informer l'expéditeur et de 
supprimer immédiatement de votre système informatique ce courriel ainsi que 
tous les documents qui y sont attachés."
**
" This e-mail and any attached documents may contain confidential or 
proprietary information and may be subject to export control laws and 
regulations. If you are not the intended recipient, you are notified that any 
dissemination, copying of this e-mail and any attachments thereto or use of 
their contents by any means whatsoever is strictly prohibited. Unauthorized 
export or re-export is prohibited. If you have received this e-mail in error, 
please advise the sender immediately and delete this e-mail and all attached 
documents from your computer system."
#

RE: nifi-app log rotation

2022-01-18 Thread Isha Lamboo

Hi Emmanuel,

The filename pattern needs to match with the rolling policy. You omitted that
line in your snippet, but by default, the nifi-app looks like this:

SizeAndTimeBased means you have to use both %d (time) and %i (size). If you
only want to use one of those, change the policy to TimeBased or SizeBased.

Regards,

Isha

Van: QUEVILLON EMMANUEL - EXT-SAFRAN ENGINEERING SERVICES (SAFRAN)

Verzonden: dinsdag 18 januari 2022 10:54
Aan: users@nifi.apache.org
Onderwerp: nifi-app log rotation

Hi guys,

…

${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{-MM-dd_HH}.%i.log
…

To this one

…

${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d.log.gz
…

1. 'nifi-app_%d.%i.log.gz'
2. 'nifi-app_%d{-MM-dd_HH}.%i.log.gz'

But either 'nifi-app_%d{-MM-dd_HH}.log.gz' nor nifi-app_%d.log.gz are
working.

Thanks for any help

Emmanuel
C2 - Restricted

Re: nifi-app log rotation

2022-01-18 Thread Juan Pablo Gardella

Hi Emmanuel,

That is not related to Nifi itself, I suggest to chek logback manual or, if
the disk partition where are you storing logs are not full.

Juan

On Tue, 18 Jan 2022 at 06:54, QUEVILLON EMMANUEL - EXT-SAFRAN ENGINEERING
SERVICES (SAFRAN)  wrote:

> Hi guys,
>
>
>
> I’m trying to configure the nifi-app.log rotation process by decreasing
> the rotation time and compressing the final rotated file.
>
> To do this, I’ve updated ‘conf/logback.xml’ file by this default
> configuration:
>
>
>
>  class="ch.qos.logback.core.rolling.RollingFileAppender">
>
>…
>
>
> ${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{-MM-dd_HH}.%i.log
>
> …
>
> 
>
>
>
> To this one
>
>
>
>  class="ch.qos.logback.core.rolling.RollingFileAppender">
>
> …
>
>
> ${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d.log.gz
>
>…
>
> 
>
>
>
> However, since I’ve updated `conf/logback.xml`, nifi-appl.log is not
> updated anymore and not new file is created.
>
> I’ve tried different patterns and the following are working ok (log append
> to nifi-app.log):
>
>
>
> 1. 'nifi-app_%d.%i.log.gz'
>
> 2. 'nifi-app_%d{-MM-dd_HH}.%i.log.gz'
>
>
>
> But either 'nifi-app_%d{-MM-dd_HH}.log.gz' nor nifi-app_%d.log.gz are
> working.
>
>
>
> It there a reason for that? Sounds like the ‘%i’ in the fileNamePattern is
> required, whereas it is not the case for ‘nifi-user.log’ for example, this
> configuration ‘nifi-app_%d.log.gz’ is working ok, logs are appended into
> ‘nifi-user.log’
>
>
>
> Thanks for any help
>
>
>
> Emmanuel
>
> C2 - Restricted
>
>
> #
> " Ce courriel et les documents qui lui sont joints peuvent contenir des
> informations confidentielles, être soumis aux règlementations relatives au
> contrôle des exportations ou ayant un caractère privé. S'ils ne vous sont
> pas destinés, nous vous signalons qu'il est strictement interdit de les
> divulguer, de les reproduire ou d'en utiliser de quelque manière que ce
> soit le contenu. Toute exportation ou réexportation non autorisée est
> interdite Si ce message vous a été transmis par erreur, merci d'en informer
> l'expéditeur et de supprimer immédiatement de votre système informatique ce
> courriel ainsi que tous les documents qui y sont attachés."
> **
> " This e-mail and any attached documents may contain confidential or
> proprietary information and may be subject to export control laws and
> regulations. If you are not the intended recipient, you are notified that
> any dissemination, copying of this e-mail and any attachments thereto or
> use of their contents by any means whatsoever is strictly prohibited.
> Unauthorized export or re-export is prohibited. If you have received this
> e-mail in error, please advise the sender immediately and delete this
> e-mail and all attached documents from your computer system."
> #
>

nifi-app log rotation

2022-01-18 Thread QUEVILLON EMMANUEL - EXT-SAFRAN ENGINEERING SERVICES (SAFRAN)

Hi guys,

I'm trying to configure the nifi-app.log rotation process by decreasing the 
rotation time and compressing the final rotated file.
To do this, I've updated 'conf/logback.xml' file by this default configuration:


   ...

${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{-MM-dd_HH}.%i.log
...


To this one


...

${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d.log.gz
   ...


However, since I've updated `conf/logback.xml`, nifi-appl.log is not updated 
anymore and not new file is created.
I've tried different patterns and the following are working ok (log append to 
nifi-app.log):

1. 'nifi-app_%d.%i.log.gz'
2. 'nifi-app_%d{-MM-dd_HH}.%i.log.gz'

But either 'nifi-app_%d{-MM-dd_HH}.log.gz' nor nifi-app_%d.log.gz are 
working.

It there a reason for that? Sounds like the '%i' in the fileNamePattern is 
required, whereas it is not the case for 'nifi-user.log' for example, this 
configuration 'nifi-app_%d.log.gz' is working ok, logs are appended into 
'nifi-user.log'

Thanks for any help

Emmanuel
C2 - Restricted

#
" Ce courriel et les documents qui lui sont joints peuvent contenir des 
informations confidentielles, être soumis aux règlementations relatives au 
contrôle des exportations ou ayant un caractère privé. S'ils ne vous sont pas 
destinés, nous vous signalons qu'il est strictement interdit de les divulguer, 
de les reproduire ou d'en utiliser de quelque manière que ce soit le contenu. 
Toute exportation ou réexportation non autorisée est interdite Si ce message 
vous a été transmis par erreur, merci d'en informer l'expéditeur et de 
supprimer immédiatement de votre système informatique ce courriel ainsi que 
tous les documents qui y sont attachés."
**
" This e-mail and any attached documents may contain confidential or 
proprietary information and may be subject to export control laws and 
regulations. If you are not the intended recipient, you are notified that any 
dissemination, copying of this e-mail and any attachments thereto or use of 
their contents by any means whatsoever is strictly prohibited. Unauthorized 
export or re-export is prohibited. If you have received this e-mail in error, 
please advise the sender immediately and delete this e-mail and all attached 
documents from your computer system."
#

NiFi V1.15.2 Conversion to Avro from JSON

2022-01-18 Thread nathan.english

Hi There,

We've recently upgraded to NiFi V1.15.2 from V1.12.1 and have noticed a 
difference between the JSON Tree Reader Controller Service, which is a slight 
annoyance.

When we transform our records, we have some fields that will exist or won't 
after the transformation, depending on whether there is a value for the field. 
Previously with NiFi v1.12.1, when we would convert these records to Avro using 
the Avro Writer Controller and JSON Tree Reader Services (Both with Schemas 
Set), these fields would be created and set with the default value in the Avro 
Schema. But in NiFi v1.15.2, the conversion fails with "null of boolean in 
field field_name of Avro.Schema.Name", it doesn't matter which field, type or 
default value. It all seems to fail if the field doesn't exist.

Interestingly, there isn't an issue when I set the JSON Tree Reader to Infer 
Schema instead of 'use Schema Name' with a Schema Registry.

Annoyingly before we convert to Avro from JSON, we go through a Validate Schema 
Processor, which uses the same Schema, and JSON Tree Reader but instead uses a 
JSON Writer and successfully validates the record against Schema!

It seems similar to the NIFI-9335 issue on Jira [1], but maybe with the JSON 
Tree Reader Controller service instead?

I have attached a sample flow to reproduce the issue.

I didn't know if it was worth me raising a bug ticket?

Kind Regards,

Nathan

[1] https://issues.apache.org/jira/browse/NIFI-9335



NiFi_Field_Conversion_Issue.xml
Description: NiFi_Field_Conversion_Issue.xml

[ANNOUNCE] Apache NiFi 1.15.3 release

Re: PutSQL in combination with ConvertJSONToSQL gives java.sql.SQLException: Invalid column type for Orcale DataType BINARY_DOUBLE

PutSQL in combination with ConvertJSONToSQL gives java.sql.SQLException: Invalid column type for Orcale DataType BINARY_DOUBLE

RE: nifi-app log rotation

RE: nifi-app log rotation

RE: nifi-app log rotation

Re: nifi-app log rotation

nifi-app log rotation

NiFi V1.15.2 Conversion to Avro from JSON

9 matches

Site Navigation

Mail list logo

Footer information