[jira] [Updated] (NIFI-8108) PublishKafka should allow to input message timestamp by attribute
[ https://issues.apache.org/jira/browse/NIFI-8108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] HondaWei updated NIFI-8108: --- Summary: PublishKafka should allow to input message timestamp by attribute (was: PublishKafka should allow to input message timestamp) > PublishKafka should allow to input message timestamp by attribute > - > > Key: NIFI-8108 > URL: https://issues.apache.org/jira/browse/NIFI-8108 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.12.1 >Reporter: HondaWei >Assignee: HondaWei >Priority: Minor > > PublishKafka should allow to input message timestamp. In some cases, users > want to keep the original timestamp from source message. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-8108) PublishKafka should allow to input message timestamp
HondaWei created NIFI-8108: -- Summary: PublishKafka should allow to input message timestamp Key: NIFI-8108 URL: https://issues.apache.org/jira/browse/NIFI-8108 Project: Apache NiFi Issue Type: Improvement Affects Versions: 1.12.1 Reporter: HondaWei Assignee: HondaWei PublishKafka should allow to input message timestamp. In some cases, users want to keep the original timestamp from source message. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (NIFI-8045) Avro Schemas should support 'fixed' field types
[
https://issues.apache.org/jira/browse/NIFI-8045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
HondaWei reassigned NIFI-8045:
--
Assignee: HondaWei
> Avro Schemas should support 'fixed' field types
> ---
>
> Key: NIFI-8045
> URL: https://issues.apache.org/jira/browse/NIFI-8045
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.11.4
>Reporter: Chris Sampson
>Assignee: HondaWei
>Priority: Major
> Attachments: avro-fixed.json, avro-fixed.xml
>
>
> Attempts to use the Avro schema's "fixed" type result in an error:
> {code:java}
> 2020-11-25 15:36:28,187 ERROR [Timer-Driven Process Thread-5]
> o.a.n.processors.standard.ValidateRecord
> ValidateRecord[id=0008e1f7-0176-1000--dc31ab2d] Failed to process
> StandardFlowFileRecord[uuid=fd2c0bb8-6d2e-4468-a1da-cb862ff33f61,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1606318132008-3243,
> container=default, section=171], offset=552175,
> length=47],offset=0,name=fd2c0bb8-6d2e-4468-a1da-cb862ff33f61,size=47]; will
> route to failure: org.apache.avro.SchemaParseException: "fixed" is not a
> defined name. The type of the "id" field must be a defined name or a {"type":
> ...} expression.
> org.apache.avro.SchemaParseException: "fixed" is not a defined name. The type
> of the "id" field must be a defined name or a {"type": ...} expression.
> at org.apache.avro.Schema.parse(Schema.java:1265)
> at org.apache.avro.Schema$Parser.parse(Schema.java:1032)
> at org.apache.avro.Schema$Parser.parse(Schema.java:1020)
> at
> org.apache.nifi.processors.standard.ValidateRecord.getValidationSchema(ValidateRecord.java:478)
> at
> org.apache.nifi.processors.standard.ValidateRecord.onTrigger(ValidateRecord.java:272)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> Attached template/flow re-create this with a simply generated JSON content
> passed to a ValidateRecord processor.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7899) InvokeHTTP does not timeout
[ https://issues.apache.org/jira/browse/NIFI-7899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17244461#comment-17244461 ] HondaWei commented on NIFI-7899: Hi Do you try to test your flow by invoking another restful service? Maybe it is the problem of remote service. > InvokeHTTP does not timeout > --- > > Key: NIFI-7899 > URL: https://issues.apache.org/jira/browse/NIFI-7899 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.11.4 > Environment: Ubuntu 18.04. Nifi 1.11.4. > 4 core, 8GB mem. Java set to 4GB mem >Reporter: Jens M Kofoed >Priority: Major > > We have some issues with the InvokeHTTP process. It "randomly" hangs in the > process without timing out. The processor shows that there are 1 task running > (upper right corner) and it can runs for hours without any outputs, but with > multiply flowfiles in the queue. > Trying to stop it takes forever so I have to terminate it. restart the > processor and everything works fine for a long time. until next time it hangs. > Our configuration of the process is as follow: > Penalty: 30s, Yield: 1s, > Scheduling: timer driven, Concurrent Task: 1, Run Schedule: 0, Run duration: > 0 > HTTP Method: GET > Connection timeout: 5s > Read timeout: 15s > Idle Timeout: 5m > Max idle Connection: 5 > I could not find any other bug reports here. but there are other people > metion same issues: > [https://webcache.googleusercontent.com/search?q=cache:LMqcymQiM-IJ:https://community.cloudera.com/t5/Support-Questions/InvokeHTTP-randomly-hangs/td-p/296184+=1=da=clnk=dk] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (NIFI-7215) ScanHbase : Get row key when set to col-qual-and-val in json format
[ https://issues.apache.org/jira/browse/NIFI-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] HondaWei reassigned NIFI-7215: -- Assignee: HondaWei > ScanHbase : Get row key when set to col-qual-and-val in json format > > > Key: NIFI-7215 > URL: https://issues.apache.org/jira/browse/NIFI-7215 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: CHANDAN KUMAR >Assignee: HondaWei >Priority: Major > Labels: triage > > ScanHBase processor does not return the row key, when configured to use > col-qual-and-val JSON Format. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-6619) RouteOnAttribute: Create new Routing Strategy to route only first rule that is true
[ https://issues.apache.org/jira/browse/NIFI-6619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16940054#comment-16940054 ] HondaWei commented on NIFI-6619: [~joewitt] [~skin27] If we change the structure of storing relationships from HashSet to LinkedHashSet, so we can have the order of properties. So we can have this change in RouteOnAttribute. What do you think? > RouteOnAttribute: Create new Routing Strategy to route only first rule that > is true > --- > > Key: NIFI-6619 > URL: https://issues.apache.org/jira/browse/NIFI-6619 > Project: Apache NiFi > Issue Type: Improvement > Components: Configuration >Affects Versions: 1.9.2 >Reporter: Raymond >Priority: Major > Attachments: image-2019-09-12-22-15-51-027.png > > > Currently the RouteOnAttribute has the strategy "Route to Property name". > The behavior is that for each rule that is true a message (clone of flowfile) > will be sent to the next step. > I would like to have another strategy: > "Route to first matched Property name" or "Route to Property name by first > match" or Route to first Property name which evaluates true". > This will ensure that next step gets exactly one message. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-6619) RouteOnAttribute: Create new Routing Strategy to route only first rule that is true
[ https://issues.apache.org/jira/browse/NIFI-6619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928587#comment-16928587 ] HondaWei commented on NIFI-6619: It's interesting. I would like to add a strategy which name is RouteByOrder. It will evaluate the following properties and route the message to the first match. If you have any suggestion, please let me know. . !image-2019-09-12-22-15-51-027.png! > RouteOnAttribute: Create new Routing Strategy to route only first rule that > is true > --- > > Key: NIFI-6619 > URL: https://issues.apache.org/jira/browse/NIFI-6619 > Project: Apache NiFi > Issue Type: Improvement > Components: Configuration >Affects Versions: 1.9.2 >Reporter: Raymond >Priority: Major > Attachments: image-2019-09-12-22-15-51-027.png > > > Currently the RouteOnAttribute has the strategy "Route to Property name". > The behavior is that for each rule that is true a message (clone of flowfile) > will be sent to the next step. > I would like to have another strategy: > "Route to first matched Property name" or "Route to Property name by first > match" or Route to first Property name which evaluates true". > This will ensure that next step gets exactly one message. > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (NIFI-6619) RouteOnAttribute: Create new Routing Strategy to route only first rule that is true
[ https://issues.apache.org/jira/browse/NIFI-6619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] HondaWei updated NIFI-6619: --- Attachment: image-2019-09-12-22-15-51-027.png > RouteOnAttribute: Create new Routing Strategy to route only first rule that > is true > --- > > Key: NIFI-6619 > URL: https://issues.apache.org/jira/browse/NIFI-6619 > Project: Apache NiFi > Issue Type: Improvement > Components: Configuration >Affects Versions: 1.9.2 >Reporter: Raymond >Priority: Major > Attachments: image-2019-09-12-22-15-51-027.png > > > Currently the RouteOnAttribute has the strategy "Route to Property name". > The behavior is that for each rule that is true a message (clone of flowfile) > will be sent to the next step. > I would like to have another strategy: > "Route to first matched Property name" or "Route to Property name by first > match" or Route to first Property name which evaluates true". > This will ensure that next step gets exactly one message. > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6480) PutORC/PutParquet can't overwrite file even if set 'Overwrite Files' to true
[ https://issues.apache.org/jira/browse/NIFI-6480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920569#comment-16920569 ] HondaWei commented on NIFI-6480: OK, I didn't notice it. It's good. > PutORC/PutParquet can't overwrite file even if set 'Overwrite Files' to true > > > Key: NIFI-6480 > URL: https://issues.apache.org/jira/browse/NIFI-6480 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: archon gum >Assignee: archon gum >Priority: Major > Attachments: Snipaste_2019-07-16_11-07-05.jpg > > Time Spent: 1h > Remaining Estimate: 0h > > !Snipaste_2019-07-16_11-07-05.jpg! > > Solution: > Reference to > [PutHDFS|https://github.com/apache/nifi/blob/72244d09ff193131119c10492b9327af35a64f02/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java], > delete dest file before rename temp file to dest file. > Tested on my local machine. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6480) PutORC/PutParquet can't overwrite file even if set 'Overwrite Files' to true
[ https://issues.apache.org/jira/browse/NIFI-6480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919679#comment-16919679 ] HondaWei commented on NIFI-6480: I'll tackle this and send PR ASAP. > PutORC/PutParquet can't overwrite file even if set 'Overwrite Files' to true > > > Key: NIFI-6480 > URL: https://issues.apache.org/jira/browse/NIFI-6480 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: archon gum >Assignee: archon gum >Priority: Major > Attachments: Snipaste_2019-07-16_11-07-05.jpg > > Time Spent: 50m > Remaining Estimate: 0h > > !Snipaste_2019-07-16_11-07-05.jpg! > > Solution: > Reference to > [PutHDFS|https://github.com/apache/nifi/blob/72244d09ff193131119c10492b9327af35a64f02/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java], > delete dest file before rename temp file to dest file. > Tested on my local machine. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Comment Edited] (NIFI-6483) Apache Nifi (version 1.9.2) error when update data in a table (SQL server, MySQL, Postgresql) (it is running well with insert and update operations)
[ https://issues.apache.org/jira/browse/NIFI-6483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919545#comment-16919545 ] HondaWei edited comment on NIFI-6483 at 8/30/19 1:44 PM: - I traced the code, it failed if your column name in record schema is the lower case even though set "Translate Field Names" to false. I think it will work well if you set column name in upper case in record schema and set "Translate Field Names" to false. was (Author: hondawei): I traced the code, it still failed even though set "Translate Field Names" to false due to case-sensitive comparison of column name comparison. > Apache Nifi (version 1.9.2) error when update data in a table (SQL server, > MySQL, Postgresql) (it is running well with insert and update operations) > > > Key: NIFI-6483 > URL: https://issues.apache.org/jira/browse/NIFI-6483 > Project: Apache NiFi > Issue Type: Bug > Components: Configuration >Affects Versions: 1.9.2 > Environment: error when update data in a table (SQL server, MySQL, > Postgresql) (it is running well with insert and update operations) >Reporter: thuy le >Assignee: HondaWei >Priority: Major > Attachments: image-2019-07-25-11-32-19-469.png > > > Apache Nifi (version 1.9.2) error when update data in a table (SQL server, > MySQL, Postgresql) (it is running well with insert and update operations) > > !https://community.hortonworks.com/storage/attachments/110025-1563909312644.png|width=736,height=242! > > _With Delete they do correct_ > !https://community.hortonworks.com/storage/attachments/110007-1563910599280.png|width=932,height=127! > !https://community.hortonworks.com/storage/attachments/110013-1563910653675.png|width=828,height=156! > > > *+with Update+*_: They automatic change Primarykey from_ *product_key* _to > PRODUCTKEY (_I tested with *_PutSQL_* and *_PutDatabaseRecord_* it have the > same issue) > !https://community.hortonworks.com/storage/attachments/110006-1563910561236.png|width=703,height=94!!https://community.hortonworks.com/storage/attachments/110005-1563910541777.png|width=605,height=83! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6483) Apache Nifi (version 1.9.2) error when update data in a table (SQL server, MySQL, Postgresql) (it is running well with insert and update operations)
[ https://issues.apache.org/jira/browse/NIFI-6483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919545#comment-16919545 ] HondaWei commented on NIFI-6483: I traced the code, it still failed even though set "Translate Field Names" to false due to case-sensitive comparison of column name comparison. > Apache Nifi (version 1.9.2) error when update data in a table (SQL server, > MySQL, Postgresql) (it is running well with insert and update operations) > > > Key: NIFI-6483 > URL: https://issues.apache.org/jira/browse/NIFI-6483 > Project: Apache NiFi > Issue Type: Bug > Components: Configuration >Affects Versions: 1.9.2 > Environment: error when update data in a table (SQL server, MySQL, > Postgresql) (it is running well with insert and update operations) >Reporter: thuy le >Priority: Major > Attachments: image-2019-07-25-11-32-19-469.png > > > Apache Nifi (version 1.9.2) error when update data in a table (SQL server, > MySQL, Postgresql) (it is running well with insert and update operations) > > !https://community.hortonworks.com/storage/attachments/110025-1563909312644.png|width=736,height=242! > > _With Delete they do correct_ > !https://community.hortonworks.com/storage/attachments/110007-1563910599280.png|width=932,height=127! > !https://community.hortonworks.com/storage/attachments/110013-1563910653675.png|width=828,height=156! > > > *+with Update+*_: They automatic change Primarykey from_ *product_key* _to > PRODUCTKEY (_I tested with *_PutSQL_* and *_PutDatabaseRecord_* it have the > same issue) > !https://community.hortonworks.com/storage/attachments/110006-1563910561236.png|width=703,height=94!!https://community.hortonworks.com/storage/attachments/110005-1563910541777.png|width=605,height=83! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (NIFI-6483) Apache Nifi (version 1.9.2) error when update data in a table (SQL server, MySQL, Postgresql) (it is running well with insert and update operations)
[ https://issues.apache.org/jira/browse/NIFI-6483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] HondaWei reassigned NIFI-6483: -- Assignee: HondaWei > Apache Nifi (version 1.9.2) error when update data in a table (SQL server, > MySQL, Postgresql) (it is running well with insert and update operations) > > > Key: NIFI-6483 > URL: https://issues.apache.org/jira/browse/NIFI-6483 > Project: Apache NiFi > Issue Type: Bug > Components: Configuration >Affects Versions: 1.9.2 > Environment: error when update data in a table (SQL server, MySQL, > Postgresql) (it is running well with insert and update operations) >Reporter: thuy le >Assignee: HondaWei >Priority: Major > Attachments: image-2019-07-25-11-32-19-469.png > > > Apache Nifi (version 1.9.2) error when update data in a table (SQL server, > MySQL, Postgresql) (it is running well with insert and update operations) > > !https://community.hortonworks.com/storage/attachments/110025-1563909312644.png|width=736,height=242! > > _With Delete they do correct_ > !https://community.hortonworks.com/storage/attachments/110007-1563910599280.png|width=932,height=127! > !https://community.hortonworks.com/storage/attachments/110013-1563910653675.png|width=828,height=156! > > > *+with Update+*_: They automatic change Primarykey from_ *product_key* _to > PRODUCTKEY (_I tested with *_PutSQL_* and *_PutDatabaseRecord_* it have the > same issue) > !https://community.hortonworks.com/storage/attachments/110006-1563910561236.png|width=703,height=94!!https://community.hortonworks.com/storage/attachments/110005-1563910541777.png|width=605,height=83! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6483) Apache Nifi (version 1.9.2) error when update data in a table (SQL server, MySQL, Postgresql) (it is running well with insert and update operations)
[ https://issues.apache.org/jira/browse/NIFI-6483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919528#comment-16919528 ] HondaWei commented on NIFI-6483: Hi [~thuylevn] Can you post your properties of PutDatabaseRecord? Do you set "Translate Field Names" to true? If this property is true, it removes underscore in the column name. > Apache Nifi (version 1.9.2) error when update data in a table (SQL server, > MySQL, Postgresql) (it is running well with insert and update operations) > > > Key: NIFI-6483 > URL: https://issues.apache.org/jira/browse/NIFI-6483 > Project: Apache NiFi > Issue Type: Bug > Components: Configuration >Affects Versions: 1.9.2 > Environment: error when update data in a table (SQL server, MySQL, > Postgresql) (it is running well with insert and update operations) >Reporter: thuy le >Priority: Major > Attachments: image-2019-07-25-11-32-19-469.png > > > Apache Nifi (version 1.9.2) error when update data in a table (SQL server, > MySQL, Postgresql) (it is running well with insert and update operations) > > !https://community.hortonworks.com/storage/attachments/110025-1563909312644.png|width=736,height=242! > > _With Delete they do correct_ > !https://community.hortonworks.com/storage/attachments/110007-1563910599280.png|width=932,height=127! > !https://community.hortonworks.com/storage/attachments/110013-1563910653675.png|width=828,height=156! > > > *+with Update+*_: They automatic change Primarykey from_ *product_key* _to > PRODUCTKEY (_I tested with *_PutSQL_* and *_PutDatabaseRecord_* it have the > same issue) > !https://community.hortonworks.com/storage/attachments/110006-1563910561236.png|width=703,height=94!!https://community.hortonworks.com/storage/attachments/110005-1563910541777.png|width=605,height=83! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-4542) Directory Created indicator in putHDFS
[ https://issues.apache.org/jira/browse/NIFI-4542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899425#comment-16899425 ] HondaWei commented on NIFI-4542: [~mgupt...@sapient.com] I'm going to add a new attribute to indicate if the target directory is created this time. Thanks, Honda > Directory Created indicator in putHDFS > -- > > Key: NIFI-4542 > URL: https://issues.apache.org/jira/browse/NIFI-4542 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Manish Gupta >Priority: Minor > Labels: easyfix, usability > > Many times after using a putHDFS processor, it's required to call a one time > processing flow (like create database or create table or add partition, > etc.). Would be great if putHDFS processor can signal a "new directory > created" event as a flowfile attribute, something like - targetDirCreated for > success relationship. Most of the time, for every project, we have to write > this custom putHDFS processor. > Please let me know if more details are required. > Thanks, > Manish -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (NIFI-6132) NiFi defaults to using G1GC but now also uses the Write-Ahead Provenance Repo by default
[
https://issues.apache.org/jira/browse/NIFI-6132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883942#comment-16883942
]
HondaWei commented on NIFI-6132:
Hi [~markap14]
I would like to change the seeting in bootstrap.conf.
{code:java}
java.arg.13=-XX:+UseG1GC{code}
{{Should we use Parallel or CMS? }}
{{thanks}}
> NiFi defaults to using G1GC but now also uses the Write-Ahead Provenance Repo
> by default
>
>
> Key: NIFI-6132
> URL: https://issues.apache.org/jira/browse/NIFI-6132
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Mark Payne
>Priority: Blocker
> Labels: beginner, easy, newbie
> Fix For: 1.10.0
>
>
> It is not recommended to use G1GC along with the Write-Ahead Provenance
> Repository due to known bugs in the G1GC in versions of Java earlier than
> 1.9. The default for the Provenance Repository was changed to the Write-Ahead
> Provenance Repository, but the garbage collector still defaults to G1GC.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Assigned] (NIFI-6130) Add More information about exceptions from record readers/writers
[ https://issues.apache.org/jira/browse/NIFI-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] HondaWei reassigned NIFI-6130: -- Assignee: HondaWei > Add More information about exceptions from record readers/writers > - > > Key: NIFI-6130 > URL: https://issues.apache.org/jira/browse/NIFI-6130 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.9.0 >Reporter: Nimrod Avni >Assignee: HondaWei >Priority: Major > Attachments: image-2019-03-18-21-46-22-468.png, > image-2019-03-18-21-47-16-888.png > > > When using record readers and writers i ran into a problem where i tried to > give it a CSV with the same header twice, i.e something like > -- > header1,header2,header1 > a,b,c > -- > (obviously not real data) > When the reader tried to parse the flowfile as records it threw this error: > "Failed to read Header line from CSV" > Which is not very indicative, when i went ahead and debugged the source code > i could see the real error is: > java.lang.IllegalArgumentException: The header contains a duplicate name: > "header1" in [header1, header2, header1] > (Pictures are provided) > This is a case i ran into recently. > I also have another case where tried to read an avro file which has a '#' > (hashtag) character in one of the headers, which is illegal in the avro > schema defenition. > But the error i got there was : "failed to parse incoming data", > And again, when i debugged the code i found the real error > What the main point is to give a more indicative error message for record > readers / writers, which not only gives the error itself but its cause > I could probably look up many more reasons readers / writers could fail and > they will not give a good error message, but i believe these 2 are fine as an > example -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (NIFI-6130) Add More information about exceptions from record readers/writers
[ https://issues.apache.org/jira/browse/NIFI-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883649#comment-16883649 ] HondaWei commented on NIFI-6130: Hi [~Max Kelada] I'll try to fix cases that you mentioned. If you have other cases, please let me know. thanks. > Add More information about exceptions from record readers/writers > - > > Key: NIFI-6130 > URL: https://issues.apache.org/jira/browse/NIFI-6130 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.9.0 >Reporter: Nimrod Avni >Priority: Major > Attachments: image-2019-03-18-21-46-22-468.png, > image-2019-03-18-21-47-16-888.png > > > When using record readers and writers i ran into a problem where i tried to > give it a CSV with the same header twice, i.e something like > -- > header1,header2,header1 > a,b,c > -- > (obviously not real data) > When the reader tried to parse the flowfile as records it threw this error: > "Failed to read Header line from CSV" > Which is not very indicative, when i went ahead and debugged the source code > i could see the real error is: > java.lang.IllegalArgumentException: The header contains a duplicate name: > "header1" in [header1, header2, header1] > (Pictures are provided) > This is a case i ran into recently. > I also have another case where tried to read an avro file which has a '#' > (hashtag) character in one of the headers, which is illegal in the avro > schema defenition. > But the error i got there was : "failed to parse incoming data", > And again, when i debugged the code i found the real error > What the main point is to give a more indicative error message for record > readers / writers, which not only gives the error itself but its cause > I could probably look up many more reasons readers / writers could fail and > they will not give a good error message, but i believe these 2 are fine as an > example -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (NIFI-4542) Directory Created indicator in putHDFS
[ https://issues.apache.org/jira/browse/NIFI-4542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16879867#comment-16879867 ] HondaWei commented on NIFI-4542: Hi [~mgupt...@sapient.com] Do you mean you want a new relationship or an attribute in output flowfile to indicate if the target HDFS directory is created in this processing? > Directory Created indicator in putHDFS > -- > > Key: NIFI-4542 > URL: https://issues.apache.org/jira/browse/NIFI-4542 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Manish Gupta >Priority: Minor > Labels: easyfix, usability > > Many times after using a putHDFS processor, it's required to call a one time > processing flow (like create database or create table or add partition, > etc.). Would be great if putHDFS processor can signal a "new directory > created" event as a flowfile attribute, something like - targetDirCreated for > success relationship. Most of the time, for every project, we have to write > this custom putHDFS processor. > Please let me know if more details are required. > Thanks, > Manish -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-6271) ExecuteSQL incoming flowfile attributes not copied into output flowfiles when Output Batch Size is set
[
https://issues.apache.org/jira/browse/NIFI-6271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16879856#comment-16879856
]
HondaWei commented on NIFI-6271:
I've send pull request. Thanks!
> ExecuteSQL incoming flowfile attributes not copied into output flowfiles when
> Output Batch Size is set
> --
>
> Key: NIFI-6271
> URL: https://issues.apache.org/jira/browse/NIFI-6271
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.9.2
>Reporter: Arnaud Rivero
>Priority: Major
> Labels: easyfix, features, usability
> Original Estimate: 0.5h
> Time Spent: 10m
> Remaining Estimate: 20m
>
> When using the executeSQL and executeSQLRecord processors, we can use input
> flowfiles with a certain number of attributes. If we don't set the Output
> Batch Size, all these attributes are copied to the output flowfile. However,
> if we set it, only the flowfiles from the first batch will have the
> attributes copied to. The flowfiles in the following batches will only have
> the default attributes.
> h2. Root cause
> In the source code of the method _onTrigger_ in the class
> _AbstractExecuteSQL,_ we have the following piece of code that is supposed to
> create an output flowfile and copy the original attributes into it:
> {code:java}
> FlowFile resultSetFF;
> if (fileToProcess == null) {
> resultSetFF = session.create();
> } else {
> resultSetFF = session.create(fileToProcess);
> resultSetFF = session.putAllAttributes(resultSetFF,
> fileToProcess.getAttributes());
> }
> {code}
> However the fix for the issue NIFI-6040 introduced this snippet way below in
> the same method:
>
> {code:java}
> // If we've reached the batch size, send out the flow files
> if (outputBatchSize > 0 && resultSetFlowFiles.size() >= outputBatchSize) {
> session.transfer(resultSetFlowFiles, REL_SUCCESS);
> // Need to remove the original input file if it exists
> if (fileToProcess != null) {
> session.remove(fileToProcess);
> fileToProcess = null;
> }
> session.commit();
> resultSetFlowFiles.clear();
> }
> {code}
> As you can see, it sets the variable fileToProcess to null, preventing the
> flowfiles in the next batch to copy its attributes
>
> h2.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-6271) ExecuteSQL incoming flowfile attributes not copied into output flowfiles when Output Batch Size is set
[
https://issues.apache.org/jira/browse/NIFI-6271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16879648#comment-16879648
]
HondaWei commented on NIFI-6271:
I would like to fix it. Thanks!:)
> ExecuteSQL incoming flowfile attributes not copied into output flowfiles when
> Output Batch Size is set
> --
>
> Key: NIFI-6271
> URL: https://issues.apache.org/jira/browse/NIFI-6271
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.9.2
>Reporter: Arnaud Rivero
>Priority: Major
> Labels: easyfix, features, usability
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
> When using the executeSQL and executeSQLRecord processors, we can use input
> flowfiles with a certain number of attributes. If we don't set the Output
> Batch Size, all these attributes are copied to the output flowfile. However,
> if we set it, only the flowfiles from the first batch will have the
> attributes copied to. The flowfiles in the following batches will only have
> the default attributes.
> h2. Root cause
> In the source code of the method _onTrigger_ in the class
> _AbstractExecuteSQL,_ we have the following piece of code that is supposed to
> create an output flowfile and copy the original attributes into it:
> {code:java}
> FlowFile resultSetFF;
> if (fileToProcess == null) {
> resultSetFF = session.create();
> } else {
> resultSetFF = session.create(fileToProcess);
> resultSetFF = session.putAllAttributes(resultSetFF,
> fileToProcess.getAttributes());
> }
> {code}
> However the fix for the issue NIFI-6040 introduced this snippet way below in
> the same method:
>
> {code:java}
> // If we've reached the batch size, send out the flow files
> if (outputBatchSize > 0 && resultSetFlowFiles.size() >= outputBatchSize) {
> session.transfer(resultSetFlowFiles, REL_SUCCESS);
> // Need to remove the original input file if it exists
> if (fileToProcess != null) {
> session.remove(fileToProcess);
> fileToProcess = null;
> }
> session.commit();
> resultSetFlowFiles.clear();
> }
> {code}
> As you can see, it sets the variable fileToProcess to null, preventing the
> flowfiles in the next batch to copy its attributes
>
> h2.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-1893) Add processor for validating JSON
[
https://issues.apache.org/jira/browse/NIFI-1893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16728910#comment-16728910
]
HondaWei commented on NIFI-1893:
Hi [~khill] and [~samox]
Sorry for late reply. I've sent a pull request NIFI-1893. This is my first
contribution to NiFi and please gives me some suggestion.
> Add processor for validating JSON
> -
>
> Key: NIFI-1893
> URL: https://issues.apache.org/jira/browse/NIFI-1893
> Project: Apache NiFi
> Issue Type: New Feature
>Reporter: Matt Burgess
>Priority: Major
> Attachments: image-2018-12-07-18-02-52-813.png
>
>
> NiFi has a ValidateXml processor to validate incoming XML files against a
> schema. It would be good to have one to validate JSON files as well.
> For example, an input JSON of:
> {
> name: "Test",
> timestamp: 1463499695,
> tags: {
>"host": "Test_1",
>"ip" : "1.1.1.1"
> },
> fields: {
> "cpu": 10.2,
> "load": 15.6
> }
> }
> Could be validated successfully against the following "schema":
> {
> "type": "object",
> "required": ["name", "tags", "timestamp", "fields"],
> "properties": {
> "name": {"type": "string"},
> "timestamp": {"type": "integer"},
> "tags": {"type": "object", "items": {"type": "string"}},
> "fields": { "type": "object"}
> }
> }
> There is at least one ASF-friendly library that could be used for
> implementation: https://github.com/everit-org/json-schema
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-1893) Add processor for validating JSON
[
https://issues.apache.org/jira/browse/NIFI-1893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16712596#comment-16712596
]
HondaWei commented on NIFI-1893:
Hi [~samox], thanks for your comment. This processor works well in my local
NiFi. Now the processor is reviewed by my partner and I am going to upload my
processor next week.
Thank you very much!
!image-2018-12-07-18-02-52-813.png!
> Add processor for validating JSON
> -
>
> Key: NIFI-1893
> URL: https://issues.apache.org/jira/browse/NIFI-1893
> Project: Apache NiFi
> Issue Type: New Feature
>Reporter: Matt Burgess
>Priority: Major
> Attachments: image-2018-12-07-18-02-52-813.png
>
>
> NiFi has a ValidateXml processor to validate incoming XML files against a
> schema. It would be good to have one to validate JSON files as well.
> For example, an input JSON of:
> {
> name: "Test",
> timestamp: 1463499695,
> tags: {
>"host": "Test_1",
>"ip" : "1.1.1.1"
> },
> fields: {
> "cpu": 10.2,
> "load": 15.6
> }
> }
> Could be validated successfully against the following "schema":
> {
> "type": "object",
> "required": ["name", "tags", "timestamp", "fields"],
> "properties": {
> "name": {"type": "string"},
> "timestamp": {"type": "integer"},
> "tags": {"type": "object", "items": {"type": "string"}},
> "fields": { "type": "object"}
> }
> }
> There is at least one ASF-friendly library that could be used for
> implementation: https://github.com/everit-org/json-schema
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-1893) Add processor for validating JSON
[
https://issues.apache.org/jira/browse/NIFI-1893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
HondaWei updated NIFI-1893:
---
Attachment: image-2018-12-07-18-02-52-813.png
> Add processor for validating JSON
> -
>
> Key: NIFI-1893
> URL: https://issues.apache.org/jira/browse/NIFI-1893
> Project: Apache NiFi
> Issue Type: New Feature
>Reporter: Matt Burgess
>Priority: Major
> Attachments: image-2018-12-07-18-02-52-813.png
>
>
> NiFi has a ValidateXml processor to validate incoming XML files against a
> schema. It would be good to have one to validate JSON files as well.
> For example, an input JSON of:
> {
> name: "Test",
> timestamp: 1463499695,
> tags: {
>"host": "Test_1",
>"ip" : "1.1.1.1"
> },
> fields: {
> "cpu": 10.2,
> "load": 15.6
> }
> }
> Could be validated successfully against the following "schema":
> {
> "type": "object",
> "required": ["name", "tags", "timestamp", "fields"],
> "properties": {
> "name": {"type": "string"},
> "timestamp": {"type": "integer"},
> "tags": {"type": "object", "items": {"type": "string"}},
> "fields": { "type": "object"}
> }
> }
> There is at least one ASF-friendly library that could be used for
> implementation: https://github.com/everit-org/json-schema
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (NIFI-5570) ExecuteSQL processor has error when datatype is float in Teradata
HondaWei created NIFI-5570: -- Summary: ExecuteSQL processor has error when datatype is float in Teradata Key: NIFI-5570 URL: https://issues.apache.org/jira/browse/NIFI-5570 Project: Apache NiFi Issue Type: Bug Affects Versions: 1.7.1 Reporter: HondaWei I encounter the problem which is mentioned in the following link. I would like to moidfy the code to fix this issue. https://community.hortonworks.com/questions/144420/apache-nifi-executesql-processor-error-orgapacheav.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-1893) Add processor for validating JSON
[
https://issues.apache.org/jira/browse/NIFI-1893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595846#comment-16595846
]
HondaWei commented on NIFI-1893:
Hi
I would like to use the following library to implement this processor
[https://github.com/java-json-tools/json-schema-validator]
I have already finished it, may someone assign this issue to me?
thank you!
> Add processor for validating JSON
> -
>
> Key: NIFI-1893
> URL: https://issues.apache.org/jira/browse/NIFI-1893
> Project: Apache NiFi
> Issue Type: New Feature
>Reporter: Matt Burgess
>Priority: Major
>
> NiFi has a ValidateXml processor to validate incoming XML files against a
> schema. It would be good to have one to validate JSON files as well.
> For example, an input JSON of:
> {
> name: "Test",
> timestamp: 1463499695,
> tags: {
>"host": "Test_1",
>"ip" : "1.1.1.1"
> },
> fields: {
> "cpu": 10.2,
> "load": 15.6
> }
> }
> Could be validated successfully against the following "schema":
> {
> "type": "object",
> "required": ["name", "tags", "timestamp", "fields"],
> "properties": {
> "name": {"type": "string"},
> "timestamp": {"type": "integer"},
> "tags": {"type": "object", "items": {"type": "string"}},
> "fields": { "type": "object"}
> }
> }
> There is at least one ASF-friendly library that could be used for
> implementation: https://github.com/everit-org/json-schema
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
