[jira] [Updated] (NIFI-5887) JOLTTransformJSON advanced window JSON format validation should accept EL without double quote
[
https://issues.apache.org/jira/browse/NIFI-5887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
meh updated NIFI-5887:
--
Attachment: Screenshot (93).png
> JOLTTransformJSON advanced window JSON format validation should accept EL
> without double quote
> --
>
> Key: NIFI-5887
> URL: https://issues.apache.org/jira/browse/NIFI-5887
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.7.0
>Reporter: meh
>Priority: Major
> Labels: attribute, jolt, json
> Attachments: NIFI-5887_JOLT_with_JSON_attribute_value.xml, Screenshot
> (93).png
>
>
> JOLTTransformJSON advanced configuration window allows user to design a Jolt
> spec using NiFi EL. Also it does provide a feature to set FlowFile attributes
> to confirm conversion result for testing.
> However, the advanced window spec JSON format validation doesn't allow EL
> expression without double quote. Because of this, user can not try a JOLT
> spec using EL which references a FlowFile attribute having a valid JSON
> object. See details below.
> It would be more user friendly if the advanced window allows a Jolt spec even
> if it is not a valid JSON format. Probably adding a toggle switch to disable
> JSON validation might be a solution.
>
> i have a attribute (that produced by a REST service and catched by invokeHTTP
> processor) in JSON format like this:
> {code:java}
> test => {"key":"value"}{code}
> and then i want to put it in flows JSON content using JOLT processor, my
> content is something like this:
> {code:java}
> { "id": 123, "user": "foo" }{code}
> and my JOLT specification is this:
> {code:java}
> [{ "operation": "default", "spec": { "interest": "${test}" } }]{code}
> the problem is here that, in JOLT advanced window with test attribute nifi
> cannot put json object and shown this error:
> {quote}*"Error occurred during transformation"*
> {quote}
> and when run processor this detailed error is become alerted:
> {quote}*"unable to unmarshal json to an object"*
> {quote}
>
> my desired result is this:
> {code:java}
> { "id": 123, "user": "foo", "interest": {"key":"value"} }{code}
> is this a bug or am i wrong?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5887) JOLTTransformJSON advanced window JSON format validation should accept EL without double quote
[
https://issues.apache.org/jira/browse/NIFI-5887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16753119#comment-16753119
]
meh commented on NIFI-5887:
---
cool!
Regardless of the error display, the result was obtained correctly.
!Screenshot (93).png!
> JOLTTransformJSON advanced window JSON format validation should accept EL
> without double quote
> --
>
> Key: NIFI-5887
> URL: https://issues.apache.org/jira/browse/NIFI-5887
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.7.0
>Reporter: meh
>Priority: Major
> Labels: attribute, jolt, json
> Attachments: NIFI-5887_JOLT_with_JSON_attribute_value.xml, Screenshot
> (93).png
>
>
> JOLTTransformJSON advanced configuration window allows user to design a Jolt
> spec using NiFi EL. Also it does provide a feature to set FlowFile attributes
> to confirm conversion result for testing.
> However, the advanced window spec JSON format validation doesn't allow EL
> expression without double quote. Because of this, user can not try a JOLT
> spec using EL which references a FlowFile attribute having a valid JSON
> object. See details below.
> It would be more user friendly if the advanced window allows a Jolt spec even
> if it is not a valid JSON format. Probably adding a toggle switch to disable
> JSON validation might be a solution.
>
> i have a attribute (that produced by a REST service and catched by invokeHTTP
> processor) in JSON format like this:
> {code:java}
> test => {"key":"value"}{code}
> and then i want to put it in flows JSON content using JOLT processor, my
> content is something like this:
> {code:java}
> { "id": 123, "user": "foo" }{code}
> and my JOLT specification is this:
> {code:java}
> [{ "operation": "default", "spec": { "interest": "${test}" } }]{code}
> the problem is here that, in JOLT advanced window with test attribute nifi
> cannot put json object and shown this error:
> {quote}*"Error occurred during transformation"*
> {quote}
> and when run processor this detailed error is become alerted:
> {quote}*"unable to unmarshal json to an object"*
> {quote}
>
> my desired result is this:
> {code:java}
> { "id": 123, "user": "foo", "interest": {"key":"value"} }{code}
> is this a bug or am i wrong?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5887) unable to unmarshal json to an object
[
https://issues.apache.org/jira/browse/NIFI-5887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16735768#comment-16735768
]
meh commented on NIFI-5887:
---
one other interesting thing... you cannot type caps f ("F") in JOLT advanced
mode
when you type it, cursor move to beginning of the line
> unable to unmarshal json to an object
> -
>
> Key: NIFI-5887
> URL: https://issues.apache.org/jira/browse/NIFI-5887
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: meh
>Priority: Major
> Labels: attribute, jolt, json
>
> i have a attribute (that produced by a REST service and catched by invokeHTTP
> processor) in JSON format like this:
> {code:java}
> test => {"key":"value"}{code}
> and then i want to put it in flows JSON content using JOLT processor, my
> content is something like this:
> {code:java}
> { "id": 123, "user": "foo" }{code}
> and my JOLT specification is this:
> {code:java}
> [{ "operation": "default", "spec": { "interest": "${test}" } }]{code}
> the problem is here that, in JOLT advanced window with test attribute nifi
> cannot put json object and shown this error:
> {quote}*"Error occurred during transformation"*
> {quote}
> and when run processor this detailed error is become alerted:
> {quote}*"unable to unmarshal json to an object"*
> {quote}
>
> my desired result is this:
> {code:java}
> { "id": 123, "user": "foo", "interest": {"key":"value"} }{code}
> is this a bug or am i wrong?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5887) unable to unmarshal json to an object
[
https://issues.apache.org/jira/browse/NIFI-5887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16716839#comment-16716839
]
meh commented on NIFI-5887:
---
it's worked with escaping JSON:
{code:java}
{\"key\":\"value\"}{code}
but when insert in JSON that have stringed because of double quotation (" ")
{code:java}
{ "id": 123, "user": "foo", "interest": "{\"key\":\"value\"}" }{code}
any idea?
> unable to unmarshal json to an object
> -
>
> Key: NIFI-5887
> URL: https://issues.apache.org/jira/browse/NIFI-5887
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: meh
>Priority: Major
> Labels: attribute, jolt, json
>
> i have a attribute (that produced by a REST service and catched by invokeHTTP
> processor) in JSON format like this:
> {code:java}
> test => {"key":"value"}{code}
> and then i want to put it in flows JSON content using JOLT processor, my
> content is something like this:
> {code:java}
> { "id": 123, "user": "foo" }{code}
> and my JOLT specification is this:
> {code:java}
> [{ "operation": "default", "spec": { "interest": "${test}" } }]{code}
> the problem is here that, in JOLT advanced window with test attribute nifi
> cannot put json object and shown this error:
> {quote}*"Error occurred during transformation"*
> {quote}
> and when run processor this detailed error is become alerted:
> {quote}*"unable to unmarshal json to an object"*
> {quote}
>
> my desired result is this:
> {code:java}
> { "id": 123, "user": "foo", "interest": {"key":"value"} }{code}
> is this a bug or am i wrong?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (NIFI-5887) unable to unmarshal json to an object
meh created NIFI-5887:
-
Summary: unable to unmarshal json to an object
Key: NIFI-5887
URL: https://issues.apache.org/jira/browse/NIFI-5887
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 1.7.0
Reporter: meh
i have a attribute (that produced by a REST service and catched by invokeHTTP
processor) in JSON format like this:
{code:java}
test => {"key":"value"}{code}
and then i want to put it in flows JSON content using JOLT processor, my
content is something like this:
{code:java}
{ "id": 123, "user": "foo" }{code}
and my JOLT specification is this:
{code:java}
[{ "operation": "default", "spec": { "interest": "${test}" } }]{code}
the problem is here that, in JOLT advanced window with test attribute nifi
cannot put json object and shown this error:
{quote}*"Error occurred during transformation"*
{quote}
and when run processor this detailed error is become alerted:
{quote}*"unable to unmarshal json to an object"*
{quote}
my desired result is this:
{code:java}
{ "id": 123, "user": "foo", "interest": {"key":"value"} }{code}
is this a bug or am i wrong?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-4457) "Maximum-value" not increasing when "initial.maxvalue" is set and "Maximum-value column" name is different from "id"
[ https://issues.apache.org/jira/browse/NIFI-4457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213869#comment-16213869 ] meh commented on NIFI-4457: --- [~ijokarumawak] thanks for your support. but same issue... state has NOT been updated. you can reproduce it with this files: [^tw_json.csv] [^create.txt] create a table in maria db with attached create script and insert that some data, then run nifi to take state, and then insert left record to take table change, and for next nifi query you see state is still not changing. > "Maximum-value" not increasing when "initial.maxvalue" is set and > "Maximum-value column" name is different from "id" > - > > Key: NIFI-4457 > URL: https://issues.apache.org/jira/browse/NIFI-4457 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.3.0 > Environment: windows 10 >Reporter: meh > Attachments: Picture1.png, Picture2.png, create.txt, subquery.csv, > tw_json.csv > > Original Estimate: 24h > Remaining Estimate: 24h > > when "Maximum-value column" name is "id" there is no problem, when i add > "initial.maxvalue.id" property in "QueryDatabaseTable" processor, it works > well and maxvalue is increasing by every running. > !Picture1.png|thumbnail! > but... > when the "Maximum-value column" name is different from "id" (such as > "tweet_id"), after initial processor working, only given > "initial.maxvalue.id" is saves and that repeating just same value for every > run. > !Picture2.png|thumbnail! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (NIFI-4457) "Maximum-value" not increasing when "initial.maxvalue" is set and "Maximum-value column" name is different from "id"
[ https://issues.apache.org/jira/browse/NIFI-4457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] meh updated NIFI-4457: -- Attachment: create.txt tw_json.csv > "Maximum-value" not increasing when "initial.maxvalue" is set and > "Maximum-value column" name is different from "id" > - > > Key: NIFI-4457 > URL: https://issues.apache.org/jira/browse/NIFI-4457 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.3.0 > Environment: windows 10 >Reporter: meh > Attachments: Picture1.png, Picture2.png, create.txt, subquery.csv, > tw_json.csv > > Original Estimate: 24h > Remaining Estimate: 24h > > when "Maximum-value column" name is "id" there is no problem, when i add > "initial.maxvalue.id" property in "QueryDatabaseTable" processor, it works > well and maxvalue is increasing by every running. > !Picture1.png|thumbnail! > but... > when the "Maximum-value column" name is different from "id" (such as > "tweet_id"), after initial processor working, only given > "initial.maxvalue.id" is saves and that repeating just same value for every > run. > !Picture2.png|thumbnail! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFI-4457) "Maximum-value" not increasing when "initial.maxvalue" is set and "Maximum-value column" name is different from "id"
[ https://issues.apache.org/jira/browse/NIFI-4457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200121#comment-16200121 ] meh commented on NIFI-4457: --- [~ijokarumawak] i cannot alter database and add id column ... ; also when i changed max value from "tweet_id" to "date", problem is still happened and now "date" max value is not changing. i think it is a bug :| > "Maximum-value" not increasing when "initial.maxvalue" is set and > "Maximum-value column" name is different from "id" > - > > Key: NIFI-4457 > URL: https://issues.apache.org/jira/browse/NIFI-4457 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.3.0 > Environment: windows 10 >Reporter: meh > Attachments: Picture1.png, Picture2.png, subquery.csv > > Original Estimate: 24h > Remaining Estimate: 24h > > when "Maximum-value column" name is "id" there is no problem, when i add > "initial.maxvalue.id" property in "QueryDatabaseTable" processor, it works > well and maxvalue is increasing by every running. > !Picture1.png|thumbnail! > but... > when the "Maximum-value column" name is different from "id" (such as > "tweet_id"), after initial processor working, only given > "initial.maxvalue.id" is saves and that repeating just same value for every > run. > !Picture2.png|thumbnail! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFI-4457) "Maximum-value" not increasing when "initial.maxvalue" is set and "Maximum-value column" name is different from "id"
[ https://issues.apache.org/jira/browse/NIFI-4457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16192890#comment-16192890 ] meh commented on NIFI-4457: --- [~ijokarumawak] hi. i'm working on a database that data's are collecting by twitter API, the application that do this work fetch whole represented json to string and save it on other table, and request time and tweet id of that json parse into this table i talk about it. the logic behind twitter id's is incrementally and basically this is a proper field for max value tracking, but if you say so, you advise me to change field to date? Apart from these cases, this a problem that maybe a bug and must be solved :P > "Maximum-value" not increasing when "initial.maxvalue" is set and > "Maximum-value column" name is different from "id" > - > > Key: NIFI-4457 > URL: https://issues.apache.org/jira/browse/NIFI-4457 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.3.0 > Environment: windows 10 >Reporter: meh > Attachments: Picture1.png, Picture2.png, subquery.csv > > Original Estimate: 24h > Remaining Estimate: 24h > > when "Maximum-value column" name is "id" there is no problem, when i add > "initial.maxvalue.id" property in "QueryDatabaseTable" processor, it works > well and maxvalue is increasing by every running. > !Picture1.png|thumbnail! > but... > when the "Maximum-value column" name is different from "id" (such as > "tweet_id"), after initial processor working, only given > "initial.maxvalue.id" is saves and that repeating just same value for every > run. > !Picture2.png|thumbnail! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFI-4457) "Maximum-value" not increasing when "initial.maxvalue" is set and "Maximum-value column" name is different from "id"
[ https://issues.apache.org/jira/browse/NIFI-4457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16192863#comment-16192863 ] meh commented on NIFI-4457: --- [~patricker] yes, this is schema; and I think that is maybe as you said... :thinking: (on) USE twt_new; CREATE TABLE tweets_json ( tweet_id bigint(20) UNSIGNED NOT NULL, date datetime NOT NULL, PRIMARY KEY (tweet_id), INDEX date (date), CONSTRAINT FK_tweets_json_tweets FOREIGN KEY (tweet_id) REFERENCES tweets (id) ON DELETE RESTRICT ON UPDATE RESTRICT ) ENGINE = INNODB AVG_ROW_LENGTH = 782 CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_520_ci ROW_FORMAT = COMPRESSED; > "Maximum-value" not increasing when "initial.maxvalue" is set and > "Maximum-value column" name is different from "id" > - > > Key: NIFI-4457 > URL: https://issues.apache.org/jira/browse/NIFI-4457 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.3.0 > Environment: windows 10 >Reporter: meh > Attachments: Picture1.png, Picture2.png, subquery.csv > > Original Estimate: 24h > Remaining Estimate: 24h > > when "Maximum-value column" name is "id" there is no problem, when i add > "initial.maxvalue.id" property in "QueryDatabaseTable" processor, it works > well and maxvalue is increasing by every running. > !Picture1.png|thumbnail! > but... > when the "Maximum-value column" name is different from "id" (such as > "tweet_id"), after initial processor working, only given > "initial.maxvalue.id" is saves and that repeating just same value for every > run. > !Picture2.png|thumbnail! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFI-4457) "Maximum-value" not increasing when "initial.maxvalue" is set and "Maximum-value column" name is different from "id"
[ https://issues.apache.org/jira/browse/NIFI-4457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190876#comment-16190876 ] meh commented on NIFI-4457: --- [~patricker] hi peter. no that is not already the maximum value in the table, at same time new records are inserting and max value is increasing. try this dataset: [^subquery.csv] i connect to mariaDB database using jdbc, and query this table to return records greater than X by initial.maxvalue.tweet_id . at first use this works well, but i need this processor scheduled and for next run this problem is happening and maxvalue on view state not changing. i using this processor on other tables and there is not any problem, but this different! is it probable that relates to "Use Avro Logical Types" or "Default Decimal Precision" or "Default Decimal Scale"? because of length of tweet_id? > "Maximum-value" not increasing when "initial.maxvalue" is set and > "Maximum-value column" name is different from "id" > - > > Key: NIFI-4457 > URL: https://issues.apache.org/jira/browse/NIFI-4457 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.3.0 > Environment: windows 10 >Reporter: meh > Attachments: Picture1.png, Picture2.png, subquery.csv > > Original Estimate: 24h > Remaining Estimate: 24h > > when "Maximum-value column" name is "id" there is no problem, when i add > "initial.maxvalue.id" property in "QueryDatabaseTable" processor, it works > well and maxvalue is increasing by every running. > !Picture1.png|thumbnail! > but... > when the "Maximum-value column" name is different from "id" (such as > "tweet_id"), after initial processor working, only given > "initial.maxvalue.id" is saves and that repeating just same value for every > run. > !Picture2.png|thumbnail! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (NIFI-4457) "Maximum-value" not increasing when "initial.maxvalue" is set and "Maximum-value column" name is different from "id"
[ https://issues.apache.org/jira/browse/NIFI-4457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] meh updated NIFI-4457: -- Attachment: subquery.csv > "Maximum-value" not increasing when "initial.maxvalue" is set and > "Maximum-value column" name is different from "id" > - > > Key: NIFI-4457 > URL: https://issues.apache.org/jira/browse/NIFI-4457 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.3.0 > Environment: windows 10 >Reporter: meh > Attachments: Picture1.png, Picture2.png, subquery.csv > > Original Estimate: 24h > Remaining Estimate: 24h > > when "Maximum-value column" name is "id" there is no problem, when i add > "initial.maxvalue.id" property in "QueryDatabaseTable" processor, it works > well and maxvalue is increasing by every running. > !Picture1.png|thumbnail! > but... > when the "Maximum-value column" name is different from "id" (such as > "tweet_id"), after initial processor working, only given > "initial.maxvalue.id" is saves and that repeating just same value for every > run. > !Picture2.png|thumbnail! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (NIFI-4457) Maximum-value not increasing when initial.maxvalue is set and "Maximum-value column" name is different from "id"
[ https://issues.apache.org/jira/browse/NIFI-4457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] meh updated NIFI-4457: -- Summary: Maximum-value not increasing when initial.maxvalue is set and "Maximum-value column" name is different from "id" (was: Maximum-value not increasing when initial.maxvalue is set) > Maximum-value not increasing when initial.maxvalue is set and "Maximum-value > column" name is different from "id" > - > > Key: NIFI-4457 > URL: https://issues.apache.org/jira/browse/NIFI-4457 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.3.0 > Environment: windows 10 >Reporter: meh > Attachments: Picture1.png, Picture2.png > > Original Estimate: 24h > Remaining Estimate: 24h > > when "Maximum-value column" name is "id" there is no problem, when i add > "initial.maxvalue.id" property in "QueryDatabaseTable" processor, it works > well and maxvalue is increasing by every running. > !Picture1.png|thumbnail! > but... > when the "Maximum-value column" name is different from "id" (such as > "tweet_id"), after initial processor working, only given > "initial.maxvalue.id" is saves and that repeating just same value for every > run. > !Picture2.png|thumbnail! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (NIFI-4457) "Maximum-value" not increasing when "initial.maxvalue" is set and "Maximum-value column" name is different from "id"
[ https://issues.apache.org/jira/browse/NIFI-4457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] meh updated NIFI-4457: -- Summary: "Maximum-value" not increasing when "initial.maxvalue" is set and "Maximum-value column" name is different from "id" (was: Maximum-value not increasing when initial.maxvalue is set and "Maximum-value column" name is different from "id" ) > "Maximum-value" not increasing when "initial.maxvalue" is set and > "Maximum-value column" name is different from "id" > - > > Key: NIFI-4457 > URL: https://issues.apache.org/jira/browse/NIFI-4457 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.3.0 > Environment: windows 10 >Reporter: meh > Attachments: Picture1.png, Picture2.png > > Original Estimate: 24h > Remaining Estimate: 24h > > when "Maximum-value column" name is "id" there is no problem, when i add > "initial.maxvalue.id" property in "QueryDatabaseTable" processor, it works > well and maxvalue is increasing by every running. > !Picture1.png|thumbnail! > but... > when the "Maximum-value column" name is different from "id" (such as > "tweet_id"), after initial processor working, only given > "initial.maxvalue.id" is saves and that repeating just same value for every > run. > !Picture2.png|thumbnail! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (NIFI-4457) Maximum-value not increasing when initial.maxvalue is set
meh created NIFI-4457: - Summary: Maximum-value not increasing when initial.maxvalue is set Key: NIFI-4457 URL: https://issues.apache.org/jira/browse/NIFI-4457 Project: Apache NiFi Issue Type: Bug Components: Core Framework Affects Versions: 1.3.0 Environment: windows 10 Reporter: meh Attachments: Picture1.png, Picture2.png when "Maximum-value column" name is "id" there is no problem, when i add "initial.maxvalue.id" property in "QueryDatabaseTable" processor, it works well and maxvalue is increasing by every running. !Picture1.png|thumbnail! but... when the "Maximum-value column" name is different from "id" (such as "tweet_id"), after initial processor working, only given "initial.maxvalue.id" is saves and that repeating just same value for every run. !Picture2.png|thumbnail! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
