[jira] [Updated] (NIFI-8295) Upgrade Cassandra driver to latest 3.x
[ https://issues.apache.org/jira/browse/NIFI-8295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wouter de Vries updated NIFI-8295: -- Fix Version/s: (was: 1.14.0) > Upgrade Cassandra driver to latest 3.x > -- > > Key: NIFI-8295 > URL: https://issues.apache.org/jira/browse/NIFI-8295 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.13.0 >Reporter: madhavan >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > The Cassandra processors use the [3.3.0 > driver|https://github.com/apache/nifi/blob/rel/nifi-1.13.0/nifi-nar-bundles/nifi-cassandra-bundle/pom.xml#L26]. > The latest driver at the time of this writing is > [3.10.2|https://docs.datastax.com/en/developer/java-driver/3.10/changelog/#3-10-2], > which is fully backwards-compatible. > Upgrading the driver will enable NiFi users to interact with any Cassandra > cluster of version 1.x, 2.x, or 3.x and leverage the fixes, feature and > improvements done. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-8383) PutCassandraRecord always uses batch statements
Wouter de Vries created NIFI-8383:
-
Summary: PutCassandraRecord always uses batch statements
Key: NIFI-8383
URL: https://issues.apache.org/jira/browse/NIFI-8383
Project: Apache NiFi
Issue Type: Bug
Components: Extensions
Affects Versions: 1.13.2
Reporter: Wouter de Vries
The PutCassandraRecord processor uses a BatchStatement for all inserts. This is
not recommended for most situations, particularly when the records that are
inserted belong to different partitions.
Instead, we should use parallel/asynchronous insertions to optimize for write
speed, and still allow batch statements when some form of atomicity is required.
([https://docs.datastax.com/en/dse/6.0/cql/cql/cql_reference/cql_commands/cqlBatch.html)]
This setup leads to log warnings of the following form:
{noformat}
stdout F WARN [Native-Transport-Requests-3] 2021-03-30 08:33:01,760
NoSpamLogger.java:94 - Unlogged batch covering 100 partitions detected against
table [KEYSPACE.TABLE]. You should use a logged batch for atomicity, or
asynchronous writes for performance.{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (NIFI-8382) PutCassandraRecord does not allow specifying consistency
Wouter de Vries created NIFI-8382: - Summary: PutCassandraRecord does not allow specifying consistency Key: NIFI-8382 URL: https://issues.apache.org/jira/browse/NIFI-8382 Project: Apache NiFi Issue Type: Bug Components: Extensions Affects Versions: 1.13.2 Reporter: Wouter de Vries The PutCassandraRecord processor only allows specifying the serial consistency and not the "regular" consistency, this serial consistency type is only used for the paxos phase during a lightweight transaction. [https://docs.datastax.com/en/cql-oss/3.3/cql/cql_reference/cqlshSerialConsistency.html] * SERIAL * LOCAL_SERIAL [https://docs.datastax.com/en/cql-oss/3.3/cql/cql_reference/cqlshConsistency.html] * ALL * EACH_QUORUM * QUORUM * LOCAL_QUORUM * ONE * TWO * THREE * LOCAL_ONE * ANY For writes the use of consistency ONE (the default if unspecified) is not recommended. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-7737) Add support for String[] to PutCassandraRecord
Wouter de Vries created NIFI-7737:
-
Summary: Add support for String[] to PutCassandraRecord
Key: NIFI-7737
URL: https://issues.apache.org/jira/browse/NIFI-7737
Project: Apache NiFi
Issue Type: Improvement
Components: Extensions
Affects Versions: 1.11.4
Reporter: Wouter de Vries
Assignee: Wouter de Vries
Currently the PutCassandraRecord processor does support string arrays. Trying
to use them results in the following error:
{noformat}
| 2020-08-13 15:15:37,861 ERROR [Timer-Driven Process Thread-5]
o.a.n.p.cassandra.PutCassandraRecord
PutCassandraRecord[id=af756410-1ef0-3d80-045e-160f02632c54] Unable to write the
records into Cassandra table due to
com.datastax.driver.core.exceptions.InvalidTypeException: Value 3 of type class
[Ljava.lang.Object; does not correspond to any CQL3 type:
com.datastax.driver.core.exceptions.InvalidTypeException: Value 3 of type class
[Ljava.lang.Object; does not correspond to any CQL3 type
| com.datastax.driver.core.exceptions.InvalidTypeException: Value 3 of type
class [Ljava.lang.Object; does not correspond to any CQL3 type
| at com.datastax.driver.core.querybuilder.Utils.convert(Utils.java:361)
| at
com.datastax.driver.core.querybuilder.BuiltStatement.getValues(BuiltStatement.java:265)
| at
com.datastax.driver.core.BatchStatement.getIdAndValues(BatchStatement.java:92)
| at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:597)
| at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
| at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:68)
| at
org.apache.nifi.processors.cassandra.PutCassandraRecord.onTrigger(PutCassandraRecord.java:300)
| at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
| at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
| at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
| at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
| at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
| at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
| at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
| at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
| at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
| at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
| at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
| at java.lang.Thread.run(Thread.java:748)
| Caused by: com.datastax.driver.core.exceptions.CodecNotFoundException: Codec
not found for requested operation: [ANY <-> [Ljava.lang.Object;]
| at
com.datastax.driver.core.CodecRegistry.notFound(CodecRegistry.java:741)
| at
com.datastax.driver.core.CodecRegistry.createCodec(CodecRegistry.java:602)
| at
com.datastax.driver.core.CodecRegistry.findCodec(CodecRegistry.java:582)
| at
com.datastax.driver.core.CodecRegistry.codecFor(CodecRegistry.java:429)
| at com.datastax.driver.core.querybuilder.Utils.convert(Utils.java:357)
| ... 18 common frames omitted{noformat}
The solution is to add the appropriate codec for this type conversion. I will
submit a PR for this change.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (NIFI-6666) InvokeHTTP should provide a User-agent header
[
https://issues.apache.org/jira/browse/NIFI-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17176862#comment-17176862
]
Wouter de Vries edited comment on NIFI- at 8/13/20, 1:33 PM:
-
Looking at user-agents in general, we are (were) including a significant amount
of additional information in the user agent (exact kernel and java versions?),
that in my opinion should not be there. Imho this should be something like
"apache-nifi/1.12.0 (OkHttp/3)" or "apache-nifi/1.11.4-SNAPSHOT (OkHttp/3)".
Edit: I was under the impression that this change added a default user-agent,
but it does not.
was (Author: woutifier):
Looking at user-agents in general, we are (were) including a significant amount
of additional information in the user agent (exact kernel and java versions?),
that in my opinion should not be there. Imho this should be something like
"apache-nifi/1.12.0 (OkHttp/3)" or "apache-nifi/1.11.4-SNAPSHOT (OkHttp/3)".
> InvokeHTTP should provide a User-agent header
> -
>
> Key: NIFI-
> URL: https://issues.apache.org/jira/browse/NIFI-
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Niels Basjes
>Assignee: Joe Witt
>Priority: Major
> Fix For: 1.12.0
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Currently the InvokeHTTP processor does not send out a User-agent header at
> all.
> It should provide such a header with an honest indication the request was
> done using Apache Nifi.
> I have created an implementation for this (pull request will follow shortly)
> that creates the following value automatically based on the Nifi version and
> the current system.
> *{{Apache Nifi/1.10.0-SNAPSHOT (git:rel/nifi-1.9.0-347-gec3ea46;
> Java/1.8.0_222; Linux 4.4.0-159-generic; amd64; https://nifi.apache.org/)}}*
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-6666) InvokeHTTP should provide a User-agent header
[
https://issues.apache.org/jira/browse/NIFI-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17176862#comment-17176862
]
Wouter de Vries commented on NIFI-:
---
Looking at user-agents in general, we are (were) including a significant amount
of additional information in the user agent (exact kernel and java versions?),
that in my opinion should not be there. Imho this should be something like
"apache-nifi/1.12.0 (OkHttp/3)" or "apache-nifi/1.11.4-SNAPSHOT (OkHttp/3)".
> InvokeHTTP should provide a User-agent header
> -
>
> Key: NIFI-
> URL: https://issues.apache.org/jira/browse/NIFI-
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Niels Basjes
>Assignee: Joe Witt
>Priority: Major
> Fix For: 1.12.0
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Currently the InvokeHTTP processor does not send out a User-agent header at
> all.
> It should provide such a header with an honest indication the request was
> done using Apache Nifi.
> I have created an implementation for this (pull request will follow shortly)
> that creates the following value automatically based on the Nifi version and
> the current system.
> *{{Apache Nifi/1.10.0-SNAPSHOT (git:rel/nifi-1.9.0-347-gec3ea46;
> Java/1.8.0_222; Linux 4.4.0-159-generic; amd64; https://nifi.apache.org/)}}*
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7579) Create a GetS3Object Processor
[ https://issues.apache.org/jira/browse/NIFI-7579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17148342#comment-17148342 ] Wouter de Vries commented on NIFI-7579: --- [~ArpStorm1]do you have a specific reason why that should not happen? As far as I can see the code of this new processor would be by and large identical to the existing FetchS3Object processor, is that not the case? > Create a GetS3Object Processor > -- > > Key: NIFI-7579 > URL: https://issues.apache.org/jira/browse/NIFI-7579 > Project: Apache NiFi > Issue Type: New Feature >Reporter: ArpStorm1 >Assignee: YoungGyu Chun >Priority: Major > > Sometimes the client needs to get only specific object or a subset of objects > from its bucket. Now, the only way to do it is using ListS3 Processor and > after that using FetchS3Object processor. Creating a GetS3Object processor > for such cases can be great -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7579) Create a GetS3Object Processor
[ https://issues.apache.org/jira/browse/NIFI-7579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147661#comment-17147661 ] Wouter de Vries commented on NIFI-7579: --- Would that not be solved with the GenerateFlowFile processor? If that does not solve it, I would still argue that this behavior should be merged with the FetchS3Object processor, so that it supports being triggered with and without an upstream connection. > Create a GetS3Object Processor > -- > > Key: NIFI-7579 > URL: https://issues.apache.org/jira/browse/NIFI-7579 > Project: Apache NiFi > Issue Type: New Feature >Reporter: ArpStorm1 >Assignee: YoungGyu Chun >Priority: Major > > Sometimes the client needs to get only specific object or a subset of objects > from its bucket. Now, the only way to do it is using ListS3 Processor and > after that using FetchS3Object processor. Creating a GetS3Object processor > for such cases can be great -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (NIFI-7393) Expose max idle time and max idle connections in InvokeHTTP processor
[ https://issues.apache.org/jira/browse/NIFI-7393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wouter de Vries resolved NIFI-7393. --- Fix Version/s: 1.12.0 Resolution: Implemented > Expose max idle time and max idle connections in InvokeHTTP processor > - > > Key: NIFI-7393 > URL: https://issues.apache.org/jira/browse/NIFI-7393 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Wouter de Vries >Assignee: Wouter de Vries >Priority: Major > Fix For: 1.12.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > In some cases we don't want idle connections to some HTTP servers to remain > open. Or we want to limit the time that those connections spend being idle > before closing. > OkHttp's connectionPool exposes two parameters for this, but they are not > configurable from Nifi. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (NIFI-7393) Expose max idle time and max idle connections in InvokeHTTP processor
[ https://issues.apache.org/jira/browse/NIFI-7393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wouter de Vries reassigned NIFI-7393: - Assignee: Wouter de Vries > Expose max idle time and max idle connections in InvokeHTTP processor > - > > Key: NIFI-7393 > URL: https://issues.apache.org/jira/browse/NIFI-7393 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Wouter de Vries >Assignee: Wouter de Vries >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > In some cases we don't want idle connections to some HTTP servers to remain > open. Or we want to limit the time that those connections spend being idle > before closing. > OkHttp's connectionPool exposes two parameters for this, but they are not > configurable from Nifi. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-7393) Expose max idle time and max idle connections in InvokeHTTP processor
Wouter de Vries created NIFI-7393: - Summary: Expose max idle time and max idle connections in InvokeHTTP processor Key: NIFI-7393 URL: https://issues.apache.org/jira/browse/NIFI-7393 Project: Apache NiFi Issue Type: Improvement Components: Extensions Reporter: Wouter de Vries In some cases we don't want idle connections to some HTTP servers to remain open. Or we want to limit the time that those connections spend being idle before closing. OkHttp's connectionPool exposes two parameters for this, but they are not configurable from Nifi. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-7241) Unable to remove port from a child ProcessGroup
Wouter de Vries created NIFI-7241: - Summary: Unable to remove port from a child ProcessGroup Key: NIFI-7241 URL: https://issues.apache.org/jira/browse/NIFI-7241 Project: Apache NiFi Issue Type: Bug Components: Flow Versioning Affects Versions: 1.11.2 Reporter: Wouter de Vries Attachments: trace.log Given the following scenario: * Output port 1 in process group A * Input port 2 in process group B * A connection between A and B * Process group C is versioned and contains A and B Removing the connection as well as output port 1 and then committing the version creates a flow version that cannot be upgraded to. (Try reverting to an older version, and back to this version). Throwing an IllegalStateException. I believe this is caused by the order of updates in "StandardProcessGroup.java" where child groups (line 3781) are updated prior to the connections of the group itself (line 3986). This causes the output port to be removed before the connection from it is removed. I haven't tested this but probably removing the input port causes similar issues. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-7132) PutCassandraQL is handling UUIDs as Strings
Wouter de Vries created NIFI-7132: - Summary: PutCassandraQL is handling UUIDs as Strings Key: NIFI-7132 URL: https://issues.apache.org/jira/browse/NIFI-7132 Project: Apache NiFi Issue Type: Bug Components: Extensions Affects Versions: 1.11.1 Reporter: Wouter de Vries PutCassandraQL fails when attempting to insert a UUID/timeUUID on Nifi 1.11.1 (and presumably all prior versions), due to handling UUIDs as strings. Input: FlowFile Attribute Map Content Key: 'cql.args.1.type' Value: 'timeuuid' Key: 'cql.args.1.value' Value: '5442b1f6-4c16-11ea-87f5-45a32dbc5199' -- INSERT INTO test.test (id) VALUES ( ? ) Output: com.datastax.driver.core.exceptions.CodecNotFoundException: Codec not found for requested operation: [timeuuid <-> java.lang.String] Record does not get inserted into Cassandra Expected output: Record gets inserted into Cassandra -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7055) createListValidator returns valid for empty list with "," input
[ https://issues.apache.org/jira/browse/NIFI-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021249#comment-17021249 ] Wouter de Vries commented on NIFI-7055: --- Sounds good to me (y) > createListValidator returns valid for empty list with "," input > --- > > Key: NIFI-7055 > URL: https://issues.apache.org/jira/browse/NIFI-7055 > Project: Apache NiFi > Issue Type: Bug >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > from Slack: > > "I'm looking at the createListValidator, and to my surprise passing in a list > of (essentially) two empty elements "," validates, while a totally empty > string "" does not. Apparently due to some underlying behavior of > String.split." > The string "," does return a String[0] from split. This should fail > validation as if here were no elements as null, "", " " do possibly. > But that kind of goes against or doesn't consider the ignore empty entries. > I think the difference is whether or not you consider "," to be a list of two > empty elements or an empty list. > The current implementation with String.spilt() will produce an empty list. > Is that correct? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7055) createListValidator returns valid for empty list with "," input
[ https://issues.apache.org/jira/browse/NIFI-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021160#comment-17021160 ] Wouter de Vries commented on NIFI-7055: --- My take on this, in the "," case we do have at least one empty element, even if you exclude trailing commas. And so passing false to excludeEmpty and a NON_EMPTY validator should result in a fail. This is due to how String.split works "," results in a zero-length array, and so the NON_EMPTY validator is not called. Concretely: "", ",", "," should all not pass validation if a NON_EMPTY validator is provided and excludeEmpty is false. > createListValidator returns valid for empty list with "," input > --- > > Key: NIFI-7055 > URL: https://issues.apache.org/jira/browse/NIFI-7055 > Project: Apache NiFi > Issue Type: Bug >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > from Slack: > > "I'm looking at the createListValidator, and to my surprise passing in a list > of (essentially) two empty elements "," validates, while a totally empty > string "" does not. Apparently due to some underlying behavior of > String.split." > The string "," does return a String[0] from split. This should fail > validation as if here were no elements as null, "", " " do possibly. > But that kind of goes against or doesn't consider the ignore empty entries. > I think the difference is whether or not you consider "," to be a list of two > empty elements or an empty list. > The current implementation with String.spilt() will produce an empty list. > Is that correct? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-7007) Add update capability to PutCassandraRecord
Wouter de Vries created NIFI-7007: - Summary: Add update capability to PutCassandraRecord Key: NIFI-7007 URL: https://issues.apache.org/jira/browse/NIFI-7007 Project: Apache NiFi Issue Type: Improvement Components: Extensions Reporter: Wouter de Vries The current PutCassandraRecord processor (1.10.0) only supports the INSERT statement. It would be useful to extend it to also support UPDATE, especially in the context of COUNTER tables which do not allow INSERTs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
