[jira] [Assigned] (NIFI-12956) Add semantic styles for the basic theme colors
[
https://issues.apache.org/jira/browse/NIFI-12956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Scott Aslan reassigned NIFI-12956:
--
Assignee: Scott Aslan
> Add semantic styles for the basic theme colors
> --
>
> Key: NIFI-12956
> URL: https://issues.apache.org/jira/browse/NIFI-12956
> Project: Apache NiFi
> Issue Type: Sub-task
> Components: Core UI
>Reporter: Rob Fellows
>Assignee: Scott Aslan
>Priority: Minor
>
> We should add themed styles for the commonly used colors from the palettes.
> I'm thinking like the 3 (light, default, dark) main colors from each palette.
> Something like this:
> {code:java}
> .warn-light {
> color: $warn-palette-light
> }
> .warn-default {
> color: $warn-palette-default
> }
> .warn-dark {
> color: $warn-palette-dark
> }
> .primary-light {
> color: $primary-palette-light
> }
> .primary-default {
> color: $primary-palette-default
> }
> .primary-dark {
> color: $primary-palette-dark
> }
> .accent-light {
> color: $accent-palette-light
> }
> .accent-default {
> color: $accent-palette-default
> }
> .accent-dark {
> color: $accent-palette-dark
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (NIFI-12990) BUG - enable CS dialog does not close when go to parameter provider
Scott Aslan created NIFI-12990: -- Summary: BUG - enable CS dialog does not close when go to parameter provider Key: NIFI-12990 URL: https://issues.apache.org/jira/browse/NIFI-12990 Project: Apache NiFi Issue Type: Sub-task Reporter: Scott Aslan Attachments: image-2024-04-01-21-38-39-843.png !image-2024-04-01-21-38-39-843.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12988) Streamline Iceberg Test Utils Dependencies
[
https://issues.apache.org/jira/browse/NIFI-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Villard updated NIFI-12988:
--
Fix Version/s: 2.0.0-M3
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Streamline Iceberg Test Utils Dependencies
> --
>
> Key: NIFI-12988
> URL: https://issues.apache.org/jira/browse/NIFI-12988
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
> Fix For: 2.0.0-M3
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Following the move of {{nifi-hive-test-utils}} to
> {{{}nifi-iceberg-test-utils{}}}, several of the dependencies and exclusions
> can be streamlined to reduce duplication and unnecessary references.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12988) Streamline Iceberg Test Utils Dependencies
[
https://issues.apache.org/jira/browse/NIFI-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832954#comment-17832954
]
ASF subversion and git services commented on NIFI-12988:
Commit 322deaa23eb5141174bb2c25b97b2a96b2596eac in nifi's branch
refs/heads/main from David Handermann
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=322deaa23e ]
NIFI-12988 Streamlined dependencies in nifi-iceberg-test-utils
Signed-off-by: Pierre Villard
This closes #8592.
> Streamline Iceberg Test Utils Dependencies
> --
>
> Key: NIFI-12988
> URL: https://issues.apache.org/jira/browse/NIFI-12988
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Following the move of {{nifi-hive-test-utils}} to
> {{{}nifi-iceberg-test-utils{}}}, several of the dependencies and exclusions
> can be streamlined to reduce duplication and unnecessary references.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] NIFI-12988 Streamline dependencies in nifi-iceberg-test-utils [nifi]
asfgit closed pull request #8592: NIFI-12988 Streamline dependencies in nifi-iceberg-test-utils URL: https://github.com/apache/nifi/pull/8592 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] NIFI-12987 allow controller service type to be searchable [nifi]
mosermw opened a new pull request, #8593: URL: https://github.com/apache/nifi/pull/8593 # Summary [NIFI-12987](https://issues.apache.org/jira/browse/NIFI-12987) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [x] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [x] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [x] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [x] Pull Request based on current revision of the `main` branch - [x] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [x] Build completed using `mvn clean install -P contrib-check` - [x] JDK 21 ### Licensing - [n/a] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [n/a] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [n/a] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-12987) Controller Service type should be searchable
[ https://issues.apache.org/jira/browse/NIFI-12987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael W Moser updated NIFI-12987: --- Status: Patch Available (was: Open) > Controller Service type should be searchable > > > Key: NIFI-12987 > URL: https://issues.apache.org/jira/browse/NIFI-12987 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.25.0 >Reporter: Michael W Moser >Assignee: Michael W Moser >Priority: Minor > Labels: backport-needed > > Currently controller service name, uuid, property name, property value and > comments are all searchable. However you cannot search by controller service > type (class name) like you can with other components. Allow the search to > find controller services by type. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-12989) hostname seems truncated to 16 bytes
venu created NIFI-12989: --- Summary: hostname seems truncated to 16 bytes Key: NIFI-12989 URL: https://issues.apache.org/jira/browse/NIFI-12989 Project: Apache NiFi Issue Type: Bug Affects Versions: 1.25.0 Reporter: venu Attachments: image-2024-04-01-15-24-31-012.png, image-2024-04-01-15-25-18-646.png, image-2024-04-01-15-26-45-362.png We are host name truncated one of the messages. !image-2024-04-01-15-24-31-012.png! !image-2024-04-01-15-25-18-646.png! !image-2024-04-01-15-26-45-362.png! We can that hostName is truncated, -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-12858) Processor change history on property hover
[ https://issues.apache.org/jira/browse/NIFI-12858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann reassigned NIFI-12858: --- Assignee: David Handermann > Processor change history on property hover > -- > > Key: NIFI-12858 > URL: https://issues.apache.org/jira/browse/NIFI-12858 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.25.0 >Reporter: Chris Conklin >Assignee: David Handermann >Priority: Minor > Attachments: nifi1.23.2.png, nifi1.24.png > > > The history details list on processors changed order after release 1.23.2. > Previous sort on history upon hovering over some property was latest update > first. > Starting with release 1.24 this order changed to oldest change first. > Since only 5 history lines appear this makes the feature not very useful. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-12858) Processor change history on property hover
[ https://issues.apache.org/jira/browse/NIFI-12858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann reassigned NIFI-12858: --- Assignee: (was: David Handermann) > Processor change history on property hover > -- > > Key: NIFI-12858 > URL: https://issues.apache.org/jira/browse/NIFI-12858 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.25.0 >Reporter: Chris Conklin >Priority: Minor > Attachments: nifi1.23.2.png, nifi1.24.png > > > The history details list on processors changed order after release 1.23.2. > Previous sort on history upon hovering over some property was latest update > first. > Starting with release 1.24 this order changed to oldest change first. > Since only 5 history lines appear this makes the feature not very useful. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12988) Streamline Iceberg Test Utils Dependencies
[
https://issues.apache.org/jira/browse/NIFI-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-12988:
Status: Patch Available (was: Open)
> Streamline Iceberg Test Utils Dependencies
> --
>
> Key: NIFI-12988
> URL: https://issues.apache.org/jira/browse/NIFI-12988
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Following the move of {{nifi-hive-test-utils}} to
> {{{}nifi-iceberg-test-utils{}}}, several of the dependencies and exclusions
> can be streamlined to reduce duplication and unnecessary references.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] NIFI-12988 Streamline dependencies in nifi-iceberg-test-utils [nifi]
exceptionfactory opened a new pull request, #8592: URL: https://github.com/apache/nifi/pull/8592 # Summary [NIFI-12988](https://issues.apache.org/jira/browse/NIFI-12988) Reduces the number of dependencies in `nifi-iceberg-test-utils`, such as several Hadoop-related dependencies that have no direct or indirect references. The `nifi-iceberg-processors` module is the only reference to `nifi-iceberg-test-utils`. Changes to `nifi-iceberg-processors` Maven configuration include explicitly listing `derby` to align with the version of `derbytools` already included, and excluding an unnecessary dependency on Hadoop Yarn Server. # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [X] Pull Request based on current revision of the `main` branch - [X] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [X] Build completed using `mvn clean install -P contrib-check` - [X] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-12988) Streamline Iceberg Test Utils Dependencies
David Handermann created NIFI-12988:
---
Summary: Streamline Iceberg Test Utils Dependencies
Key: NIFI-12988
URL: https://issues.apache.org/jira/browse/NIFI-12988
Project: Apache NiFi
Issue Type: Improvement
Components: Extensions
Reporter: David Handermann
Assignee: David Handermann
Following the move of {{nifi-hive-test-utils}} to
{{{}nifi-iceberg-test-utils{}}}, several of the dependencies and exclusions can
be streamlined to reduce duplication and unnecessary references.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] NIFI-12875: Add unsuccessful HTTP status code handling logic [nifi]
gforeman02 commented on PR #8484: URL: https://github.com/apache/nifi/pull/8484#issuecomment-2030290474 @dan-s1 and @exceptionfactory thanks for the reviews/suggestions. Updates committed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (NIFI-12987) Controller Service type should be searchable
[ https://issues.apache.org/jira/browse/NIFI-12987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael W Moser reassigned NIFI-12987: -- Assignee: Michael W Moser > Controller Service type should be searchable > > > Key: NIFI-12987 > URL: https://issues.apache.org/jira/browse/NIFI-12987 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.25.0 >Reporter: Michael W Moser >Assignee: Michael W Moser >Priority: Minor > Labels: backport-needed > > Currently controller service name, uuid, property name, property value and > comments are all searchable. However you cannot search by controller service > type (class name) like you can with other components. Allow the search to > find controller services by type. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-12987) Controller Service type should be searchable
Michael W Moser created NIFI-12987: -- Summary: Controller Service type should be searchable Key: NIFI-12987 URL: https://issues.apache.org/jira/browse/NIFI-12987 Project: Apache NiFi Issue Type: Improvement Components: Core Framework Affects Versions: 1.25.0 Reporter: Michael W Moser Currently controller service name, uuid, property name, property value and comments are all searchable. However you cannot search by controller service type (class name) like you can with other components. Allow the search to find controller services by type. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-5125) Unable to select Controller Service UUID from Bulletin Board
[ https://issues.apache.org/jira/browse/NIFI-5125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832902#comment-17832902 ] Michael W Moser commented on NIFI-5125: --- I found this ticket, and tested again. The "Error: Unable to find specified component" still happens. However, after NIFI-7188, controller service UUID values are searchable from the toolbar. So the search capability is unrelated to the error. > Unable to select Controller Service UUID from Bulletin Board > > > Key: NIFI-5125 > URL: https://issues.apache.org/jira/browse/NIFI-5125 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.6.0 >Reporter: Mark Bean >Priority: Major > > When Selecting the linked UUID value of a Controller Service from the > Bulletin Board, the result is "Error: Unable to find the specified > component." Expected behavior would be to open the appropriate Controller > Services tab (either Component or Reporting Tasks) > Perhaps related is the fact that Controller Service UUID values are not > searchable from the toolbar. This ticket may need to add this capability as > well. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12960) ExcelReader - password protected XLSX
[ https://issues.apache.org/jira/browse/NIFI-12960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832898#comment-17832898 ] Daniel Stieglitz commented on NIFI-12960: - [~iiojj2] Did you want this feature for 2.x and back ported to 1.x or just for 2.x? > ExcelReader - password protected XLSX > - > > Key: NIFI-12960 > URL: https://issues.apache.org/jira/browse/NIFI-12960 > Project: Apache NiFi > Issue Type: New Feature > Components: Core Framework >Affects Versions: 1.25.0 >Reporter: Philipp Korniets >Priority: Trivial > > As we are trying to decomission ConvertExcelToCSVProcessor and replace it > with new ExcelReader it looks reasonable to implement functionality to read > password protected XLSX > Currently the only option is ExecuteScript with some code. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-12986) Tidy up JavaDoc of ProcessSession
endzeit created NIFI-12986:
--
Summary: Tidy up JavaDoc of ProcessSession
Key: NIFI-12986
URL: https://issues.apache.org/jira/browse/NIFI-12986
Project: Apache NiFi
Issue Type: Sub-task
Reporter: endzeit
Assignee: endzeit
While working on NIFI-12982 I noticed that the JavaDoc of {{ProcessSession}}
has some minor typos and documentation drifts between method overloads.
The goal of this ticket is to aim make the JavaDoc for the current
{{ProcessSession}} specification more consistent. The specified contract must
not be altered.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] NIFI-12875: Add unsuccessful HTTP status code handling logic [nifi]
dan-s1 commented on code in PR #8484:
URL: https://github.com/apache/nifi/pull/8484#discussion_r1546454088
##
nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/test/java/org/apache/nifi/lookup/TestRestLookupService.java:
##
@@ -170,6 +171,30 @@ void testLookupRecordFoundPostMethod() throws Exception {
assertPostRecordedRequestFound();
}
+@Test
+void testLookupErrorPassThrough() throws Exception {
+runner.setProperty(restLookupService,
RestLookupService.PROP_RESPONSE_CODE_HANDLING,
ResponseHandlingStrategy.RETURNED);
+runner.enableControllerService(restLookupService);
+
+when(recordReaderFactory.createRecordReader(any(), any(), anyLong(),
any())).thenReturn(recordReader);
+when(recordReader.nextRecord()).thenReturn(record);
+
+mockWebServer.enqueue(new
MockResponse().setResponseCode(HTTP_NOT_FOUND)
+.setHeader("Content-type", "application/json")
+.setBody("{\"error\": { \"code\": 404 } }"));
+
+final Optional recordFound =
restLookupService.lookup(Collections.emptyMap());
+assertTrue(recordFound.isPresent());
+}
Review Comment:
```suggestion
}
```
##
nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/test/java/org/apache/nifi/lookup/TestRestLookupService.java:
##
@@ -170,6 +171,30 @@ void testLookupRecordFoundPostMethod() throws Exception {
assertPostRecordedRequestFound();
}
+@Test
+void testLookupErrorPassThrough() throws Exception {
+runner.setProperty(restLookupService,
RestLookupService.PROP_RESPONSE_CODE_HANDLING,
ResponseHandlingStrategy.RETURNED);
+runner.enableControllerService(restLookupService);
+
+when(recordReaderFactory.createRecordReader(any(), any(), anyLong(),
any())).thenReturn(recordReader);
+when(recordReader.nextRecord()).thenReturn(record);
+
+mockWebServer.enqueue(new
MockResponse().setResponseCode(HTTP_NOT_FOUND)
+.setHeader("Content-type", "application/json")
+.setBody("{\"error\": { \"code\": 404 } }"));
+
+final Optional recordFound =
restLookupService.lookup(Collections.emptyMap());
+assertTrue(recordFound.isPresent());
+}
+@Test
+void testLookupErrorHandle() {
+runner.setProperty(restLookupService,
RestLookupService.PROP_RESPONSE_CODE_HANDLING,
ResponseHandlingStrategy.EVALUATED);
+runner.enableControllerService(restLookupService);
+mockWebServer.enqueue(new
MockResponse().setResponseCode(HTTP_NOT_FOUND));
+
+final LookupFailureException exception =
assertThrows(LookupFailureException.class, () ->
restLookupService.lookup(Collections.emptyMap()));
+assertInstanceOf(IOException.class, exception.getCause());
+}
Review Comment:
```suggestion
}
```
##
nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/java/org/apache/nifi/lookup/RestLookupService.java:
##
@@ -172,6 +172,17 @@ public class RestLookupService extends
AbstractControllerService implements Reco
.addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
.build();
+
+public static final PropertyDescriptor RESPONSE_HANDLING_STRATEGY = new
PropertyDescriptor.Builder()
+.name("rest-lookup-response-code-handling")
+.displayName("Response Code Handling Strategy")
+.description("How to handle response codes from remote service.
'Pass-through' sends successful and unsuccessful HTTP responses to the caller.
" +
+"'Handle Service Errors' generates an exception when an
unsuccessful HTTP response code is received.")
+.required(true)
+.defaultValue(ResponseHandlingStrategy.RETURNED)
+.allowableValues(ResponseHandlingStrategy.RETURNED,
ResponseHandlingStrategy.EVALUATED)
Review Comment:
```suggestion
.allowableValues(ResponseHandlingStrategy.class)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-12982 Extend test suite of MockProcessSession [nifi]
EndzeitBegins commented on code in PR #8589:
URL: https://github.com/apache/nifi/pull/8589#discussion_r1546391808
##
nifi-mock/src/test/java/org/apache/nifi/util/TestMockProcessSession.java:
##
@@ -45,189 +52,588 @@
public class TestMockProcessSession {
-@Test
-public void testReadWithoutCloseThrowsExceptionOnCommit() throws
IOException {
-final Processor processor = new PoorlyBehavedProcessor();
-final MockProcessSession session = new MockProcessSession(new
SharedSessionState(processor, new AtomicLong(0L)), processor, true, new
MockStateManager(processor), true);
-FlowFile flowFile = session.createFlowFile("hello, world".getBytes());
-final InputStream in = session.read(flowFile);
-final byte[] buffer = new byte[12];
-fillBuffer(in, buffer);
+private final Processor processor = new TestProcessor();
+private final SharedSessionState sharedState = new
SharedSessionState(processor, new AtomicLong(0L));
+private final MockStateManager stateManager = new
MockStateManager(processor);
+private final MockProcessSession session = new
MockProcessSession(sharedState, processor, stateManager);
+
+private final Processor statefulProcessor = new StatefulTestProcessor();
+private final SharedSessionState sharedStateOfStatefulProcessor = new
SharedSessionState(statefulProcessor, new AtomicLong(0L));
+private final MockStateManager stateManagerOfStatefulProcessor = new
MockStateManager(statefulProcessor);
+private final MockProcessSession sessionOfStatefulProcessor = new
MockProcessSession(sharedStateOfStatefulProcessor, statefulProcessor,
stateManagerOfStatefulProcessor);
Review Comment:
Thank you for your feedback @exceptionfactory.
I totally agree!
However, as far as I'm aware the default `@TestInstance` lifecycle of JUnit
5 is `PER_METHOD`.
That is, JUnit 5 is creating a new instance of the `TestMockProcessSession`
class for every test method invoked.
As a result, those objects aren't actually reused between test methods.
But there might be some global setting in the NiFi project changing this
default I'm not aware of.
I started out having the "stateful" variants in a single nested scope, but
ended up needing them in another nested scope, as far as I remember, which is
why I moved them up.
I'm fine fine with making these field non final and moving those
instantiations in a `@BeforeEach` if that's preferred. From my perspective
(using mainly Kotlin) it's just more verbose and less immutable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-12982 Extend test suite of MockProcessSession [nifi]
EndzeitBegins commented on code in PR #8589:
URL: https://github.com/apache/nifi/pull/8589#discussion_r1546391808
##
nifi-mock/src/test/java/org/apache/nifi/util/TestMockProcessSession.java:
##
@@ -45,189 +52,588 @@
public class TestMockProcessSession {
-@Test
-public void testReadWithoutCloseThrowsExceptionOnCommit() throws
IOException {
-final Processor processor = new PoorlyBehavedProcessor();
-final MockProcessSession session = new MockProcessSession(new
SharedSessionState(processor, new AtomicLong(0L)), processor, true, new
MockStateManager(processor), true);
-FlowFile flowFile = session.createFlowFile("hello, world".getBytes());
-final InputStream in = session.read(flowFile);
-final byte[] buffer = new byte[12];
-fillBuffer(in, buffer);
+private final Processor processor = new TestProcessor();
+private final SharedSessionState sharedState = new
SharedSessionState(processor, new AtomicLong(0L));
+private final MockStateManager stateManager = new
MockStateManager(processor);
+private final MockProcessSession session = new
MockProcessSession(sharedState, processor, stateManager);
+
+private final Processor statefulProcessor = new StatefulTestProcessor();
+private final SharedSessionState sharedStateOfStatefulProcessor = new
SharedSessionState(statefulProcessor, new AtomicLong(0L));
+private final MockStateManager stateManagerOfStatefulProcessor = new
MockStateManager(statefulProcessor);
+private final MockProcessSession sessionOfStatefulProcessor = new
MockProcessSession(sharedStateOfStatefulProcessor, statefulProcessor,
stateManagerOfStatefulProcessor);
Review Comment:
Thank you for your feedback @exceptionfactory.
I totally agree!
However, as far as I'm aware the default `@TestInstance` lifecycle of JUnit
5 is `PER_METHOD`.
That is, JUnit 5 is creating a new instance of the `TestMockProcessSession`
class for every test method invoked.
As a result, those objects aren't actually reused between test methods.
But there might be some global setting in the NiFi project changing this
default I'm not aware of.
I started out having the "stateful" variants in a single nested scope, but
ended up needing them in another nested scope, as far as I remember, which is
why I moved them up.
I'm fine fine with making these field non final and moving those
instantiations in a `@BeforeEach` if that's preferred. From my perspective
(using mainly Kotlin) it's just more verbose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-12982 Extend test suite of MockProcessSession [nifi]
EndzeitBegins commented on code in PR #8589:
URL: https://github.com/apache/nifi/pull/8589#discussion_r1546391808
##
nifi-mock/src/test/java/org/apache/nifi/util/TestMockProcessSession.java:
##
@@ -45,189 +52,588 @@
public class TestMockProcessSession {
-@Test
-public void testReadWithoutCloseThrowsExceptionOnCommit() throws
IOException {
-final Processor processor = new PoorlyBehavedProcessor();
-final MockProcessSession session = new MockProcessSession(new
SharedSessionState(processor, new AtomicLong(0L)), processor, true, new
MockStateManager(processor), true);
-FlowFile flowFile = session.createFlowFile("hello, world".getBytes());
-final InputStream in = session.read(flowFile);
-final byte[] buffer = new byte[12];
-fillBuffer(in, buffer);
+private final Processor processor = new TestProcessor();
+private final SharedSessionState sharedState = new
SharedSessionState(processor, new AtomicLong(0L));
+private final MockStateManager stateManager = new
MockStateManager(processor);
+private final MockProcessSession session = new
MockProcessSession(sharedState, processor, stateManager);
+
+private final Processor statefulProcessor = new StatefulTestProcessor();
+private final SharedSessionState sharedStateOfStatefulProcessor = new
SharedSessionState(statefulProcessor, new AtomicLong(0L));
+private final MockStateManager stateManagerOfStatefulProcessor = new
MockStateManager(statefulProcessor);
+private final MockProcessSession sessionOfStatefulProcessor = new
MockProcessSession(sharedStateOfStatefulProcessor, statefulProcessor,
stateManagerOfStatefulProcessor);
Review Comment:
Thank you for your feedback @exceptionfactory.
I totally agree!
However, as far as I'm aware the default `@TestInstance` lifecycle of JUnit
5 is `PER_METHOD`.
That is, JUnit 5 is creating a new instance of the `TestMockProcessSession`
class for every test method invoked.
As a result, those objects aren't actually reused between test methods.
I started out having the "stateful" variants in a single nested scope, but
ended up needing them in another nested scope, as far as I remember, which is
why I moved them up.
I'm fine fine with making these field non final and moving those
instantiations in a `@BeforeEach` if that's preferred. From my perspective
(using mainly Kotlin) it's just more verbose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-12896: Add Endpoint Override URL property for PutSNS processor [nifi]
dan-s1 commented on code in PR #8529:
URL: https://github.com/apache/nifi/pull/8529#discussion_r1546339608
##
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/sns/ITPutSNS.java:
##
@@ -36,43 +45,75 @@
*/
public class ITPutSNS {
-private static final String CREDENTIALS_FILE =
System.getProperty("user.home") + "/aws-credentials.properties";
-private final String TOPIC_ARN = "Add SNS ARN here";
+private static final DockerImageName localstackImage =
DockerImageName.parse("localstack/localstack:latest");
+
+private static final LocalStackContainer localstack = new
LocalStackContainer(localstackImage)
+.withServices(LocalStackContainer.Service.SNS);
+
+private static final String CREDENTIALS_FILE =
"src/test/resources/mock-aws-credentials.properties";
+private static String topicARN;
+private static SnsClient client;
@BeforeAll
-public static void assumeCredentialsFileExists() {
-Assumptions.assumeTrue(new File(CREDENTIALS_FILE).exists());
+public static void setup() throws InterruptedException {
+System.setProperty("software.amazon.awssdk.http.service.impl",
"software.amazon.awssdk.http.urlconnection.UrlConnectionSdkHttpService");
+localstack.start();
+
+client = SnsClient.builder()
+.endpointOverride(localstack.getEndpoint())
+.credentialsProvider(
+StaticCredentialsProvider.create(
+
AwsBasicCredentials.create(localstack.getAccessKey(), localstack.getSecretKey())
+)
+)
+.region(Region.of(localstack.getRegion()))
+.build();
+
+final CreateTopicResponse response =
client.createTopic(CreateTopicRequest.builder()
+.name("SnsSystemTest")
+.build());
+assertTrue(response.sdkHttpResponse().isSuccessful());
+topicARN = response.topicArn();
+}
+
+@AfterAll
+public static void shutdown() {
+client.close();
+localstack.stop();
}
@Test
public void testPublish() throws IOException {
-final TestRunner runner = TestRunners.newTestRunner(new PutSNS());
-AuthUtils.enableCredentialsFile(runner, CREDENTIALS_FILE);
-runner.setProperty(PutSNS.ARN, TOPIC_ARN);
+final TestRunner runner = initRunner(PutSNS.class);
+AuthUtils.enableAccessKey(runner, localstack.getAccessKey(),
localstack.getSecretKey());
Review Comment:
I did not realize that. Keep your changes if this is the only way to test
the override.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-12896: Add Endpoint Override URL property for PutSNS processor [nifi]
sujkm commented on code in PR #8529:
URL: https://github.com/apache/nifi/pull/8529#discussion_r1546337329
##
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/sns/ITPutSNS.java:
##
@@ -36,43 +45,75 @@
*/
public class ITPutSNS {
-private static final String CREDENTIALS_FILE =
System.getProperty("user.home") + "/aws-credentials.properties";
-private final String TOPIC_ARN = "Add SNS ARN here";
+private static final DockerImageName localstackImage =
DockerImageName.parse("localstack/localstack:latest");
+
+private static final LocalStackContainer localstack = new
LocalStackContainer(localstackImage)
+.withServices(LocalStackContainer.Service.SNS);
+
+private static final String CREDENTIALS_FILE =
"src/test/resources/mock-aws-credentials.properties";
+private static String topicARN;
+private static SnsClient client;
@BeforeAll
-public static void assumeCredentialsFileExists() {
-Assumptions.assumeTrue(new File(CREDENTIALS_FILE).exists());
+public static void setup() throws InterruptedException {
+System.setProperty("software.amazon.awssdk.http.service.impl",
"software.amazon.awssdk.http.urlconnection.UrlConnectionSdkHttpService");
+localstack.start();
+
+client = SnsClient.builder()
+.endpointOverride(localstack.getEndpoint())
+.credentialsProvider(
+StaticCredentialsProvider.create(
+
AwsBasicCredentials.create(localstack.getAccessKey(), localstack.getSecretKey())
+)
+)
+.region(Region.of(localstack.getRegion()))
+.build();
+
+final CreateTopicResponse response =
client.createTopic(CreateTopicRequest.builder()
+.name("SnsSystemTest")
+.build());
+assertTrue(response.sdkHttpResponse().isSuccessful());
+topicARN = response.topicArn();
+}
+
+@AfterAll
+public static void shutdown() {
+client.close();
+localstack.stop();
}
@Test
public void testPublish() throws IOException {
-final TestRunner runner = TestRunners.newTestRunner(new PutSNS());
-AuthUtils.enableCredentialsFile(runner, CREDENTIALS_FILE);
-runner.setProperty(PutSNS.ARN, TOPIC_ARN);
+final TestRunner runner = initRunner(PutSNS.class);
+AuthUtils.enableAccessKey(runner, localstack.getAccessKey(),
localstack.getSecretKey());
Review Comment:
I'm not sure how to test this since all the current examples of testing
endpoint override are done in the integration tests. Should I leave the
integration test as it was?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-12885 Add convenience method to `Record` to parse DateTime objects [nifi]
exceptionfactory commented on code in PR #8502:
URL: https://github.com/apache/nifi/pull/8502#discussion_r1546320728
##
nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/MapRecord.java:
##
@@ -279,10 +282,27 @@ public Boolean getAsBoolean(final String fieldName) {
}
@Override
-public Date getAsDate(final String fieldName, final String format) {
+public LocalDate getAsLocalDate(final String fieldName, final String
format) {
final FieldConverter converter =
StandardFieldConverterRegistry.getRegistry().getFieldConverter(LocalDate.class);
-final LocalDate localDate =
converter.convertField(getValue(fieldName), Optional.ofNullable(format),
fieldName);
-return localDate == null ? null : java.sql.Date.valueOf(localDate);
+return converter.convertField(getValue(fieldName),
Optional.ofNullable(format), fieldName);
+}
+
+@Override
+public OffsetDateTime getAsOffsetDateTime(final String fieldName, final
String format) {
+// Support three use-cases:
+// 1) the datetime string does not contain timezone information,
e.g. `2022-01-01 12:34:56`
+// 2) the datetime string contains timezone information, e.g.
`Sat, 01 Jan 2022 12:34:56 GMT`
+// 3) the datetime string contains timezone offset, e.g.
`2022-01-01 12:34:56.789-05:00`
+try {
+// First, try to parse the data as if it can be converted to
`LocalDateTime` then set `ZoneOffset.UTC`
+final FieldConverter localDateTimeConverter
=
StandardFieldConverterRegistry.getRegistry().getFieldConverter(LocalDateTime.class);
+final LocalDateTime localDateTime =
localDateTimeConverter.convertField(getValue(fieldName),
Optional.ofNullable(format), fieldName);
+return OffsetDateTime.of(localDateTime, ZoneOffset.UTC);
+} catch (DateTimeParseException exc) {
+// Parsing fails if offset is not provided, error: `Unable to
obtain OffsetDateTime from TemporalAccessor`
+final FieldConverter
offsetDateTimeConverter =
StandardFieldConverterRegistry.getRegistry().getFieldConverter(OffsetDateTime.class);
+return offsetDateTimeConverter.convertField(getValue(fieldName),
Optional.ofNullable(format), fieldName);
+}
Review Comment:
The try-catch approach to parsing is not optimal from a performance
perspective. There is a more general date parsing method that returns a
`TemporalAccessor`, which can be evaluated for various time fields. It looks
like this method might become more complicated in order to support a wider
variety of string formats.
On closer consideration, however, throwing an exception may be more
appropriate. Requesting an `OffsetDateTime` implies that the timezone is
available in source field. Returning an `OffsetDateTime` with `UTC` can create
confusion in calling functions, so this does not seem like the best approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-12875: Add unsuccessful HTTP status code handling logic [nifi]
exceptionfactory commented on code in PR #8484:
URL: https://github.com/apache/nifi/pull/8484#discussion_r1546313115
##
nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/java/org/apache/nifi/lookup/RestLookupService.java:
##
@@ -251,6 +264,8 @@ public void onEnabled(final ConfigurationContext context) {
buildHeaders(context);
urlTemplate = context.getProperty(URL);
+
+responseHandler = context.getProperty(PROP_RESPONSE_CODE_HANDLING);
Review Comment:
After changing the member variable type, the `asAllowableValue()` method can
be used to resolve the `enum` value from the property value.
##
nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/java/org/apache/nifi/lookup/RestLookupService.java:
##
@@ -214,6 +226,7 @@ protected List
getSupportedPropertyDescriptors() {
private volatile String basicUser;
private volatile String basicPass;
private volatile boolean isDigest;
+private volatile PropertyValue responseHandler;
Review Comment:
This can be changed to use the `ResponseHandlingStrategy` enum:
```suggestion
private volatile ResponseHandlingStrategy responseHandlingStrategy;
```
##
nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/java/org/apache/nifi/lookup/RestLookupService.java:
##
@@ -320,13 +335,20 @@ public Optional lookup(Map
coordinates, Map" : responseBody.string();
+throw new IOException("Failed to download content from URL " +
request.url() +
+": Response code was " + response.code() + ": " +
responseText);
Review Comment:
Recommend using String formatting instead of concatenation for readability:
```suggestion
throw new IOException("Request failed with HTTP %d for [%s]:
%s".formatted(response.code(), request.url(), responseText));
```
##
nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/java/org/apache/nifi/lookup/RestLookupService.java:
##
@@ -320,13 +335,20 @@ public Optional lookup(Map
coordinates, Map lookup(Map
coordinates, Map" : responseBody.string();
Review Comment:
Recommend replacing angle brackets with square brackets in the placeholder
message:
```suggestion
final String responseText = responseBody == null ? "[No
Message Received]" : responseBody.string();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-12982 Extend test suite of MockProcessSession [nifi]
exceptionfactory commented on code in PR #8589:
URL: https://github.com/apache/nifi/pull/8589#discussion_r1546278329
##
nifi-mock/src/test/java/org/apache/nifi/util/TestMockProcessSession.java:
##
@@ -45,189 +52,588 @@
public class TestMockProcessSession {
-@Test
-public void testReadWithoutCloseThrowsExceptionOnCommit() throws
IOException {
-final Processor processor = new PoorlyBehavedProcessor();
-final MockProcessSession session = new MockProcessSession(new
SharedSessionState(processor, new AtomicLong(0L)), processor, true, new
MockStateManager(processor), true);
-FlowFile flowFile = session.createFlowFile("hello, world".getBytes());
-final InputStream in = session.read(flowFile);
-final byte[] buffer = new byte[12];
-fillBuffer(in, buffer);
+private final Processor processor = new TestProcessor();
+private final SharedSessionState sharedState = new
SharedSessionState(processor, new AtomicLong(0L));
+private final MockStateManager stateManager = new
MockStateManager(processor);
+private final MockProcessSession session = new
MockProcessSession(sharedState, processor, stateManager);
+
+private final Processor statefulProcessor = new StatefulTestProcessor();
+private final SharedSessionState sharedStateOfStatefulProcessor = new
SharedSessionState(statefulProcessor, new AtomicLong(0L));
+private final MockStateManager stateManagerOfStatefulProcessor = new
MockStateManager(statefulProcessor);
+private final MockProcessSession sessionOfStatefulProcessor = new
MockProcessSession(sharedStateOfStatefulProcessor, statefulProcessor,
stateManagerOfStatefulProcessor);
Review Comment:
As a general rule, reusing an instance of an object across test method
invocations can lead to unexpected results.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (NIFI-12985) Refactor MockProcessSession using current API methods
endzeit created NIFI-12985: -- Summary: Refactor MockProcessSession using current API methods Key: NIFI-12985 URL: https://issues.apache.org/jira/browse/NIFI-12985 Project: Apache NiFi Issue Type: Sub-task Reporter: endzeit Assignee: endzeit -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] NIFI-12984 - Bump Snowflake Ingest SDK to 2.1.0 [nifi]
pvillard31 opened a new pull request, #8591: URL: https://github.com/apache/nifi/pull/8591 # Summary [NIFI-12984](https://issues.apache.org/jira/browse/NIFI-12984) - Bump Snowflake Ingest SDK to 2.1.0 # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [ ] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [ ] Pull Request based on current revision of the `main` branch - [ ] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-12984) Bump Snowflake Ingest SDK to 2.1.0
[ https://issues.apache.org/jira/browse/NIFI-12984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-12984: -- Status: Patch Available (was: Open) > Bump Snowflake Ingest SDK to 2.1.0 > -- > > Key: NIFI-12984 > URL: https://issues.apache.org/jira/browse/NIFI-12984 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Labels: backport-needed > > Bump Snowflake Ingest SDK to 2.1.0. We keep the JDBC version as-is given the > release notes. See: > [https://docs.snowflake.com/en/release-notes/clients-drivers/ingest-java-sdk-2024] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12984) Bump Snowflake Ingest SDK to 2.1.0
[ https://issues.apache.org/jira/browse/NIFI-12984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-12984: -- Labels: backport-needed (was: ) > Bump Snowflake Ingest SDK to 2.1.0 > -- > > Key: NIFI-12984 > URL: https://issues.apache.org/jira/browse/NIFI-12984 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Labels: backport-needed > > Bump Snowflake Ingest SDK to 2.1.0. We keep the JDBC version as-is given the > release notes. See: > [https://docs.snowflake.com/en/release-notes/clients-drivers/ingest-java-sdk-2024] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-12984) Bump Snowflake Ingest SDK to 2.1.0
Pierre Villard created NIFI-12984: - Summary: Bump Snowflake Ingest SDK to 2.1.0 Key: NIFI-12984 URL: https://issues.apache.org/jira/browse/NIFI-12984 Project: Apache NiFi Issue Type: Improvement Components: Extensions Reporter: Pierre Villard Assignee: Pierre Villard Bump Snowflake Ingest SDK to 2.1.0. We keep the JDBC version as-is given the release notes. See: [https://docs.snowflake.com/en/release-notes/clients-drivers/ingest-java-sdk-2024] -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12983: Qdrant vector store support [nifi]
Anush008 commented on code in PR #8590:
URL: https://github.com/apache/nifi/pull/8590#discussion_r1546015399
##
nifi-python-extensions/nifi-text-embeddings-module/src/main/python/vectorstores/QueryQdrant.py:
##
@@ -0,0 +1,192 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from langchain.vectorstores.qdrant import Qdrant
+from nifiapi.flowfiletransform import FlowFileTransform,
FlowFileTransformResult
+from nifiapi.properties import (
+PropertyDescriptor,
+StandardValidators,
+ExpressionLanguageScope,
+)
+import QueryUtils
+import json
+from EmbeddingUtils import (
+create_embedding_service,
+)
+
+from nifiapi.documentation import use_case
+
+from qdrant_client import QdrantClient
+
+import QdrantUtils
+
+
+@use_case(
+description="Semantically search for documents stored in Qdrant -
https://qdrant.tech/;,
+keywords=["qdrant", "embedding", "vector", "text", "vectorstore",
"search"],
+configuration="""
+Configure 'Collection Name' to the name of the Qdrant
collection to use.
+Configure 'Qdrant URL' to the fully qualified URL of the
Qdrant instance.
+Configure 'Qdrant API Key' to the API Key to use in order to
authenticate with Qdrant.
+Configure 'Prefer gRPC' to True if you want to use gRPC for
interfacing with Qdrant.
+Configure 'Use HTTPS' to True if you want to use TLS(HTTPS)
while interfacing with Qdrant.
+Configure 'Embedding Model' to indicate whether OpenAI
embeddings should be used or a HuggingFace embedding model should be used:
'Hugging Face Model' or 'OpenAI Model'
+Configure 'HuggingFace API Key' or 'OpenAI API Key', depending
on the chosen Embedding Model.
+Configure 'HuggingFace Model' or 'OpenAI Model' to the name of
the model to use.
+Configure 'Query' to the text of the query to send to Qdrant.
+Configure 'Number of Results' to the number of results to
return from Qdrant.
+Configure 'Metadata Filter' to apply an optional metadata
filter with the query. For example: { "author": "john.doe" }
+Configure 'Output Strategy' to indicate how the output should
be formatted: 'Row-Oriented', 'Text', or 'Column-Oriented'.
+Configure 'Results Field' to the name of the field to insert
the results, if the input FlowFile is JSON Formatted,.
+Configure 'Include Metadatas' to True if metadata should be
included in the output.
+Configure 'Include Distances' to True if distances should be
included in the output.
+""",
+)
+class QueryQdrant(FlowFileTransform):
+class Java:
+implements = ["org.apache.nifi.python.processor.FlowFileTransform"]
+
+class ProcessorDetails:
+version = "2.0.0-SNAPSHOT"
+description = "Queries Qdrant in order to gather a specified number of
documents that are most closely related to the given query."
+tags = [
+"qdrant",
+"vector",
+"vectordb",
+"vectorstore",
+"embeddings",
+"ai",
+"artificial intelligence",
+"ml",
+"machine learning",
+"text",
+"LLM",
+]
+
+QUERY = PropertyDescriptor(
+name="Query",
+description="The text of the query to send to Qdrant.",
+required=True,
+validators=[StandardValidators.NON_EMPTY_VALIDATOR],
+expression_language_scope=ExpressionLanguageScope.FLOWFILE_ATTRIBUTES,
+)
+NUMBER_OF_RESULTS = PropertyDescriptor(
+name="Number of Results",
+description="The number of results to return from Qdrant.",
+required=True,
+validators=[StandardValidators.POSITIVE_INTEGER_VALIDATOR],
+default_value="10",
+expression_language_scope=ExpressionLanguageScope.FLOWFILE_ATTRIBUTES,
+)
+FILTER = PropertyDescriptor(
+name="Metadata Filter",
+description='Optional metadata filter to apply with the query. For
example: { "author": "john.doe" }',
+required=False,
+
Re: [PR] NIFI-12983: Qdrant vector store support [nifi]
Anush008 commented on code in PR #8590:
URL: https://github.com/apache/nifi/pull/8590#discussion_r1546015399
##
nifi-python-extensions/nifi-text-embeddings-module/src/main/python/vectorstores/QueryQdrant.py:
##
@@ -0,0 +1,192 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from langchain.vectorstores.qdrant import Qdrant
+from nifiapi.flowfiletransform import FlowFileTransform,
FlowFileTransformResult
+from nifiapi.properties import (
+PropertyDescriptor,
+StandardValidators,
+ExpressionLanguageScope,
+)
+import QueryUtils
+import json
+from EmbeddingUtils import (
+create_embedding_service,
+)
+
+from nifiapi.documentation import use_case
+
+from qdrant_client import QdrantClient
+
+import QdrantUtils
+
+
+@use_case(
+description="Semantically search for documents stored in Qdrant -
https://qdrant.tech/;,
+keywords=["qdrant", "embedding", "vector", "text", "vectorstore",
"search"],
+configuration="""
+Configure 'Collection Name' to the name of the Qdrant
collection to use.
+Configure 'Qdrant URL' to the fully qualified URL of the
Qdrant instance.
+Configure 'Qdrant API Key' to the API Key to use in order to
authenticate with Qdrant.
+Configure 'Prefer gRPC' to True if you want to use gRPC for
interfacing with Qdrant.
+Configure 'Use HTTPS' to True if you want to use TLS(HTTPS)
while interfacing with Qdrant.
+Configure 'Embedding Model' to indicate whether OpenAI
embeddings should be used or a HuggingFace embedding model should be used:
'Hugging Face Model' or 'OpenAI Model'
+Configure 'HuggingFace API Key' or 'OpenAI API Key', depending
on the chosen Embedding Model.
+Configure 'HuggingFace Model' or 'OpenAI Model' to the name of
the model to use.
+Configure 'Query' to the text of the query to send to Qdrant.
+Configure 'Number of Results' to the number of results to
return from Qdrant.
+Configure 'Metadata Filter' to apply an optional metadata
filter with the query. For example: { "author": "john.doe" }
+Configure 'Output Strategy' to indicate how the output should
be formatted: 'Row-Oriented', 'Text', or 'Column-Oriented'.
+Configure 'Results Field' to the name of the field to insert
the results, if the input FlowFile is JSON Formatted,.
+Configure 'Include Metadatas' to True if metadata should be
included in the output.
+Configure 'Include Distances' to True if distances should be
included in the output.
+""",
+)
+class QueryQdrant(FlowFileTransform):
+class Java:
+implements = ["org.apache.nifi.python.processor.FlowFileTransform"]
+
+class ProcessorDetails:
+version = "2.0.0-SNAPSHOT"
+description = "Queries Qdrant in order to gather a specified number of
documents that are most closely related to the given query."
+tags = [
+"qdrant",
+"vector",
+"vectordb",
+"vectorstore",
+"embeddings",
+"ai",
+"artificial intelligence",
+"ml",
+"machine learning",
+"text",
+"LLM",
+]
+
+QUERY = PropertyDescriptor(
+name="Query",
+description="The text of the query to send to Qdrant.",
+required=True,
+validators=[StandardValidators.NON_EMPTY_VALIDATOR],
+expression_language_scope=ExpressionLanguageScope.FLOWFILE_ATTRIBUTES,
+)
+NUMBER_OF_RESULTS = PropertyDescriptor(
+name="Number of Results",
+description="The number of results to return from Qdrant.",
+required=True,
+validators=[StandardValidators.POSITIVE_INTEGER_VALIDATOR],
+default_value="10",
+expression_language_scope=ExpressionLanguageScope.FLOWFILE_ATTRIBUTES,
+)
+FILTER = PropertyDescriptor(
+name="Metadata Filter",
+description='Optional metadata filter to apply with the query. For
example: { "author": "john.doe" }',
+required=False,
+
[jira] [Assigned] (NIFI-12971) Provide a utility to detect leaked ProcessSession objects in unit tests or the UI
[
https://issues.apache.org/jira/browse/NIFI-12971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
endzeit reassigned NIFI-12971:
--
Assignee: endzeit
> Provide a utility to detect leaked ProcessSession objects in unit tests or
> the UI
> -
>
> Key: NIFI-12971
> URL: https://issues.apache.org/jira/browse/NIFI-12971
> Project: Apache NiFi
> Issue Type: Improvement
>Affects Versions: 1.25.0, 2.0.0-M2
>Reporter: endzeit
>Assignee: endzeit
>Priority: Major
>
> When developing processors for NiFi, developers need to implement
> [Processor|https://www.javadoc.io/doc/org.apache.nifi/nifi-api/latest/org/apache/nifi/processor/Processor.html].
> Most often this is done by extending
> [AbstractProcessor|https://www.javadoc.io/doc/org.apache.nifi/nifi-api/latest/org/apache/nifi/processor/AbstractProcessor.html]
> which ensures that the
> [ProcessSession|https://www.javadoc.io/doc/org.apache.nifi/nifi-api/latest/org/apache/nifi/processor/ProcessSession.html]
> used is either commited or, if that's not possible, rolled back.
> In cases where the developer needs more control over session management, they
> might extend from
> [AbstractSessionFactoryProcessor|https://www.javadoc.io/doc/org.apache.nifi/nifi-api/latest/org/apache/nifi/processor/AbstractSessionFactoryProcessor.html]
> instead, which allows to create and handle {{ProcessSessions}} on their own
> terms.
> When using the latter, developers need to ensure they handle all sessions
> created gracefully, that is, to commit or roll back all sessions they create,
> like {{AbstractProcessor}} ensures.
> However, failing to do so may lead to unnoticed leakage / lost of
> [FlowFile|https://www.javadoc.io/doc/org.apache.nifi/nifi-api/latest/org/apache/nifi/flowfile/FlowFile.html]s
> and their associated data.
> While data might be recovered from provenance, users are most likely not even
> aware of the data loss, as
> there won't be a bulletin visible in the UI indicating data loss due to no
> Exception occuring or am error being logged.
> The following is a minimal example, which reproduces the problem. All
> {{FlowFiles}} that enter the processor leak and eventually get lost when the
> processor is shut down.
> {code:java}
> @InputRequirement(INPUT_REQUIRED)
> public class LeakFlowFile extends AbstractSessionFactoryProcessor {
> public static final Relationship REL_SUCCESS = new Relationship.Builder()
> .name("success")
> .description("All FlowFiles are routed to this relationship.")
> .build();
> private static final Set RELATIONSHIPS =
> Set.of(REL_SUCCESS);
> @Override
> public Set getRelationships() {
> return RELATIONSHIPS;
> }
> @Override
> public void onTrigger(ProcessContext context, ProcessSessionFactory
> sessionFactory) throws ProcessException {
> ProcessSession session = sessionFactory.createSession();
> FlowFile flowFile = session.get();
> if (flowFile == null) {
> return;
> }
> session.transfer(flowFile, REL_SUCCESS);
> // whoops, no commit or rollback
> }
> } {code}
> While the issue is quite obvious in this example, it might not be for more
> complex processors, e.g. when based on
> [BinFiles|https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-extension-utils/nifi-bin-manager/src/main/java/org/apache/nifi/processor/util/bin/BinFiles.java].
> In case a developer misses to commit / rollback the session in
> {{{}processBin{}}}, the same behaviour can be observed.
> The behavior also is not made visible by tests. The following test passes,
> even though the session has not been committed (or rolled back).
> {code:java}
> class LeakFlowFileTest {
> private final TestRunner testRunner =
> TestRunners.newTestRunner(LeakFlowFile.class);
> @Test
> void doesNotDetectLeak() {
> testRunner.enqueue("some data");
> testRunner.run();
> testRunner.assertAllFlowFilesTransferred(LeakFlowFile.REL_SUCCESS, 1);
> }
> } {code}
>
> I would like to propose enhancements to NiFi in order to ease detection of
> such implementation faults or even confine the harm they might incur.
> One approach is to extend the capabilities of TestRunner such that on
> shutdown of a tested processor, it checks whether all sessions that were
> created during the test and had a change associated with them, e.g. pulling a
> FlowFile or adjusting state, do not have pending changes left but were
> properly handled, e.g. by committing the session. In case that's not the
> case, the test may fail, similar to trying to commit a session where
> FlowFiles haven't been transferred / removed.
> This way, developers that test their processors
Re: [PR] NIFI-12972 - Only show selected relationships in read-only connection details [nifi]
pvillard31 commented on code in PR #8582:
URL: https://github.com/apache/nifi/pull/8582#discussion_r1546026094
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/js/nf/nf-connection-details.js:
##
@@ -496,10 +496,10 @@
var availableRelationships =
connection.availableRelationships;
var selectedRelationships =
connection.selectedRelationships;
-// show the available relationship if applicable
+// show the selected relationships if applicable
if
(nfCommon.isDefinedAndNotNull(availableRelationships) ||
nfCommon.isDefinedAndNotNull(selectedRelationships)) {
-// populate the available connections
-$.each(availableRelationships, function (i, name) {
+// populate the selected connections
+$.each(selectedRelationships, function (i, name) {
createRelationshipOption(name);
});
Review Comment:
Thanks Matt for catching this. I just pushed a new commit with the suggested
changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-12983: Qdrant vector store support [nifi]
pvillard31 commented on code in PR #8590:
URL: https://github.com/apache/nifi/pull/8590#discussion_r1546024211
##
nifi-python-extensions/nifi-text-embeddings-module/src/main/python/vectorstores/QueryQdrant.py:
##
@@ -0,0 +1,192 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from langchain.vectorstores.qdrant import Qdrant
+from nifiapi.flowfiletransform import FlowFileTransform,
FlowFileTransformResult
+from nifiapi.properties import (
+PropertyDescriptor,
+StandardValidators,
+ExpressionLanguageScope,
+)
+import QueryUtils
+import json
+from EmbeddingUtils import (
+create_embedding_service,
+)
+
+from nifiapi.documentation import use_case
+
+from qdrant_client import QdrantClient
+
+import QdrantUtils
+
+
+@use_case(
+description="Semantically search for documents stored in Qdrant -
https://qdrant.tech/;,
+keywords=["qdrant", "embedding", "vector", "text", "vectorstore",
"search"],
+configuration="""
+Configure 'Collection Name' to the name of the Qdrant
collection to use.
+Configure 'Qdrant URL' to the fully qualified URL of the
Qdrant instance.
+Configure 'Qdrant API Key' to the API Key to use in order to
authenticate with Qdrant.
+Configure 'Prefer gRPC' to True if you want to use gRPC for
interfacing with Qdrant.
+Configure 'Use HTTPS' to True if you want to use TLS(HTTPS)
while interfacing with Qdrant.
+Configure 'Embedding Model' to indicate whether OpenAI
embeddings should be used or a HuggingFace embedding model should be used:
'Hugging Face Model' or 'OpenAI Model'
+Configure 'HuggingFace API Key' or 'OpenAI API Key', depending
on the chosen Embedding Model.
+Configure 'HuggingFace Model' or 'OpenAI Model' to the name of
the model to use.
+Configure 'Query' to the text of the query to send to Qdrant.
+Configure 'Number of Results' to the number of results to
return from Qdrant.
+Configure 'Metadata Filter' to apply an optional metadata
filter with the query. For example: { "author": "john.doe" }
+Configure 'Output Strategy' to indicate how the output should
be formatted: 'Row-Oriented', 'Text', or 'Column-Oriented'.
+Configure 'Results Field' to the name of the field to insert
the results, if the input FlowFile is JSON Formatted,.
+Configure 'Include Metadatas' to True if metadata should be
included in the output.
+Configure 'Include Distances' to True if distances should be
included in the output.
+""",
+)
+class QueryQdrant(FlowFileTransform):
+class Java:
+implements = ["org.apache.nifi.python.processor.FlowFileTransform"]
+
+class ProcessorDetails:
+version = "2.0.0-SNAPSHOT"
+description = "Queries Qdrant in order to gather a specified number of
documents that are most closely related to the given query."
+tags = [
+"qdrant",
+"vector",
+"vectordb",
+"vectorstore",
+"embeddings",
+"ai",
+"artificial intelligence",
+"ml",
+"machine learning",
+"text",
+"LLM",
+]
+
+QUERY = PropertyDescriptor(
+name="Query",
+description="The text of the query to send to Qdrant.",
+required=True,
+validators=[StandardValidators.NON_EMPTY_VALIDATOR],
+expression_language_scope=ExpressionLanguageScope.FLOWFILE_ATTRIBUTES,
+)
+NUMBER_OF_RESULTS = PropertyDescriptor(
+name="Number of Results",
+description="The number of results to return from Qdrant.",
+required=True,
+validators=[StandardValidators.POSITIVE_INTEGER_VALIDATOR],
+default_value="10",
+expression_language_scope=ExpressionLanguageScope.FLOWFILE_ATTRIBUTES,
+)
+FILTER = PropertyDescriptor(
+name="Metadata Filter",
+description='Optional metadata filter to apply with the query. For
example: { "author": "john.doe" }',
+required=False,
+
Re: [PR] NIFI-12983: Qdrant vector store support [nifi]
Anush008 commented on code in PR #8590:
URL: https://github.com/apache/nifi/pull/8590#discussion_r1546015399
##
nifi-python-extensions/nifi-text-embeddings-module/src/main/python/vectorstores/QueryQdrant.py:
##
@@ -0,0 +1,192 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from langchain.vectorstores.qdrant import Qdrant
+from nifiapi.flowfiletransform import FlowFileTransform,
FlowFileTransformResult
+from nifiapi.properties import (
+PropertyDescriptor,
+StandardValidators,
+ExpressionLanguageScope,
+)
+import QueryUtils
+import json
+from EmbeddingUtils import (
+create_embedding_service,
+)
+
+from nifiapi.documentation import use_case
+
+from qdrant_client import QdrantClient
+
+import QdrantUtils
+
+
+@use_case(
+description="Semantically search for documents stored in Qdrant -
https://qdrant.tech/;,
+keywords=["qdrant", "embedding", "vector", "text", "vectorstore",
"search"],
+configuration="""
+Configure 'Collection Name' to the name of the Qdrant
collection to use.
+Configure 'Qdrant URL' to the fully qualified URL of the
Qdrant instance.
+Configure 'Qdrant API Key' to the API Key to use in order to
authenticate with Qdrant.
+Configure 'Prefer gRPC' to True if you want to use gRPC for
interfacing with Qdrant.
+Configure 'Use HTTPS' to True if you want to use TLS(HTTPS)
while interfacing with Qdrant.
+Configure 'Embedding Model' to indicate whether OpenAI
embeddings should be used or a HuggingFace embedding model should be used:
'Hugging Face Model' or 'OpenAI Model'
+Configure 'HuggingFace API Key' or 'OpenAI API Key', depending
on the chosen Embedding Model.
+Configure 'HuggingFace Model' or 'OpenAI Model' to the name of
the model to use.
+Configure 'Query' to the text of the query to send to Qdrant.
+Configure 'Number of Results' to the number of results to
return from Qdrant.
+Configure 'Metadata Filter' to apply an optional metadata
filter with the query. For example: { "author": "john.doe" }
+Configure 'Output Strategy' to indicate how the output should
be formatted: 'Row-Oriented', 'Text', or 'Column-Oriented'.
+Configure 'Results Field' to the name of the field to insert
the results, if the input FlowFile is JSON Formatted,.
+Configure 'Include Metadatas' to True if metadata should be
included in the output.
+Configure 'Include Distances' to True if distances should be
included in the output.
+""",
+)
+class QueryQdrant(FlowFileTransform):
+class Java:
+implements = ["org.apache.nifi.python.processor.FlowFileTransform"]
+
+class ProcessorDetails:
+version = "2.0.0-SNAPSHOT"
+description = "Queries Qdrant in order to gather a specified number of
documents that are most closely related to the given query."
+tags = [
+"qdrant",
+"vector",
+"vectordb",
+"vectorstore",
+"embeddings",
+"ai",
+"artificial intelligence",
+"ml",
+"machine learning",
+"text",
+"LLM",
+]
+
+QUERY = PropertyDescriptor(
+name="Query",
+description="The text of the query to send to Qdrant.",
+required=True,
+validators=[StandardValidators.NON_EMPTY_VALIDATOR],
+expression_language_scope=ExpressionLanguageScope.FLOWFILE_ATTRIBUTES,
+)
+NUMBER_OF_RESULTS = PropertyDescriptor(
+name="Number of Results",
+description="The number of results to return from Qdrant.",
+required=True,
+validators=[StandardValidators.POSITIVE_INTEGER_VALIDATOR],
+default_value="10",
+expression_language_scope=ExpressionLanguageScope.FLOWFILE_ATTRIBUTES,
+)
+FILTER = PropertyDescriptor(
+name="Metadata Filter",
+description='Optional metadata filter to apply with the query. For
example: { "author": "john.doe" }',
+required=False,
+
Re: [PR] NIFI-12983: Qdrant vector store support [nifi]
pvillard31 commented on code in PR #8590:
URL: https://github.com/apache/nifi/pull/8590#discussion_r1546012716
##
nifi-python-extensions/nifi-text-embeddings-module/src/main/python/vectorstores/QueryQdrant.py:
##
@@ -0,0 +1,192 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from langchain.vectorstores.qdrant import Qdrant
+from nifiapi.flowfiletransform import FlowFileTransform,
FlowFileTransformResult
+from nifiapi.properties import (
+PropertyDescriptor,
+StandardValidators,
+ExpressionLanguageScope,
+)
+import QueryUtils
+import json
+from EmbeddingUtils import (
+create_embedding_service,
+)
+
+from nifiapi.documentation import use_case
+
+from qdrant_client import QdrantClient
+
+import QdrantUtils
+
+
+@use_case(
+description="Semantically search for documents stored in Qdrant -
https://qdrant.tech/;,
+keywords=["qdrant", "embedding", "vector", "text", "vectorstore",
"search"],
+configuration="""
+Configure 'Collection Name' to the name of the Qdrant
collection to use.
+Configure 'Qdrant URL' to the fully qualified URL of the
Qdrant instance.
+Configure 'Qdrant API Key' to the API Key to use in order to
authenticate with Qdrant.
+Configure 'Prefer gRPC' to True if you want to use gRPC for
interfacing with Qdrant.
+Configure 'Use HTTPS' to True if you want to use TLS(HTTPS)
while interfacing with Qdrant.
+Configure 'Embedding Model' to indicate whether OpenAI
embeddings should be used or a HuggingFace embedding model should be used:
'Hugging Face Model' or 'OpenAI Model'
+Configure 'HuggingFace API Key' or 'OpenAI API Key', depending
on the chosen Embedding Model.
+Configure 'HuggingFace Model' or 'OpenAI Model' to the name of
the model to use.
+Configure 'Query' to the text of the query to send to Qdrant.
+Configure 'Number of Results' to the number of results to
return from Qdrant.
+Configure 'Metadata Filter' to apply an optional metadata
filter with the query. For example: { "author": "john.doe" }
+Configure 'Output Strategy' to indicate how the output should
be formatted: 'Row-Oriented', 'Text', or 'Column-Oriented'.
+Configure 'Results Field' to the name of the field to insert
the results, if the input FlowFile is JSON Formatted,.
+Configure 'Include Metadatas' to True if metadata should be
included in the output.
+Configure 'Include Distances' to True if distances should be
included in the output.
+""",
+)
+class QueryQdrant(FlowFileTransform):
+class Java:
+implements = ["org.apache.nifi.python.processor.FlowFileTransform"]
+
+class ProcessorDetails:
+version = "2.0.0-SNAPSHOT"
+description = "Queries Qdrant in order to gather a specified number of
documents that are most closely related to the given query."
+tags = [
+"qdrant",
+"vector",
+"vectordb",
+"vectorstore",
+"embeddings",
+"ai",
+"artificial intelligence",
+"ml",
+"machine learning",
+"text",
+"LLM",
+]
+
+QUERY = PropertyDescriptor(
+name="Query",
+description="The text of the query to send to Qdrant.",
+required=True,
+validators=[StandardValidators.NON_EMPTY_VALIDATOR],
+expression_language_scope=ExpressionLanguageScope.FLOWFILE_ATTRIBUTES,
+)
+NUMBER_OF_RESULTS = PropertyDescriptor(
+name="Number of Results",
+description="The number of results to return from Qdrant.",
+required=True,
+validators=[StandardValidators.POSITIVE_INTEGER_VALIDATOR],
+default_value="10",
+expression_language_scope=ExpressionLanguageScope.FLOWFILE_ATTRIBUTES,
+)
+FILTER = PropertyDescriptor(
+name="Metadata Filter",
+description='Optional metadata filter to apply with the query. For
example: { "author": "john.doe" }',
+required=False,
+
Re: [PR] NIFI-12983: Qdrant vector store support [nifi]
Anush008 commented on code in PR #8590:
URL: https://github.com/apache/nifi/pull/8590#discussion_r1546008281
##
nifi-python-extensions/nifi-text-embeddings-module/src/main/python/vectorstores/QueryQdrant.py:
##
@@ -0,0 +1,192 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from langchain.vectorstores.qdrant import Qdrant
+from nifiapi.flowfiletransform import FlowFileTransform,
FlowFileTransformResult
+from nifiapi.properties import (
+PropertyDescriptor,
+StandardValidators,
+ExpressionLanguageScope,
+)
+import QueryUtils
+import json
+from EmbeddingUtils import (
+create_embedding_service,
+)
+
+from nifiapi.documentation import use_case
+
+from qdrant_client import QdrantClient
+
+import QdrantUtils
+
+
+@use_case(
+description="Semantically search for documents stored in Qdrant -
https://qdrant.tech/;,
+keywords=["qdrant", "embedding", "vector", "text", "vectorstore",
"search"],
+configuration="""
+Configure 'Collection Name' to the name of the Qdrant
collection to use.
+Configure 'Qdrant URL' to the fully qualified URL of the
Qdrant instance.
+Configure 'Qdrant API Key' to the API Key to use in order to
authenticate with Qdrant.
+Configure 'Prefer gRPC' to True if you want to use gRPC for
interfacing with Qdrant.
+Configure 'Use HTTPS' to True if you want to use TLS(HTTPS)
while interfacing with Qdrant.
+Configure 'Embedding Model' to indicate whether OpenAI
embeddings should be used or a HuggingFace embedding model should be used:
'Hugging Face Model' or 'OpenAI Model'
+Configure 'HuggingFace API Key' or 'OpenAI API Key', depending
on the chosen Embedding Model.
+Configure 'HuggingFace Model' or 'OpenAI Model' to the name of
the model to use.
+Configure 'Query' to the text of the query to send to Qdrant.
+Configure 'Number of Results' to the number of results to
return from Qdrant.
+Configure 'Metadata Filter' to apply an optional metadata
filter with the query. For example: { "author": "john.doe" }
+Configure 'Output Strategy' to indicate how the output should
be formatted: 'Row-Oriented', 'Text', or 'Column-Oriented'.
+Configure 'Results Field' to the name of the field to insert
the results, if the input FlowFile is JSON Formatted,.
+Configure 'Include Metadatas' to True if metadata should be
included in the output.
+Configure 'Include Distances' to True if distances should be
included in the output.
+""",
+)
+class QueryQdrant(FlowFileTransform):
+class Java:
+implements = ["org.apache.nifi.python.processor.FlowFileTransform"]
+
+class ProcessorDetails:
+version = "2.0.0-SNAPSHOT"
+description = "Queries Qdrant in order to gather a specified number of
documents that are most closely related to the given query."
+tags = [
+"qdrant",
+"vector",
+"vectordb",
+"vectorstore",
+"embeddings",
+"ai",
+"artificial intelligence",
+"ml",
+"machine learning",
+"text",
+"LLM",
+]
+
+QUERY = PropertyDescriptor(
+name="Query",
+description="The text of the query to send to Qdrant.",
+required=True,
+validators=[StandardValidators.NON_EMPTY_VALIDATOR],
+expression_language_scope=ExpressionLanguageScope.FLOWFILE_ATTRIBUTES,
+)
+NUMBER_OF_RESULTS = PropertyDescriptor(
+name="Number of Results",
+description="The number of results to return from Qdrant.",
+required=True,
+validators=[StandardValidators.POSITIVE_INTEGER_VALIDATOR],
+default_value="10",
+expression_language_scope=ExpressionLanguageScope.FLOWFILE_ATTRIBUTES,
+)
+FILTER = PropertyDescriptor(
+name="Metadata Filter",
+description='Optional metadata filter to apply with the query. For
example: { "author": "john.doe" }',
+required=False,
+
Re: [PR] NIFI-12983: Qdrant vector store support [nifi]
pvillard31 commented on code in PR #8590:
URL: https://github.com/apache/nifi/pull/8590#discussion_r1546004979
##
nifi-python-extensions/nifi-text-embeddings-module/src/main/python/vectorstores/QueryQdrant.py:
##
@@ -0,0 +1,192 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from langchain.vectorstores.qdrant import Qdrant
+from nifiapi.flowfiletransform import FlowFileTransform,
FlowFileTransformResult
+from nifiapi.properties import (
+PropertyDescriptor,
+StandardValidators,
+ExpressionLanguageScope,
+)
+import QueryUtils
+import json
+from EmbeddingUtils import (
+create_embedding_service,
+)
+
+from nifiapi.documentation import use_case
+
+from qdrant_client import QdrantClient
+
+import QdrantUtils
+
+
+@use_case(
+description="Semantically search for documents stored in Qdrant -
https://qdrant.tech/;,
+keywords=["qdrant", "embedding", "vector", "text", "vectorstore",
"search"],
+configuration="""
+Configure 'Collection Name' to the name of the Qdrant
collection to use.
+Configure 'Qdrant URL' to the fully qualified URL of the
Qdrant instance.
+Configure 'Qdrant API Key' to the API Key to use in order to
authenticate with Qdrant.
+Configure 'Prefer gRPC' to True if you want to use gRPC for
interfacing with Qdrant.
+Configure 'Use HTTPS' to True if you want to use TLS(HTTPS)
while interfacing with Qdrant.
+Configure 'Embedding Model' to indicate whether OpenAI
embeddings should be used or a HuggingFace embedding model should be used:
'Hugging Face Model' or 'OpenAI Model'
+Configure 'HuggingFace API Key' or 'OpenAI API Key', depending
on the chosen Embedding Model.
+Configure 'HuggingFace Model' or 'OpenAI Model' to the name of
the model to use.
+Configure 'Query' to the text of the query to send to Qdrant.
+Configure 'Number of Results' to the number of results to
return from Qdrant.
+Configure 'Metadata Filter' to apply an optional metadata
filter with the query. For example: { "author": "john.doe" }
+Configure 'Output Strategy' to indicate how the output should
be formatted: 'Row-Oriented', 'Text', or 'Column-Oriented'.
+Configure 'Results Field' to the name of the field to insert
the results, if the input FlowFile is JSON Formatted,.
+Configure 'Include Metadatas' to True if metadata should be
included in the output.
+Configure 'Include Distances' to True if distances should be
included in the output.
+""",
+)
+class QueryQdrant(FlowFileTransform):
+class Java:
+implements = ["org.apache.nifi.python.processor.FlowFileTransform"]
+
+class ProcessorDetails:
+version = "2.0.0-SNAPSHOT"
+description = "Queries Qdrant in order to gather a specified number of
documents that are most closely related to the given query."
+tags = [
+"qdrant",
+"vector",
+"vectordb",
+"vectorstore",
+"embeddings",
+"ai",
+"artificial intelligence",
+"ml",
+"machine learning",
+"text",
+"LLM",
+]
+
+QUERY = PropertyDescriptor(
+name="Query",
+description="The text of the query to send to Qdrant.",
+required=True,
+validators=[StandardValidators.NON_EMPTY_VALIDATOR],
+expression_language_scope=ExpressionLanguageScope.FLOWFILE_ATTRIBUTES,
+)
+NUMBER_OF_RESULTS = PropertyDescriptor(
+name="Number of Results",
+description="The number of results to return from Qdrant.",
+required=True,
+validators=[StandardValidators.POSITIVE_INTEGER_VALIDATOR],
+default_value="10",
+expression_language_scope=ExpressionLanguageScope.FLOWFILE_ATTRIBUTES,
+)
+FILTER = PropertyDescriptor(
+name="Metadata Filter",
+description='Optional metadata filter to apply with the query. For
example: { "author": "john.doe" }',
+required=False,
+
Re: [PR] NIFI-12983: Qdrant vector store support [nifi]
Anush008 commented on code in PR #8590:
URL: https://github.com/apache/nifi/pull/8590#discussion_r1545992040
##
nifi-python-extensions/nifi-text-embeddings-module/src/main/python/vectorstores/QueryQdrant.py:
##
@@ -0,0 +1,192 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from langchain.vectorstores.qdrant import Qdrant
+from nifiapi.flowfiletransform import FlowFileTransform,
FlowFileTransformResult
+from nifiapi.properties import (
+PropertyDescriptor,
+StandardValidators,
+ExpressionLanguageScope,
+)
+import QueryUtils
+import json
+from EmbeddingUtils import (
+create_embedding_service,
+)
+
+from nifiapi.documentation import use_case
+
+from qdrant_client import QdrantClient
+
+import QdrantUtils
+
+
+@use_case(
+description="Semantically search for documents stored in Qdrant -
https://qdrant.tech/;,
+keywords=["qdrant", "embedding", "vector", "text", "vectorstore",
"search"],
+configuration="""
+Configure 'Collection Name' to the name of the Qdrant
collection to use.
+Configure 'Qdrant URL' to the fully qualified URL of the
Qdrant instance.
+Configure 'Qdrant API Key' to the API Key to use in order to
authenticate with Qdrant.
+Configure 'Prefer gRPC' to True if you want to use gRPC for
interfacing with Qdrant.
+Configure 'Use HTTPS' to True if you want to use TLS(HTTPS)
while interfacing with Qdrant.
+Configure 'Embedding Model' to indicate whether OpenAI
embeddings should be used or a HuggingFace embedding model should be used:
'Hugging Face Model' or 'OpenAI Model'
+Configure 'HuggingFace API Key' or 'OpenAI API Key', depending
on the chosen Embedding Model.
+Configure 'HuggingFace Model' or 'OpenAI Model' to the name of
the model to use.
+Configure 'Query' to the text of the query to send to Qdrant.
+Configure 'Number of Results' to the number of results to
return from Qdrant.
+Configure 'Metadata Filter' to apply an optional metadata
filter with the query. For example: { "author": "john.doe" }
+Configure 'Output Strategy' to indicate how the output should
be formatted: 'Row-Oriented', 'Text', or 'Column-Oriented'.
+Configure 'Results Field' to the name of the field to insert
the results, if the input FlowFile is JSON Formatted,.
+Configure 'Include Metadatas' to True if metadata should be
included in the output.
+Configure 'Include Distances' to True if distances should be
included in the output.
+""",
+)
+class QueryQdrant(FlowFileTransform):
+class Java:
+implements = ["org.apache.nifi.python.processor.FlowFileTransform"]
+
+class ProcessorDetails:
+version = "2.0.0-SNAPSHOT"
+description = "Queries Qdrant in order to gather a specified number of
documents that are most closely related to the given query."
+tags = [
+"qdrant",
+"vector",
+"vectordb",
+"vectorstore",
+"embeddings",
+"ai",
+"artificial intelligence",
+"ml",
+"machine learning",
+"text",
+"LLM",
+]
+
+QUERY = PropertyDescriptor(
+name="Query",
+description="The text of the query to send to Qdrant.",
+required=True,
+validators=[StandardValidators.NON_EMPTY_VALIDATOR],
+expression_language_scope=ExpressionLanguageScope.FLOWFILE_ATTRIBUTES,
+)
+NUMBER_OF_RESULTS = PropertyDescriptor(
+name="Number of Results",
+description="The number of results to return from Qdrant.",
+required=True,
+validators=[StandardValidators.POSITIVE_INTEGER_VALIDATOR],
+default_value="10",
+expression_language_scope=ExpressionLanguageScope.FLOWFILE_ATTRIBUTES,
+)
+FILTER = PropertyDescriptor(
+name="Metadata Filter",
+description='Optional metadata filter to apply with the query. For
example: { "author": "john.doe" }',
+required=False,
+
[jira] [Updated] (NIFI-12983) Add support for Qdrant vector store
[ https://issues.apache.org/jira/browse/NIFI-12983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anush updated NIFI-12983: - Description: Qdrant - [https://github.com/qdrant/qdrant], is an open-source vector search engine and database, governed by the Apache-2.0 license. Qdrant ranks amongst the most performant and most used vector databases available today. - [https://qdrant.tech/benchmarks]/ - [https://ossinsight.io/collections/vector-search-engine]/ This is a proposal to add support for Qdrant via an extension in NiFi. Associated pull-request: [https://github.com/apache/nifi/pull/8590] was: Qdrant - [https://github.com/qdrant/qdrant], is an open-source vector search engine and database, governed by the Apache-2.0 license. Qdrant ranks amongst the most performant and most used vector databases available today. - [https://qdrant.tech/benchmarks]/ - [https://ossinsight.io/collections/vector-search-engine]/ This is a proposal to add support for Qdrant via an extension in NiFi. Associated pull-request:[https://github.com/apache/nifi/pull/8590] > Add support for Qdrant vector store > --- > > Key: NIFI-12983 > URL: https://issues.apache.org/jira/browse/NIFI-12983 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Anush >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Qdrant - [https://github.com/qdrant/qdrant], is an open-source vector search > engine and database, governed by the Apache-2.0 license. > Qdrant ranks amongst the most performant and most used vector databases > available today. > - [https://qdrant.tech/benchmarks]/ > - [https://ossinsight.io/collections/vector-search-engine]/ > This is a proposal to add support for Qdrant via an extension in NiFi. > Associated pull-request: [https://github.com/apache/nifi/pull/8590] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12983) Add support for Qdrant vector store
[ https://issues.apache.org/jira/browse/NIFI-12983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anush updated NIFI-12983: - Description: Qdrant - [https://github.com/qdrant/qdrant], is an open-source vector search engine and database, governed by the Apache-2.0 license. Qdrant ranks amongst the most performant and most used vector databases available today. - [https://qdrant.tech/benchmarks]/ - [https://ossinsight.io/collections/vector-search-engine]/ This is a proposal to add support for Qdrant via an extension in NiFi. Associated pull-request:[https://github.com/apache/nifi/pull/8590] was: Qdrant - [https://github.com/qdrant/qdrant], is an open-source vector search engine and database, governed by the Apache-2.0 license. Qdrant ranks amongst the most performant and most used vector databases available today. - [https://qdrant.tech/benchmarks]/ - [https://ossinsight.io/collections/vector-search-engine]/ This is a proposal to add support for Qdrant via an extension in NiFi. Associated pull-request:https://github.com/apache/nifi/pull/8590 > Add support for Qdrant vector store > --- > > Key: NIFI-12983 > URL: https://issues.apache.org/jira/browse/NIFI-12983 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Anush >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Qdrant - [https://github.com/qdrant/qdrant], is an open-source vector search > engine and database, governed by the Apache-2.0 license. > Qdrant ranks amongst the most performant and most used vector databases > available today. > - [https://qdrant.tech/benchmarks]/ > - [https://ossinsight.io/collections/vector-search-engine]/ > This is a proposal to add support for Qdrant via an extension in NiFi. > Associated pull-request:[https://github.com/apache/nifi/pull/8590] -- This message was sent by Atlassian Jira (v8.20.10#820010)
