[GitHub] [nifi] ChrisSamo632 commented on a diff in pull request #7537: NIFI-11778 Initial check in of refactored Groovy tests in nifi-elasticsearch-restapi-processors to Java (and JUnit 5)
ChrisSamo632 commented on code in PR #7537:
URL: https://github.com/apache/nifi/pull/7537#discussion_r1282108131
##
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/test/java/org/apache/nifi/processors/elasticsearch/AbstractPaginatedJsonQueryElasticsearchTest.java:
##
@@ -0,0 +1,350 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License") you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.nifi.processors.elasticsearch.api.AggregationResultsFormat;
+import org.apache.nifi.processors.elasticsearch.api.PaginationType;
+import org.apache.nifi.processors.elasticsearch.api.ResultOutputStrategy;
+import org.apache.nifi.processors.elasticsearch.api.SearchResultsFormat;
+import org.apache.nifi.provenance.ProvenanceEventType;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public abstract class AbstractPaginatedJsonQueryElasticsearchTest extends
AbstractJsonQueryElasticsearchTest {
+ private static final String TEST_DIR =
"src/test/resources/AbstractPaginatedJsonQueryElasticsearchTest";
+private static final String MATCH_ALL;
+ private static final String MATCH_ALL_WITH_SORT_BY_MESSAGE;
+ private static final String MATCH_ALL_WITH_SORT_BY_MSG;
+
+ static {
+ try {
+ MATCH_ALL =
Files.readString(Paths.get("src/test/resources/common/matchAllQuery.json"));
+ MATCH_ALL_WITH_SORT_BY_MESSAGE =
Files.readString(Paths.get(TEST_DIR, "matchAllWithSortByMessageQuery.json"));
+ MATCH_ALL_WITH_SORT_BY_MSG =
Files.readString(Paths.get(TEST_DIR,"matchAllWithSortByMsgQueryWithoutSize.json"));
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
Review Comment:
Yes, exactly that - you're correct that `super` isn't a thing (my bad), but
you should be able to refer to
`AbstractJsonQueryElasticsearchTest.setUpBeforeClass()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] ChrisSamo632 commented on a diff in pull request #7537: NIFI-11778 Initial check in of refactored Groovy tests in nifi-elasticsearch-restapi-processors to Java (and JUnit 5)
ChrisSamo632 commented on code in PR #7537:
URL: https://github.com/apache/nifi/pull/7537#discussion_r1282661035
##
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/test/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchRecordTest.java:
##
@@ -0,0 +1,784 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License") you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.avro.Schema;
+import org.apache.nifi.avro.AvroTypeUtil;
+import org.apache.nifi.elasticsearch.IndexOperationRequest;
+import org.apache.nifi.elasticsearch.IndexOperationResponse;
+import org.apache.nifi.json.JsonRecordSetWriter;
+import org.apache.nifi.json.JsonTreeReader;
+import org.apache.nifi.processors.elasticsearch.mock.MockBulkLoadClientService;
+import org.apache.nifi.provenance.ProvenanceEventType;
+import org.apache.nifi.schema.access.SchemaAccessUtils;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.record.MockRecordParser;
+import org.apache.nifi.serialization.record.MockSchemaRegistry;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.StringUtils;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.time.format.DateTimeFormatter;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.TimeZone;
+import java.util.function.Consumer;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class PutElasticsearchRecordTest extends
AbstractPutElasticsearchTest {
+private static final int DATE_YEAR = 2020;
+private static final int DATE_MONTH = 11;
+private static final int DATE_DAY = 27;
+private static final int TIME_HOUR = 12;
+private static final int TIME_MINUTE = 55;
+private static final int TIME_SECOND = 23;
+private static final LocalDateTime LOCAL_DATE_TIME =
LocalDateTime.of(DATE_YEAR, DATE_MONTH, DATE_DAY, TIME_HOUR, TIME_MINUTE,
TIME_SECOND);
+private static final LocalDate LOCAL_DATE = LocalDate.of(DATE_YEAR,
DATE_MONTH, DATE_DAY);
+private static final LocalTime LOCAL_TIME = LocalTime.of(TIME_HOUR,
TIME_MINUTE, TIME_SECOND);
+private static final String TEST_DIR =
"src/test/resources/PutElasticsearchRecordTest";
+private static final String TEST_COMMON_DIR = "src/test/resources/common";
+private static String simpleSchema;
+private MockBulkLoadClientService clientService;
+private MockSchemaRegistry registry;
+private TestRunner runner;
+
+@Override
+public Class getTestProcessor() {
+return PutElasticsearchRecord.class;
+}
+
+@BeforeAll
+public static void setUpBeforeClass() throws Exception {
+simpleSchema = Files.readString(Paths.get(TEST_DIR,
"simpleSchema.json"));
+TimeZone.setDefault(TimeZone.getTimeZone("UTC"));
+}
+
+@BeforeEach
+public void setup() throws Exception {
+clientService = new MockBulkLoadClientService();
+clientService.setResponse(new IndexOperationResponse(1500));
+registry = new MockSchemaRegistry();
+registry.addSchema("simple", AvroTypeUtil.createSchema(new
Schema.Parser().parse(simpleSchema)));
+RecordReaderFactory reader = new JsonTreeReader();
+runner = TestRunners.newTestRunner(getTestProcessor());
+runner.addControllerService("registry", registry);
+runner.addControllerService("reader", reader);
+runner.addControllerService("clientService", clientService);
+runner.setProperty(reader,
[jira] [Updated] (NIFI-11899) Existing nifi_bulletin metric does not reflect updated state once bulletin is resolved
[ https://issues.apache.org/jira/browse/NIFI-11899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-11899: Fix Version/s: 1.latest 2.latest Status: Patch Available (was: In Progress) > Existing nifi_bulletin metric does not reflect updated state once bulletin is > resolved > -- > > Key: NIFI-11899 > URL: https://issues.apache.org/jira/browse/NIFI-11899 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.23.0 >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Minor > Labels: metrics > Fix For: 1.latest, 2.latest > > Time Spent: 10m > Remaining Estimate: 0h > > When bulletins are generated, the existing nifi_bulletin metric will > demonstrate the existence of bulletin by registering the metric with a value > of 1 and labels reflecting the specific information for the bulletin. However > once a bulletin no longer exists or is resolved the value persists in the > metrics registry, giving a false sense of the bulletin being active. The > goal of this fix is to properly update the registry to reflect currently > active/inactive bulletins. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] YolandaMDavis opened a new pull request, #7567: NIFI-11899 - Clear bulletin metrics registry to ensure it is reset with latest bulletins
YolandaMDavis opened a new pull request, #7567: URL: https://github.com/apache/nifi/pull/7567 # Summary [NIFI-11899](https://issues.apache.org/jira/browse/NIFI-11899) The JIRA resolves an issue where the nifi_bulletin metric value was not reset to reflect bulletins that are currently firing. Previously any bulletin that fired was added to the metric registry however the registry was never updated, giving a false sense that the bulletin is still firing. # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [X] Pull Request based on current revision of the `main` branch - [X] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. 1. Create a flow which would generate an error. For example the flow below uses ConsumeTwitter and LogAttribute with ConsumeTwitter bearer token misconfigured purposefully. Do not start the flow. https://github.com/apache/nifi/assets/1371858/b0e00c73-b529-4825-abdd-7d47d1c7ae5d;> 2. Capture current metrics available using the Prometheus metrics api endpoint (e.g. https://127.0.0.1:8443/nifi-api/flow/metrics/prometheus) and search output to confirm that the nifi_bulletin metric does not appear, indicating there are no bulletins present 4. Start flow and confirm that bulletins showing an error/warning is present https://github.com/apache/nifi/assets/1371858/3cd295ea-e92b-4f15-b51d-7c7b0702deb5;> 5. On the metrics endpoint confirm that the nifi_bulletin metrics appear with values corresponding to each component demonstrating an error. With the example flow metrics show a single metric entry with a value of 1 https://github.com/apache/nifi/assets/1371858/210a8d8d-25a9-4505-be1e-ba028331c039;> 6. Stop the flow and await for bulletins to clear from the UI 7. Confirm on metrics endpoint that nifi_bulletin metric no longer appears or no longer contains the specific bulletin that was created ### Build - [X] Build completed using `mvn clean install -P contrib-check` - [X] JDK 17 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (MINIFICPP-2134) improve InvokeHTTP property parsing errors
[ https://issues.apache.org/jira/browse/MINIFICPP-2134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz resolved MINIFICPP-2134. - Resolution: Done > improve InvokeHTTP property parsing errors > -- > > Key: MINIFICPP-2134 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2134 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Affects Versions: 0.15.0 >Reporter: Marton Szasz >Assignee: Ferenc Gerlits >Priority: Major > Fix For: 0.15.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (MINIFICPP-2167) Upgrade Catch2 to newer release
[ https://issues.apache.org/jira/browse/MINIFICPP-2167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz updated MINIFICPP-2167: Fix Version/s: 0.15.0 > Upgrade Catch2 to newer release > --- > > Key: MINIFICPP-2167 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2167 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Martin Zink >Assignee: Martin Zink >Priority: Trivial > Fix For: 0.15.0 > > > Catch2 had a major release and the new release comes with improved compile > times and a lot of quality of life changes. I think it would be beneficial to > upgrade. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (MINIFICPP-2167) Upgrade Catch2 to newer release
[ https://issues.apache.org/jira/browse/MINIFICPP-2167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz updated MINIFICPP-2167: Resolution: Done Status: Resolved (was: Patch Available) > Upgrade Catch2 to newer release > --- > > Key: MINIFICPP-2167 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2167 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Martin Zink >Assignee: Martin Zink >Priority: Trivial > > Catch2 had a major release and the new release comes with improved compile > times and a lot of quality of life changes. I think it would be beneficial to > upgrade. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (MINIFICPP-2163) Running ctest leaves test repository files when finished
[
https://issues.apache.org/jira/browse/MINIFICPP-2163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Szasz updated MINIFICPP-2163:
Fix Version/s: 0.15.0
> Running ctest leaves test repository files when finished
>
>
> Key: MINIFICPP-2163
> URL: https://issues.apache.org/jira/browse/MINIFICPP-2163
> Project: Apache NiFi MiNiFi C++
> Issue Type: Test
>Reporter: Gábor Gyimesi
>Assignee: Gábor Gyimesi
>Priority: Minor
> Fix For: 0.15.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Running ctest does not clean up after the tests are finished leaving flowfile
> and provenance repository files in the git repository:
> {code:java}
> ?? ../libminifi/test/flowfile_repository/
> ?? ../libminifi/test/provenance_repository/ {code}
> The responsible test should be identified and cleanup should be added.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (MINIFICPP-2162) Remove getClassName(), use className() instead
[
https://issues.apache.org/jira/browse/MINIFICPP-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Szasz updated MINIFICPP-2162:
Fix Version/s: 0.15.0
> Remove getClassName(), use className() instead
> --
>
> Key: MINIFICPP-2162
> URL: https://issues.apache.org/jira/browse/MINIFICPP-2162
> Project: Apache NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Ferenc Gerlits
>Assignee: Ferenc Gerlits
>Priority: Minor
> Fix For: 0.15.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> After https://github.com/apache/nifi-minifi-cpp/pull/1589, we have both
> {{getClassName()}} and {{{}className(){}}}. It's enough to have one, so
> remove the former and use the latter instead.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (MINIFICPP-2159) Add heterogeneous lookup to AttributeMap (FlatMap)
[ https://issues.apache.org/jira/browse/MINIFICPP-2159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz updated MINIFICPP-2159: Fix Version/s: 0.15.0 > Add heterogeneous lookup to AttributeMap (FlatMap) > -- > > Key: MINIFICPP-2159 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2159 > Project: Apache NiFi MiNiFi C++ > Issue Type: New Feature >Reporter: Martin Zink >Assignee: Martin Zink >Priority: Minor > Fix For: 0.15.0 > > Time Spent: 3h > Remaining Estimate: 0h > > After MINIFICPP-1825 it would be nice to able to call getAttribute without > unneccesary copies. FlatMap should be able to do lookups by anything equality > comparable, instead of just T (const std::string&). > > We should also change SpecialFlowAttribute static const std::string-s to > constexpr std::string_view-s. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (MINIFICPP-2169) PutS3Object fails on Windows
[
https://issues.apache.org/jira/browse/MINIFICPP-2169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Szasz resolved MINIFICPP-2169.
-
Resolution: Fixed
> PutS3Object fails on Windows
>
>
> Key: MINIFICPP-2169
> URL: https://issues.apache.org/jira/browse/MINIFICPP-2169
> Project: Apache NiFi MiNiFi C++
> Issue Type: Bug
>Reporter: Gábor Gyimesi
>Assignee: Gábor Gyimesi
>Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> S3 upload fails with the following message when using WinHTTP client on
> Windows platform.
> {code:java}
> [2023-07-14 16:07:29.300]
> [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [error]
> [WinHttpSyncHttpClient] Failed to add HTTP request headers:
> amz-sdk-invocation-id: A47F213E-E882-4874-A672-476928DCF98D
> amz-sdk-request: attempt=1
> authorization: AWS4-HMAC-SHA256
> Credential=AKIAUYLDB5SQVHSVDTJS/20230714/eu-west-2/s3/aws4_request,
> SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-md5;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-grant-full-control;x-amz-grant-read;x-amz-grant-read-acp;x-amz-grant-write-acp;x-amz-server-side-encryption;x-amz-storage-class,
> Signature=cb2bb298a825a9c44fe8901f8aecc50cf1301e1c850e20dd1144e73ea313b27a
> content-length: 13
> content-md5: ZajifYh5KDgxtmS9i38K1A==
> content-type: application/octet-stream
> host: .s3.eu-west-2.amazonaws.com
> user-agent: aws-sdk-cpp/1.11.118 Windows/10.0.19041.3155 AMD64 MSVC/1929
> x-amz-content-sha256: UNSIGNED-PAYLOAD
> x-amz-date: 20230714T140729Z
> x-amz-grant-full-control:
> x-amz-grant-read:
> x-amz-grant-read-acp:
> x-amz-grant-write-acp:
> x-amz-server-side-encryption:
> x-amz-storage-class: STANDARD
> , with error code: 12150
> [2023-07-14 16:07:29.458]
> [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [warning]
> [AWSErrorMarshaller] Encountered AWSError 'SignatureDoesNotMatch': The
> request signature we calculated does not match the signature you provided.
> Check your key and signing method.
> [2023-07-14 16:07:29.458]
> [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [error] [AWSXmlClient]
> HTTP response code: 403
> Resolved remote host IP address:
> Request ID: EF8H8BECB3N9KE0A
> Exception name: SignatureDoesNotMatch
> Error message: The request signature we calculated does not match the
> signature you provided. Check your key and signing method.
> 7 response headers:
> connection : close
> content-type : application/xml
> date : Fri, 14 Jul 2023 14:07:27 GMT
> server : AmazonS3
> transfer-encoding : chunked
> x-amz-id-2 :
> 6gApL4IUcv8x1DyoE2jrmips9TvoCzQJsYPywjgYQA/BAb0oUUQ3fP1b6qDCUcxwn349+Wxnii4=
> x-amz-request-id : EF8H8BECB3N9KE0A
> [2023-07-14 16:07:29.458]
> [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [warning] [AWSClient]
> If the signature check failed. This could be because of a time skew.
> Attempting to adjust the signer.
> [2023-07-14 16:07:29.458]
> [org::apache::nifi::minifi::aws::s3::S3RequestSender] [error] PutS3Object
> failed with the following: 'The request signature we calculated does not
> match the signature you provided. Check your key and signing method.' {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (MINIFICPP-2169) PutS3Object fails on Windows
[
https://issues.apache.org/jira/browse/MINIFICPP-2169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Szasz updated MINIFICPP-2169:
Fix Version/s: 0.15.0
> PutS3Object fails on Windows
>
>
> Key: MINIFICPP-2169
> URL: https://issues.apache.org/jira/browse/MINIFICPP-2169
> Project: Apache NiFi MiNiFi C++
> Issue Type: Bug
>Reporter: Gábor Gyimesi
>Assignee: Gábor Gyimesi
>Priority: Major
> Fix For: 0.15.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> S3 upload fails with the following message when using WinHTTP client on
> Windows platform.
> {code:java}
> [2023-07-14 16:07:29.300]
> [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [error]
> [WinHttpSyncHttpClient] Failed to add HTTP request headers:
> amz-sdk-invocation-id: A47F213E-E882-4874-A672-476928DCF98D
> amz-sdk-request: attempt=1
> authorization: AWS4-HMAC-SHA256
> Credential=AKIAUYLDB5SQVHSVDTJS/20230714/eu-west-2/s3/aws4_request,
> SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-md5;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-grant-full-control;x-amz-grant-read;x-amz-grant-read-acp;x-amz-grant-write-acp;x-amz-server-side-encryption;x-amz-storage-class,
> Signature=cb2bb298a825a9c44fe8901f8aecc50cf1301e1c850e20dd1144e73ea313b27a
> content-length: 13
> content-md5: ZajifYh5KDgxtmS9i38K1A==
> content-type: application/octet-stream
> host: .s3.eu-west-2.amazonaws.com
> user-agent: aws-sdk-cpp/1.11.118 Windows/10.0.19041.3155 AMD64 MSVC/1929
> x-amz-content-sha256: UNSIGNED-PAYLOAD
> x-amz-date: 20230714T140729Z
> x-amz-grant-full-control:
> x-amz-grant-read:
> x-amz-grant-read-acp:
> x-amz-grant-write-acp:
> x-amz-server-side-encryption:
> x-amz-storage-class: STANDARD
> , with error code: 12150
> [2023-07-14 16:07:29.458]
> [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [warning]
> [AWSErrorMarshaller] Encountered AWSError 'SignatureDoesNotMatch': The
> request signature we calculated does not match the signature you provided.
> Check your key and signing method.
> [2023-07-14 16:07:29.458]
> [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [error] [AWSXmlClient]
> HTTP response code: 403
> Resolved remote host IP address:
> Request ID: EF8H8BECB3N9KE0A
> Exception name: SignatureDoesNotMatch
> Error message: The request signature we calculated does not match the
> signature you provided. Check your key and signing method.
> 7 response headers:
> connection : close
> content-type : application/xml
> date : Fri, 14 Jul 2023 14:07:27 GMT
> server : AmazonS3
> transfer-encoding : chunked
> x-amz-id-2 :
> 6gApL4IUcv8x1DyoE2jrmips9TvoCzQJsYPywjgYQA/BAb0oUUQ3fP1b6qDCUcxwn349+Wxnii4=
> x-amz-request-id : EF8H8BECB3N9KE0A
> [2023-07-14 16:07:29.458]
> [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [warning] [AWSClient]
> If the signature check failed. This could be because of a time skew.
> Attempting to adjust the signer.
> [2023-07-14 16:07:29.458]
> [org::apache::nifi::minifi::aws::s3::S3RequestSender] [error] PutS3Object
> failed with the following: 'The request signature we calculated does not
> match the signature you provided. Check your key and signing method.' {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (MINIFICPP-2154) Replace SecureSocketGetTCPTest with utils::net::getSSLContext tests
[ https://issues.apache.org/jira/browse/MINIFICPP-2154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz updated MINIFICPP-2154: Resolution: Done Status: Resolved (was: Patch Available) > Replace SecureSocketGetTCPTest with utils::net::getSSLContext tests > --- > > Key: MINIFICPP-2154 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2154 > Project: Apache NiFi MiNiFi C++ > Issue Type: Test >Reporter: Martin Zink >Assignee: Martin Zink >Priority: Minor > Fix For: 0.15.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > SecureSocketGetTCPTest mainly test the various SSLContext service > configurations, but in a way too convoluted way. > After MINIFICPP-2131 we should just test the > minifi::utils::net::getSSLContext part (that way it will test all the > processors that use asio with ssl, not just GetTCP. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (MINIFICPP-2163) Running ctest leaves test repository files when finished
[
https://issues.apache.org/jira/browse/MINIFICPP-2163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Szasz resolved MINIFICPP-2163.
-
Resolution: Done
> Running ctest leaves test repository files when finished
>
>
> Key: MINIFICPP-2163
> URL: https://issues.apache.org/jira/browse/MINIFICPP-2163
> Project: Apache NiFi MiNiFi C++
> Issue Type: Test
>Reporter: Gábor Gyimesi
>Assignee: Gábor Gyimesi
>Priority: Minor
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Running ctest does not clean up after the tests are finished leaving flowfile
> and provenance repository files in the git repository:
> {code:java}
> ?? ../libminifi/test/flowfile_repository/
> ?? ../libminifi/test/provenance_repository/ {code}
> The responsible test should be identified and cleanup should be added.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (MINIFICPP-2162) Remove getClassName(), use className() instead
[
https://issues.apache.org/jira/browse/MINIFICPP-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Szasz resolved MINIFICPP-2162.
-
Resolution: Done
> Remove getClassName(), use className() instead
> --
>
> Key: MINIFICPP-2162
> URL: https://issues.apache.org/jira/browse/MINIFICPP-2162
> Project: Apache NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Ferenc Gerlits
>Assignee: Ferenc Gerlits
>Priority: Minor
> Time Spent: 20m
> Remaining Estimate: 0h
>
> After https://github.com/apache/nifi-minifi-cpp/pull/1589, we have both
> {{getClassName()}} and {{{}className(){}}}. It's enough to have one, so
> remove the former and use the latter instead.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (MINIFICPP-2159) Add heterogeneous lookup to AttributeMap (FlatMap)
[ https://issues.apache.org/jira/browse/MINIFICPP-2159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz updated MINIFICPP-2159: Resolution: Done Status: Resolved (was: Patch Available) > Add heterogeneous lookup to AttributeMap (FlatMap) > -- > > Key: MINIFICPP-2159 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2159 > Project: Apache NiFi MiNiFi C++ > Issue Type: New Feature >Reporter: Martin Zink >Assignee: Martin Zink >Priority: Minor > Time Spent: 3h > Remaining Estimate: 0h > > After MINIFICPP-1825 it would be nice to able to call getAttribute without > unneccesary copies. FlatMap should be able to do lookups by anything equality > comparable, instead of just T (const std::string&). > > We should also change SpecialFlowAttribute static const std::string-s to > constexpr std::string_view-s. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi-minifi-cpp] szaszm closed pull request #1615: Update Catch2 from 2.13.10 to 3.4.0
szaszm closed pull request #1615: Update Catch2 from 2.13.10 to 3.4.0 URL: https://github.com/apache/nifi-minifi-cpp/pull/1615 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm closed pull request #1612: MINIFICPP-2159 Add heterogeneous lookup to FlatMap
szaszm closed pull request #1612: MINIFICPP-2159 Add heterogeneous lookup to FlatMap URL: https://github.com/apache/nifi-minifi-cpp/pull/1612 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm closed pull request #1608: MINIFICPP-2163 Cleanup LogMetricsPublisherTests
szaszm closed pull request #1608: MINIFICPP-2163 Cleanup LogMetricsPublisherTests URL: https://github.com/apache/nifi-minifi-cpp/pull/1608 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm closed pull request #1610: MINIFICPP-2134 Do not swallow parsing exceptions
szaszm closed pull request #1610: MINIFICPP-2134 Do not swallow parsing exceptions URL: https://github.com/apache/nifi-minifi-cpp/pull/1610 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm closed pull request #1609: MINIFICPP-1817 Use magic_enum library instead of SMART_ENUM
szaszm closed pull request #1609: MINIFICPP-1817 Use magic_enum library instead of SMART_ENUM URL: https://github.com/apache/nifi-minifi-cpp/pull/1609 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm closed pull request #1611: MINIFICPP-2162 Use className() instead of getClassName()
szaszm closed pull request #1611: MINIFICPP-2162 Use className() instead of getClassName() URL: https://github.com/apache/nifi-minifi-cpp/pull/1611 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm closed pull request #1598: MINIFICPP-2154 Replace SecureSocketGetTCPTest with utils::net::getSSLContext tests
szaszm closed pull request #1598: MINIFICPP-2154 Replace SecureSocketGetTCPTest with utils::net::getSSLContext tests URL: https://github.com/apache/nifi-minifi-cpp/pull/1598 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-11903) TestZookeeperStateServer does not need to run on windows
[ https://issues.apache.org/jira/browse/NIFI-11903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Witt updated NIFI-11903: Fix Version/s: 1.latest 2.latest > TestZookeeperStateServer does not need to run on windows > > > Key: NIFI-11903 > URL: https://issues.apache.org/jira/browse/NIFI-11903 > Project: Apache NiFi > Issue Type: Test >Reporter: Joe Witt >Assignee: Joe Witt >Priority: Major > Fix For: 1.latest, 2.latest > > Time Spent: 10m > Remaining Estimate: 0h > > On a windows machine I constantly see failures from the > testzookeeperstateserver class. This class does not need to be tested on > windows. We dont support clusters on windows. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11903) TestZookeeperStateServer does not need to run on windows
[ https://issues.apache.org/jira/browse/NIFI-11903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17750496#comment-17750496 ] ASF subversion and git services commented on NIFI-11903: Commit 490810452c092b0121c3ac3c5f3e54bbc1324103 in nifi's branch refs/heads/NIFI-11903 from Joe Witt [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=490810452c ] NIFI-11903 disable if windows > TestZookeeperStateServer does not need to run on windows > > > Key: NIFI-11903 > URL: https://issues.apache.org/jira/browse/NIFI-11903 > Project: Apache NiFi > Issue Type: Test >Reporter: Joe Witt >Assignee: Joe Witt >Priority: Major > > On a windows machine I constantly see failures from the > testzookeeperstateserver class. This class does not need to be tested on > windows. We dont support clusters on windows. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] joewitt opened a new pull request, #7566: NIFI-11903 disable if windows
joewitt opened a new pull request, #7566: URL: https://github.com/apache/nifi/pull/7566 # Summary [NIFI-0](https://issues.apache.org/jira/browse/NIFI-0) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [ ] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [ ] Pull Request based on current revision of the `main` branch - [ ] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 17 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] dan-s1 commented on a diff in pull request #7537: NIFI-11778 Initial check in of refactored Groovy tests in nifi-elasticsearch-restapi-processors to Java (and JUnit 5)
dan-s1 commented on code in PR #7537:
URL: https://github.com/apache/nifi/pull/7537#discussion_r1282485937
##
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/test/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchRecordTest.java:
##
@@ -0,0 +1,784 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License") you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.avro.Schema;
+import org.apache.nifi.avro.AvroTypeUtil;
+import org.apache.nifi.elasticsearch.IndexOperationRequest;
+import org.apache.nifi.elasticsearch.IndexOperationResponse;
+import org.apache.nifi.json.JsonRecordSetWriter;
+import org.apache.nifi.json.JsonTreeReader;
+import org.apache.nifi.processors.elasticsearch.mock.MockBulkLoadClientService;
+import org.apache.nifi.provenance.ProvenanceEventType;
+import org.apache.nifi.schema.access.SchemaAccessUtils;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.record.MockRecordParser;
+import org.apache.nifi.serialization.record.MockSchemaRegistry;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.StringUtils;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.time.format.DateTimeFormatter;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.TimeZone;
+import java.util.function.Consumer;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class PutElasticsearchRecordTest extends
AbstractPutElasticsearchTest {
+private static final int DATE_YEAR = 2020;
+private static final int DATE_MONTH = 11;
+private static final int DATE_DAY = 27;
+private static final int TIME_HOUR = 12;
+private static final int TIME_MINUTE = 55;
+private static final int TIME_SECOND = 23;
+private static final LocalDateTime LOCAL_DATE_TIME =
LocalDateTime.of(DATE_YEAR, DATE_MONTH, DATE_DAY, TIME_HOUR, TIME_MINUTE,
TIME_SECOND);
+private static final LocalDate LOCAL_DATE = LocalDate.of(DATE_YEAR,
DATE_MONTH, DATE_DAY);
+private static final LocalTime LOCAL_TIME = LocalTime.of(TIME_HOUR,
TIME_MINUTE, TIME_SECOND);
+private static final String TEST_DIR =
"src/test/resources/PutElasticsearchRecordTest";
+private static final String TEST_COMMON_DIR = "src/test/resources/common";
+private static String simpleSchema;
+private MockBulkLoadClientService clientService;
+private MockSchemaRegistry registry;
+private TestRunner runner;
+
+@Override
+public Class getTestProcessor() {
+return PutElasticsearchRecord.class;
+}
+
+@BeforeAll
+public static void setUpBeforeClass() throws Exception {
+simpleSchema = Files.readString(Paths.get(TEST_DIR,
"simpleSchema.json"));
+TimeZone.setDefault(TimeZone.getTimeZone("UTC"));
+}
+
+@BeforeEach
+public void setup() throws Exception {
+clientService = new MockBulkLoadClientService();
+clientService.setResponse(new IndexOperationResponse(1500));
+registry = new MockSchemaRegistry();
+registry.addSchema("simple", AvroTypeUtil.createSchema(new
Schema.Parser().parse(simpleSchema)));
+RecordReaderFactory reader = new JsonTreeReader();
+runner = TestRunners.newTestRunner(getTestProcessor());
+runner.addControllerService("registry", registry);
+runner.addControllerService("reader", reader);
+runner.addControllerService("clientService", clientService);
+runner.setProperty(reader,
[GitHub] [nifi] dan-s1 commented on a diff in pull request #7537: NIFI-11778 Initial check in of refactored Groovy tests in nifi-elasticsearch-restapi-processors to Java (and JUnit 5)
dan-s1 commented on code in PR #7537:
URL: https://github.com/apache/nifi/pull/7537#discussion_r1282485937
##
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/test/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchRecordTest.java:
##
@@ -0,0 +1,784 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License") you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.avro.Schema;
+import org.apache.nifi.avro.AvroTypeUtil;
+import org.apache.nifi.elasticsearch.IndexOperationRequest;
+import org.apache.nifi.elasticsearch.IndexOperationResponse;
+import org.apache.nifi.json.JsonRecordSetWriter;
+import org.apache.nifi.json.JsonTreeReader;
+import org.apache.nifi.processors.elasticsearch.mock.MockBulkLoadClientService;
+import org.apache.nifi.provenance.ProvenanceEventType;
+import org.apache.nifi.schema.access.SchemaAccessUtils;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.record.MockRecordParser;
+import org.apache.nifi.serialization.record.MockSchemaRegistry;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.StringUtils;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.time.format.DateTimeFormatter;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.TimeZone;
+import java.util.function.Consumer;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class PutElasticsearchRecordTest extends
AbstractPutElasticsearchTest {
+private static final int DATE_YEAR = 2020;
+private static final int DATE_MONTH = 11;
+private static final int DATE_DAY = 27;
+private static final int TIME_HOUR = 12;
+private static final int TIME_MINUTE = 55;
+private static final int TIME_SECOND = 23;
+private static final LocalDateTime LOCAL_DATE_TIME =
LocalDateTime.of(DATE_YEAR, DATE_MONTH, DATE_DAY, TIME_HOUR, TIME_MINUTE,
TIME_SECOND);
+private static final LocalDate LOCAL_DATE = LocalDate.of(DATE_YEAR,
DATE_MONTH, DATE_DAY);
+private static final LocalTime LOCAL_TIME = LocalTime.of(TIME_HOUR,
TIME_MINUTE, TIME_SECOND);
+private static final String TEST_DIR =
"src/test/resources/PutElasticsearchRecordTest";
+private static final String TEST_COMMON_DIR = "src/test/resources/common";
+private static String simpleSchema;
+private MockBulkLoadClientService clientService;
+private MockSchemaRegistry registry;
+private TestRunner runner;
+
+@Override
+public Class getTestProcessor() {
+return PutElasticsearchRecord.class;
+}
+
+@BeforeAll
+public static void setUpBeforeClass() throws Exception {
+simpleSchema = Files.readString(Paths.get(TEST_DIR,
"simpleSchema.json"));
+TimeZone.setDefault(TimeZone.getTimeZone("UTC"));
+}
+
+@BeforeEach
+public void setup() throws Exception {
+clientService = new MockBulkLoadClientService();
+clientService.setResponse(new IndexOperationResponse(1500));
+registry = new MockSchemaRegistry();
+registry.addSchema("simple", AvroTypeUtil.createSchema(new
Schema.Parser().parse(simpleSchema)));
+RecordReaderFactory reader = new JsonTreeReader();
+runner = TestRunners.newTestRunner(getTestProcessor());
+runner.addControllerService("registry", registry);
+runner.addControllerService("reader", reader);
+runner.addControllerService("clientService", clientService);
+runner.setProperty(reader,
[GitHub] [nifi] turcsanyip commented on a diff in pull request #7489: NIFI-11823 - fix NUMERIC support in PutBigQuery
turcsanyip commented on code in PR #7489:
URL: https://github.com/apache/nifi/pull/7489#discussion_r1282483071
##
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/bigquery/proto/ProtoUtils.java:
##
@@ -17,50 +17,101 @@
package org.apache.nifi.processors.gcp.bigquery.proto;
+import com.google.cloud.bigquery.storage.v1.BigDecimalByteStringEncoder;
+import com.google.cloud.bigquery.storage.v1.TableFieldSchema.Type;
+import com.google.cloud.bigquery.storage.v1.TableSchema;
import com.google.protobuf.Descriptors;
+import com.google.protobuf.DoubleValue;
import com.google.protobuf.DynamicMessage;
+import com.google.protobuf.FloatValue;
+import com.google.protobuf.Int64Value;
+
+import java.math.BigDecimal;
import java.util.Arrays;
import java.util.Collection;
import java.util.Map;
/**
- * Util class for protocol buffer messaging
- */
+* Util class for protocol buffer messaging
+*/
public class ProtoUtils {
-public static DynamicMessage createMessage(Descriptors.Descriptor
descriptor, Map valueMap) {
-DynamicMessage.Builder builder = DynamicMessage.newBuilder(descriptor);
+ public static DynamicMessage createMessage(Descriptors.Descriptor
descriptor, Map valueMap, TableSchema tableSchema) {
+ final DynamicMessage.Builder builder =
DynamicMessage.newBuilder(descriptor);
+
+ for (final Descriptors.FieldDescriptor field : descriptor.getFields()) {
+ final String name = field.getName();
+ Object value = valueMap.get(name);
+ if (value == null) {
+ continue;
+ }
+
+ switch (field.getType()) {
+ case MESSAGE:
+ if (field.isRepeated()) {
+ Collection collection = value.getClass().isArray() ?
Arrays.asList((Object[]) value) : (Collection) value;
+ collection.forEach(act -> builder.addRepeatedField(field,
createMessage(field.getMessageType(), (Map) act, tableSchema)));
+ } else {
+ builder.setField(field,
createMessage(field.getMessageType(), (Map) value,
tableSchema));
+ }
+ break;
+
+ // INT64 with alias INT, SMALLINT, INTEGER, BIGINT, TINYINT, BYTEINT
+ case INT64:
+ // Integer in the bigquery table schema maps back to INT64
which is considered to be Long on Java side:
+ // https://developers.google.com/protocol-buffers/docs/proto3
+ if (value instanceof Integer) {
+ value = Long.valueOf((Integer) value);
+ }
+
+ setField(value, field, builder);
+ break;
+
+ // FLOAT64
+ case DOUBLE:
+ if (value instanceof Float) {
+ value = ((Float) value).doubleValue();
Review Comment:
`Float.doubleValue()` can lead to weird numbers due to different precisions
of `float` and `double`.
E.g. `21.34` in `float` would be converted to `21.3415258789` in
`double`.
I suggest using `Double.valueOf(value.toString())` instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (NIFI-11903) TestZookeeperStateServer does not need to run on windows
Joe Witt created NIFI-11903: --- Summary: TestZookeeperStateServer does not need to run on windows Key: NIFI-11903 URL: https://issues.apache.org/jira/browse/NIFI-11903 Project: Apache NiFi Issue Type: Test Reporter: Joe Witt Assignee: Joe Witt On a windows machine I constantly see failures from the testzookeeperstateserver class. This class does not need to be tested on windows. We dont support clusters on windows. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] turcsanyip commented on pull request #7489: NIFI-11823 - fix NUMERIC support in PutBigQuery
turcsanyip commented on PR #7489:
URL: https://github.com/apache/nifi/pull/7489#issuecomment-1663063462
Thanks @pvillard31 for your detailed explanation!
I did not check the result in the BigQuery table, just saw that `long` was
successful, and that's why I thought `int` should work too. Now I understand
that none of `int`, `long`, `float` or `double` can be inserted into a
`NUMERIC` field with the current code because it creates a different binary
representation.
What we can do (if we want to support cross-type conversions, e.g. `int` to
`numeric`) is to convert the Java `int`, `long`, etc. values into `BigDecimal`
first, and then use `BigDecimalByteStringEncoder` (as for the values originally
in `BigDecimal`).
Something like this:
```
case BYTES:
if (value instanceof Integer) {
value = new BigDecimal((int) value);
} else if (value instanceof Long) {
value = new BigDecimal((long) value);
} else if (value instanceof Float || value instanceof
Double) {
value = new BigDecimal(value.toString());
}
if (value instanceof BigDecimal) {
if
(tableSchema.getFields(field.getIndex()).getType().equals(Type.BIGNUMERIC)) {
value =
BigDecimalByteStringEncoder.encodeToBigNumericByteString((BigDecimal) value);
} else if
(tableSchema.getFields(field.getIndex()).getType().equals(Type.NUMERIC)) {
value =
BigDecimalByteStringEncoder.encodeToNumericByteString((BigDecimal) value);
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] joewitt commented on pull request #7564: NIFI-11896 Correct QuestDB Status Repository Shutdown handling
joewitt commented on PR #7564: URL: https://github.com/apache/nifi/pull/7564#issuecomment-1663054829 Two full clean builds with all the trimmings worked perfectly/. And no dumpstream. Nice! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] Lehel44 opened a new pull request, #7565: NIFI-11902: Fix ListHDFS closes FileSystem in first run
Lehel44 opened a new pull request, #7565: URL: https://github.com/apache/nifi/pull/7565 # Summary [NIFI-11902](https://issues.apache.org/jira/browse/NIFI-11902) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [ ] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [ ] Pull Request based on current revision of the `main` branch - [ ] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 17 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-11902) ListHDFS closes FileSystem in OnTrigger
Lehel Boér created NIFI-11902: - Summary: ListHDFS closes FileSystem in OnTrigger Key: NIFI-11902 URL: https://issues.apache.org/jira/browse/NIFI-11902 Project: Apache NiFi Issue Type: Bug Reporter: Lehel Boér Assignee: Lehel Boér Currently the Hadoop FileSystem::getFileSystem is called in a try-with-resources block. This is unnecessary because the AbstractHadoopProcessor::abstractOnStopped closes the FileSystem when the processor is stopped. Hence the try-with-resources block should be removed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #7561: NIFI-11898: Handle auto-commit and commit based on driver capabilities in PutDatabaseRecord
exceptionfactory commented on code in PR #7561:
URL: https://github.com/apache/nifi/pull/7561#discussion_r1282457678
##
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutDatabaseRecord.java:
##
@@ -471,10 +472,19 @@ public void onTrigger(final ProcessContext context, final
ProcessSession session
connectionHolder = Optional.of(connection);
originalAutoCommit = connection.getAutoCommit();
-connection.setAutoCommit(false);
+if (originalAutoCommit) {
+try {
+connection.setAutoCommit(false);
+} catch (SQLFeatureNotSupportedException sfnse) {
+getLogger().debug("setAutoCommit(false) not supported by
this driver");
+}
+}
putToDatabase(context, session, flowFile, connection);
-connection.commit();
+// Only commit the connection if auto-commit is false
+if (!connection.getAutoCommit()) {
Review Comment:
Should this be changed to check `originalAutoCommit` instead of calling
`connection.getAutoCommit()` again?
```suggestion
if (!originalAutoCommit) {
```
##
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/PutDatabaseRecordTest.java:
##
@@ -1949,4 +1989,29 @@ SqlAndIncludedColumns generateInsert(RecordSchema
recordSchema, String tableName
return new SqlAndIncludedColumns("INSERT INTO PERSONS VALUES
(?,?,?,?)", Arrays.asList(0, 1, 2, 3));
}
}
+
+static class DBCPServiceAutoCommitTest extends AbstractControllerService
implements DBCPService {
+private final String databaseLocation;
+
+public DBCPServiceAutoCommitTest(final String databaseLocation) {
+this.databaseLocation = databaseLocation;
+}
+
+@Override
+public String getIdentifier() {
+return "dbcp";
Review Comment:
Recommend setting a static final variable named `DBCP_SERVICE_ID` and
reusing across this test class.
##
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/PutDatabaseRecordTest.java:
##
@@ -1949,4 +1989,29 @@ SqlAndIncludedColumns generateInsert(RecordSchema
recordSchema, String tableName
return new SqlAndIncludedColumns("INSERT INTO PERSONS VALUES
(?,?,?,?)", Arrays.asList(0, 1, 2, 3));
}
}
+
+static class DBCPServiceAutoCommitTest extends AbstractControllerService
implements DBCPService {
+private final String databaseLocation;
+
+public DBCPServiceAutoCommitTest(final String databaseLocation) {
+this.databaseLocation = databaseLocation;
+}
+
+@Override
+public String getIdentifier() {
+return "dbcp";
+}
+
+@Override
+public Connection getConnection() throws ProcessException {
+try {
+Class.forName("org.apache.derby.jdbc.EmbeddedDriver");
Review Comment:
Is this `Class.forName()` call required?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (NIFI-11896) QuestDB Status Repository fails to close resources on shutdown
[ https://issues.apache.org/jira/browse/NIFI-11896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-11896: Status: Patch Available (was: In Progress) > QuestDB Status Repository fails to close resources on shutdown > -- > > Key: NIFI-11896 > URL: https://issues.apache.org/jira/browse/NIFI-11896 > Project: Apache NiFi > Issue Type: Bug > Environment: Fedora 38 >Reporter: Joe Witt >Assignee: David Handermann >Priority: Major > Fix For: 1.latest, 2.latest > > Time Spent: 10m > Remaining Estimate: 0h > > io.questdb.cairo.CairoException: [-1] 'connectionStatus' is left behind on > pool shutdown > ... > at > org.apache.nifi.controller.status.history.questdb.QuestDbContext.close(QuestDbContext.java:48) > at ... EmbeddedQuestDbStatusHistoryRepository.shutdown( line 168) > at ... AbstractEmbeddedQuestDbStatusHistoryRepositoryTest.tearDown(line 52) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-11897) Nifi Framework Core tests for QuestDB create dumpstream output
[
https://issues.apache.org/jira/browse/NIFI-11897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann reassigned NIFI-11897:
---
Assignee: David Handermann
> Nifi Framework Core tests for QuestDB create dumpstream output
> --
>
> Key: NIFI-11897
> URL: https://issues.apache.org/jira/browse/NIFI-11897
> Project: Apache NiFi
> Issue Type: Test
> Environment: Maven home: /development/tools/apache-maven-3.9.1
> Java version: 17.0.7, vendor: Azul Systems, Inc., runtime:
> /usr/lib/jvm/zulu17.42.21-ca-crac-jdk17.0.7-linux_x64
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "6.4.6-200.fc38.x86_64", arch: "amd64", family:
> "unix"
>Reporter: Joe Witt
>Assignee: David Handermann
>Priority: Major
>
> Running the full build fails in nifi-framework-core due to issue filed as
> NIFI-11896. But also the following shows up in the target/surefire-reports
> folder suggesting more problematic aspects at play.
> nifi-framework-core/target/surefire-reports/2023-08-01T09-11-29_729-jvmRun5.dumpstream
> contains
> {noformat}
> # Created at 2023-08-01T09:17:48.313
> Corrupted channel by directly writing to native stream in forked JVM 5.
> Stream '2023-08-01T16:17:48.220325Z I i.q.c.t.t.InputFormatConfiguration
> loading input format config [resource=/text_loader.json]'.
> # Created at 2023-08-01T09:17:48.328
> Corrupted channel by directly writing to native stream in forked JVM 5.
> Stream '2023-08-01T16:17:48.312653Z I i.q.c.t.t.InputFormatConfiguration
> loading input format config [resource=/text_loader.json]'.
> # Created at 2023-08-01T09:17:48.797
> Corrupted channel by directly writing to native stream in forked JVM 5.
> Stream '2023-08-01T16:17:48.339926Z I i.q.g.FunctionFactoryCache loading
> functions [test=true]'.
> # Created at 2023-08-01T09:17:48.799
> Corrupted channel by directly writing to native stream in forked JVM 5.
> Stream '2023-08-01T16:17:48.796990Z I i.q.c.m.EngineMigration upgrading
> database [version=427]'.
> # Created at 2023-08-01T09:17:48.808
> Corrupted channel by directly writing to native stream in forked JVM 5.
> Stream '2023-08-01T16:17:48.799286Z I i.q.c.m.EngineMigration upgraded tables
> to 427'.
> # Created at 2023-08-01T09:17:49.029
> Corrupted channel by directly writing to native stream in forked JVM 5.
> Stream '2023-08-01T16:17:48.807919Z I i.q.c.TableNameRegistryFileStore
> reloading tables file [path=/tmp/junit2537278743248275952/ta
> bles.d.0]'.
> # Created at 2023-08-01T09:17:49.031
> Corrupted channel by directly writing to native stream in forked JVM 5.
> Stream '2023-08-01T16:17:49.028196Z I i.q.c.CairoEngine locked
> [table=`garbageCollectionStatus`, thread=1]'.
> # Created at 2023-08-01T09:17:49.050
> Corrupted channel by directly writing to native stream in forked JVM 5.
> Stream '2023-08-01T16:17:49.031580Z I i.q.c.p.WriterPool created
> [table=`garbageCollectionStatus`, thread=1]'.
> # Created at 2023-08-01T09:17:49.064
> Corrupted channel by directly writing to native stream in forked JVM 5.
> Stream '2023-08-01T16:17:49.049868Z I i.q.c.TableWriter open
> 'garbageCollectionStatus''.
> # Created at 2023-08-01T09:17:49.072
> Corrupted channel by directly writing to native stream in forked JVM 5.
> Stream '2023-08-01T16:17:49.064028Z I i.q.c.CairoEngine unlocked
> [table=`garbageCollectionStatus`]'.
> {noformat}
> that goes on and on for many many lines
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [nifi] exceptionfactory opened a new pull request, #7564: NIFI-11896 Correct QuestDB Status Repository Shutdown handling
exceptionfactory opened a new pull request, #7564: URL: https://github.com/apache/nifi/pull/7564 # Summary [NIFI-11896](https://issues.apache.org/jira/browse/NIFI-11896) Corrects QuestDB Status Repository shutdown handling at runtime and during unit tests. Changes include updating the repository start and shutdown methods to track scheduled future tasks and attempt cancellation before shutting down the executor service. This approach provides a stronger guarantee of stopping scheduled tasks than shutting down the executor service itself. Additional changes include setting the initial delay for scheduled tasks from 0 to the configured interval, avoiding initial execution when starting the repository. This approach avoids unnecessary task execution in unit tests. Test changes include setting the `out` System property pointing to the QuestDB logging configuration, avoiding unnecessary debug logs during unit testing, which can corrupt the console output stream, described in [NIFI-11897](https://issues.apache.org/jira/browse/NIFI-11897). # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [X] Pull Request based on current revision of the `main` branch - [X] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 17 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-11896) QuestDB Status Repository fails to close resources on shutdown
[ https://issues.apache.org/jira/browse/NIFI-11896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-11896: Fix Version/s: 1.latest 2.latest > QuestDB Status Repository fails to close resources on shutdown > -- > > Key: NIFI-11896 > URL: https://issues.apache.org/jira/browse/NIFI-11896 > Project: Apache NiFi > Issue Type: Bug > Environment: Fedora 38 >Reporter: Joe Witt >Assignee: David Handermann >Priority: Major > Fix For: 1.latest, 2.latest > > > io.questdb.cairo.CairoException: [-1] 'connectionStatus' is left behind on > pool shutdown > ... > at > org.apache.nifi.controller.status.history.questdb.QuestDbContext.close(QuestDbContext.java:48) > at ... EmbeddedQuestDbStatusHistoryRepository.shutdown( line 168) > at ... AbstractEmbeddedQuestDbStatusHistoryRepositoryTest.tearDown(line 52) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11896) QuestDB Status Repository fails to close resources on shutdown
[ https://issues.apache.org/jira/browse/NIFI-11896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-11896: Issue Type: Bug (was: Test) > QuestDB Status Repository fails to close resources on shutdown > -- > > Key: NIFI-11896 > URL: https://issues.apache.org/jira/browse/NIFI-11896 > Project: Apache NiFi > Issue Type: Bug > Environment: Fedora 38 >Reporter: Joe Witt >Assignee: David Handermann >Priority: Major > > io.questdb.cairo.CairoException: [-1] 'connectionStatus' is left behind on > pool shutdown > ... > at > org.apache.nifi.controller.status.history.questdb.QuestDbContext.close(QuestDbContext.java:48) > at ... EmbeddedQuestDbStatusHistoryRepository.shutdown( line 168) > at ... AbstractEmbeddedQuestDbStatusHistoryRepositoryTest.tearDown(line 52) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11896) QuestDB Status Repository fails to close resources on shutdown
[ https://issues.apache.org/jira/browse/NIFI-11896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-11896: Summary: QuestDB Status Repository fails to close resources on shutdown (was: EmbeddedQuestDbStatusHistoryRepositoryForNodeTest testReadingEmptyRepository breaks on linux) > QuestDB Status Repository fails to close resources on shutdown > -- > > Key: NIFI-11896 > URL: https://issues.apache.org/jira/browse/NIFI-11896 > Project: Apache NiFi > Issue Type: Test > Environment: Fedora 38 >Reporter: Joe Witt >Priority: Major > > io.questdb.cairo.CairoException: [-1] 'connectionStatus' is left behind on > pool shutdown > ... > at > org.apache.nifi.controller.status.history.questdb.QuestDbContext.close(QuestDbContext.java:48) > at ... EmbeddedQuestDbStatusHistoryRepository.shutdown( line 168) > at ... AbstractEmbeddedQuestDbStatusHistoryRepositoryTest.tearDown(line 52) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-11896) QuestDB Status Repository fails to close resources on shutdown
[ https://issues.apache.org/jira/browse/NIFI-11896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann reassigned NIFI-11896: --- Assignee: David Handermann > QuestDB Status Repository fails to close resources on shutdown > -- > > Key: NIFI-11896 > URL: https://issues.apache.org/jira/browse/NIFI-11896 > Project: Apache NiFi > Issue Type: Test > Environment: Fedora 38 >Reporter: Joe Witt >Assignee: David Handermann >Priority: Major > > io.questdb.cairo.CairoException: [-1] 'connectionStatus' is left behind on > pool shutdown > ... > at > org.apache.nifi.controller.status.history.questdb.QuestDbContext.close(QuestDbContext.java:48) > at ... EmbeddedQuestDbStatusHistoryRepository.shutdown( line 168) > at ... AbstractEmbeddedQuestDbStatusHistoryRepositoryTest.tearDown(line 52) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] pvillard31 commented on pull request #7489: NIFI-11823 - fix NUMERIC support in PutBigQuery
pvillard31 commented on PR #7489: URL: https://github.com/apache/nifi/pull/7489#issuecomment-1662974374 > I can also see regression in the `testStreamingNoErrorWithDateFormat` IT test. Could you please check it? For me it constantly succeeds on main but fails on the PR branch. Good catch, I guess I focused on the tests I added and forgot to re-run this one... after investigation it feels like trying to send a ``long`` into a NUMERIC column is not possible with the current code. We may need to convert int/long into a decimal type before switching to the ByteString. For now, I just fixed the test by switching this specific column to INT64 instead of NUMERIC but curious to hear your thoughts about this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] dan-s1 commented on a diff in pull request #7537: NIFI-11778 Initial check in of refactored Groovy tests in nifi-elasticsearch-restapi-processors to Java (and JUnit 5)
dan-s1 commented on code in PR #7537:
URL: https://github.com/apache/nifi/pull/7537#discussion_r1282425054
##
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/test/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchRecordTest.java:
##
@@ -0,0 +1,784 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License") you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.avro.Schema;
+import org.apache.nifi.avro.AvroTypeUtil;
+import org.apache.nifi.elasticsearch.IndexOperationRequest;
+import org.apache.nifi.elasticsearch.IndexOperationResponse;
+import org.apache.nifi.json.JsonRecordSetWriter;
+import org.apache.nifi.json.JsonTreeReader;
+import org.apache.nifi.processors.elasticsearch.mock.MockBulkLoadClientService;
+import org.apache.nifi.provenance.ProvenanceEventType;
+import org.apache.nifi.schema.access.SchemaAccessUtils;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.record.MockRecordParser;
+import org.apache.nifi.serialization.record.MockSchemaRegistry;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.StringUtils;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.time.format.DateTimeFormatter;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.TimeZone;
+import java.util.function.Consumer;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class PutElasticsearchRecordTest extends
AbstractPutElasticsearchTest {
+private static final int DATE_YEAR = 2020;
+private static final int DATE_MONTH = 11;
+private static final int DATE_DAY = 27;
+private static final int TIME_HOUR = 12;
+private static final int TIME_MINUTE = 55;
+private static final int TIME_SECOND = 23;
+private static final LocalDateTime LOCAL_DATE_TIME =
LocalDateTime.of(DATE_YEAR, DATE_MONTH, DATE_DAY, TIME_HOUR, TIME_MINUTE,
TIME_SECOND);
+private static final LocalDate LOCAL_DATE = LocalDate.of(DATE_YEAR,
DATE_MONTH, DATE_DAY);
+private static final LocalTime LOCAL_TIME = LocalTime.of(TIME_HOUR,
TIME_MINUTE, TIME_SECOND);
+private static final String TEST_DIR =
"src/test/resources/PutElasticsearchRecordTest";
+private static final String TEST_COMMON_DIR = "src/test/resources/common";
+private static String simpleSchema;
+private MockBulkLoadClientService clientService;
+private MockSchemaRegistry registry;
+private TestRunner runner;
+
+@Override
+public Class getTestProcessor() {
+return PutElasticsearchRecord.class;
+}
+
+@BeforeAll
+public static void setUpBeforeClass() throws Exception {
+simpleSchema = Files.readString(Paths.get(TEST_DIR,
"simpleSchema.json"));
+TimeZone.setDefault(TimeZone.getTimeZone("UTC"));
Review Comment:
@ChrisSamo632 @exceptionfactory This information helped a lot. I tweeked my
code to modify the read in JSON files with the same values you had used with
`Time.valueOf(LOCAL_TIME).getTime()`
`Timestamp.valueOf(LOCAL_DATE_TIME).toInstant().toEpochMilli())`
`Date.valueOf(LOCAL_DATE).getTime())`
Now even with changing the timezone with
`TimeZone.setDefault(TimeZone.getTimeZone("Europe/Paris"));`
the tests pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service,
[GitHub] [nifi] pvillard31 commented on a diff in pull request #7489: NIFI-11823 - fix NUMERIC support in PutBigQuery
pvillard31 commented on code in PR #7489:
URL: https://github.com/apache/nifi/pull/7489#discussion_r1282394555
##
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/bigquery/proto/ProtoUtils.java:
##
@@ -17,50 +17,99 @@
package org.apache.nifi.processors.gcp.bigquery.proto;
+import com.google.cloud.bigquery.storage.v1.BigDecimalByteStringEncoder;
+import com.google.cloud.bigquery.storage.v1.TableFieldSchema.Type;
+import com.google.cloud.bigquery.storage.v1.TableSchema;
import com.google.protobuf.Descriptors;
+import com.google.protobuf.DoubleValue;
import com.google.protobuf.DynamicMessage;
+import com.google.protobuf.FloatValue;
+import com.google.protobuf.Int64Value;
+
+import java.math.BigDecimal;
import java.util.Arrays;
import java.util.Collection;
import java.util.Map;
/**
- * Util class for protocol buffer messaging
- */
+* Util class for protocol buffer messaging
+*/
public class ProtoUtils {
-public static DynamicMessage createMessage(Descriptors.Descriptor
descriptor, Map valueMap) {
-DynamicMessage.Builder builder = DynamicMessage.newBuilder(descriptor);
+ public static DynamicMessage createMessage(Descriptors.Descriptor
descriptor, Map valueMap, TableSchema tableSchema) {
+ final DynamicMessage.Builder builder =
DynamicMessage.newBuilder(descriptor);
+
+ for (final Descriptors.FieldDescriptor field : descriptor.getFields()) {
+ final String name = field.getName();
+ Object value = valueMap.get(name);
+ if (value == null) {
+ continue;
+ }
+
+ switch (field.getType()) {
+ case MESSAGE:
+ if (field.isRepeated()) {
+ Collection collection = value.getClass().isArray() ?
Arrays.asList((Object[]) value) : (Collection) value;
+ collection.forEach(act -> builder.addRepeatedField(field,
createMessage(field.getMessageType(), (Map) act, tableSchema)));
+ } else {
+ builder.setField(field,
createMessage(field.getMessageType(), (Map) value,
tableSchema));
+ }
+ break;
+
+ // INT64 with alias INT, SMALLINT, INTEGER, BIGINT, TINYINT, BYTEINT
+ case INT64:
+ // Integer in the bigquery table schema maps back to INT64
which is considered to be Long on Java side:
+ // https://developers.google.com/protocol-buffers/docs/proto3
+ if (value instanceof Integer) {
+ value = Long.valueOf((Integer) value);
+ }
+
+ setField(value, field, builder);
+ break;
+
+ // FLOAT64
+ case DOUBLE:
+ if (value instanceof Float) {
+ value = ((Float) value).doubleValue();
+ }
+ setField(value, field, builder);
+ break;
+
+ // matches NUMERIC and BIGNUMERIC types in BigQuery
+ // BQTableSchemaToProtoDescriptor.class
+ case BYTES:
+ if(value instanceof BigDecimal) {
+
if(tableSchema.getFields(field.getIndex()).getType().equals(Type.BIGNUMERIC)) {
+ value =
BigDecimalByteStringEncoder.encodeToBigNumericByteString((BigDecimal) value);
+ } else if
(tableSchema.getFields(field.getIndex()).getType().equals(Type.NUMERIC)) {
+ value =
BigDecimalByteStringEncoder.encodeToNumericByteString((BigDecimal) value);
+ }
+ } else if (value instanceof Long) {
+ value = Int64Value.of((long) value).toByteString();
+ } else if (value instanceof Float) {
+ value = FloatValue.of((float) value).toByteString();
+ } else if (value instanceof Double) {
+ value = DoubleValue.of((double) value).toByteString();
+ }
Review Comment:
So I tried to consider the case where we send an Avro ``int`` into a
``NUMERIC`` column for BigQuery but this is completely messing things up. I can
add something like:
java
} else if (value instanceof Integer) {
value = Int64Value.of((int) value).toByteString();
}
But then, in the test when checking the returned value:
java
assertEquals(firstElt.get(0).getDoubleValue(), 1.0);
It would tell me:
org.opentest4j.AssertionFailedError: expected: <2.64E-7> but was: <1.0>
If sending the Avro ``int`` into a ``INT64`` column for BigQuery, then it
works as expected.
I'm going to push a commit containing the additional else if so that it does
not completely fail in NiFi but I believe this would push wrong data into the
BQ table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
[GitHub] [nifi] pvillard31 commented on a diff in pull request #7489: NIFI-11823 - fix NUMERIC support in PutBigQuery
pvillard31 commented on code in PR #7489:
URL: https://github.com/apache/nifi/pull/7489#discussion_r1282393730
##
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/bigquery/proto/ProtoUtils.java:
##
@@ -17,50 +17,99 @@
package org.apache.nifi.processors.gcp.bigquery.proto;
+import com.google.cloud.bigquery.storage.v1.BigDecimalByteStringEncoder;
+import com.google.cloud.bigquery.storage.v1.TableFieldSchema.Type;
+import com.google.cloud.bigquery.storage.v1.TableSchema;
import com.google.protobuf.Descriptors;
+import com.google.protobuf.DoubleValue;
import com.google.protobuf.DynamicMessage;
+import com.google.protobuf.FloatValue;
+import com.google.protobuf.Int64Value;
+
+import java.math.BigDecimal;
import java.util.Arrays;
import java.util.Collection;
import java.util.Map;
/**
- * Util class for protocol buffer messaging
- */
+* Util class for protocol buffer messaging
+*/
public class ProtoUtils {
-public static DynamicMessage createMessage(Descriptors.Descriptor
descriptor, Map valueMap) {
-DynamicMessage.Builder builder = DynamicMessage.newBuilder(descriptor);
+ public static DynamicMessage createMessage(Descriptors.Descriptor
descriptor, Map valueMap, TableSchema tableSchema) {
+ final DynamicMessage.Builder builder =
DynamicMessage.newBuilder(descriptor);
+
+ for (final Descriptors.FieldDescriptor field : descriptor.getFields()) {
+ final String name = field.getName();
+ Object value = valueMap.get(name);
+ if (value == null) {
+ continue;
+ }
+
+ switch (field.getType()) {
+ case MESSAGE:
+ if (field.isRepeated()) {
+ Collection collection = value.getClass().isArray() ?
Arrays.asList((Object[]) value) : (Collection) value;
+ collection.forEach(act -> builder.addRepeatedField(field,
createMessage(field.getMessageType(), (Map) act, tableSchema)));
+ } else {
+ builder.setField(field,
createMessage(field.getMessageType(), (Map) value,
tableSchema));
+ }
+ break;
+
+ // INT64 with alias INT, SMALLINT, INTEGER, BIGINT, TINYINT, BYTEINT
+ case INT64:
+ // Integer in the bigquery table schema maps back to INT64
which is considered to be Long on Java side:
+ // https://developers.google.com/protocol-buffers/docs/proto3
+ if (value instanceof Integer) {
+ value = Long.valueOf((Integer) value);
+ }
+
+ setField(value, field, builder);
+ break;
+
+ // FLOAT64
+ case DOUBLE:
+ if (value instanceof Float) {
+ value = ((Float) value).doubleValue();
+ }
+ setField(value, field, builder);
+ break;
+
+ // matches NUMERIC and BIGNUMERIC types in BigQuery
+ // BQTableSchemaToProtoDescriptor.class
+ case BYTES:
+ if(value instanceof BigDecimal) {
+
if(tableSchema.getFields(field.getIndex()).getType().equals(Type.BIGNUMERIC)) {
Review Comment:
Fixed, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] dan-s1 commented on a diff in pull request #7537: NIFI-11778 Initial check in of refactored Groovy tests in nifi-elasticsearch-restapi-processors to Java (and JUnit 5)
dan-s1 commented on code in PR #7537:
URL: https://github.com/apache/nifi/pull/7537#discussion_r1282327035
##
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/test/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchRecordTest.java:
##
@@ -0,0 +1,784 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License") you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.avro.Schema;
+import org.apache.nifi.avro.AvroTypeUtil;
+import org.apache.nifi.elasticsearch.IndexOperationRequest;
+import org.apache.nifi.elasticsearch.IndexOperationResponse;
+import org.apache.nifi.json.JsonRecordSetWriter;
+import org.apache.nifi.json.JsonTreeReader;
+import org.apache.nifi.processors.elasticsearch.mock.MockBulkLoadClientService;
+import org.apache.nifi.provenance.ProvenanceEventType;
+import org.apache.nifi.schema.access.SchemaAccessUtils;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.record.MockRecordParser;
+import org.apache.nifi.serialization.record.MockSchemaRegistry;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.StringUtils;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.time.format.DateTimeFormatter;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.TimeZone;
+import java.util.function.Consumer;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class PutElasticsearchRecordTest extends
AbstractPutElasticsearchTest {
+private static final int DATE_YEAR = 2020;
+private static final int DATE_MONTH = 11;
+private static final int DATE_DAY = 27;
+private static final int TIME_HOUR = 12;
+private static final int TIME_MINUTE = 55;
+private static final int TIME_SECOND = 23;
+private static final LocalDateTime LOCAL_DATE_TIME =
LocalDateTime.of(DATE_YEAR, DATE_MONTH, DATE_DAY, TIME_HOUR, TIME_MINUTE,
TIME_SECOND);
+private static final LocalDate LOCAL_DATE = LocalDate.of(DATE_YEAR,
DATE_MONTH, DATE_DAY);
+private static final LocalTime LOCAL_TIME = LocalTime.of(TIME_HOUR,
TIME_MINUTE, TIME_SECOND);
+private static final String TEST_DIR =
"src/test/resources/PutElasticsearchRecordTest";
+private static final String TEST_COMMON_DIR = "src/test/resources/common";
+private static String simpleSchema;
+private MockBulkLoadClientService clientService;
+private MockSchemaRegistry registry;
+private TestRunner runner;
+
+@Override
+public Class getTestProcessor() {
+return PutElasticsearchRecord.class;
+}
+
+@BeforeAll
+public static void setUpBeforeClass() throws Exception {
+simpleSchema = Files.readString(Paths.get(TEST_DIR,
"simpleSchema.json"));
+TimeZone.setDefault(TimeZone.getTimeZone("UTC"));
Review Comment:
@exceptionfactory I do not know why this did not surface earlier. I did not
make any changes to assertion code. I cut and pasted it from a Groovy `Closure`
to a Java `Consumer`.
Not sure if this helps or not but in the Groovy code I placed
`TimeZone.setDefault(TimeZone.getTimeZone("Europe/Paris"));`
in the `setup` method of `PutElasticsearchRecordTest` and the tests did not
fail as it did when I placed it in the Java equivalent (as I mentioned
previously).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail:
[GitHub] [nifi] dan-s1 commented on a diff in pull request #7537: NIFI-11778 Initial check in of refactored Groovy tests in nifi-elasticsearch-restapi-processors to Java (and JUnit 5)
dan-s1 commented on code in PR #7537:
URL: https://github.com/apache/nifi/pull/7537#discussion_r1282327035
##
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/test/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchRecordTest.java:
##
@@ -0,0 +1,784 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License") you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.avro.Schema;
+import org.apache.nifi.avro.AvroTypeUtil;
+import org.apache.nifi.elasticsearch.IndexOperationRequest;
+import org.apache.nifi.elasticsearch.IndexOperationResponse;
+import org.apache.nifi.json.JsonRecordSetWriter;
+import org.apache.nifi.json.JsonTreeReader;
+import org.apache.nifi.processors.elasticsearch.mock.MockBulkLoadClientService;
+import org.apache.nifi.provenance.ProvenanceEventType;
+import org.apache.nifi.schema.access.SchemaAccessUtils;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.record.MockRecordParser;
+import org.apache.nifi.serialization.record.MockSchemaRegistry;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.StringUtils;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.time.format.DateTimeFormatter;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.TimeZone;
+import java.util.function.Consumer;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class PutElasticsearchRecordTest extends
AbstractPutElasticsearchTest {
+private static final int DATE_YEAR = 2020;
+private static final int DATE_MONTH = 11;
+private static final int DATE_DAY = 27;
+private static final int TIME_HOUR = 12;
+private static final int TIME_MINUTE = 55;
+private static final int TIME_SECOND = 23;
+private static final LocalDateTime LOCAL_DATE_TIME =
LocalDateTime.of(DATE_YEAR, DATE_MONTH, DATE_DAY, TIME_HOUR, TIME_MINUTE,
TIME_SECOND);
+private static final LocalDate LOCAL_DATE = LocalDate.of(DATE_YEAR,
DATE_MONTH, DATE_DAY);
+private static final LocalTime LOCAL_TIME = LocalTime.of(TIME_HOUR,
TIME_MINUTE, TIME_SECOND);
+private static final String TEST_DIR =
"src/test/resources/PutElasticsearchRecordTest";
+private static final String TEST_COMMON_DIR = "src/test/resources/common";
+private static String simpleSchema;
+private MockBulkLoadClientService clientService;
+private MockSchemaRegistry registry;
+private TestRunner runner;
+
+@Override
+public Class getTestProcessor() {
+return PutElasticsearchRecord.class;
+}
+
+@BeforeAll
+public static void setUpBeforeClass() throws Exception {
+simpleSchema = Files.readString(Paths.get(TEST_DIR,
"simpleSchema.json"));
+TimeZone.setDefault(TimeZone.getTimeZone("UTC"));
Review Comment:
@exceptionfactory I do not know why this did not surface earlier. I did not
make any changes to assertion code. I cut and pasted it from a Groovy `Closure`
to a Java `Consumer`.
Not sure if this helps or not but in the Groovy code I placed
`TimeZone.setDefault(TimeZone.getTimeZone("Europe/Paris"));`
in the `setup` method of `PutElasticsearchRecordTest` and the tests did not
fail as it did when I placed it in the Java equivalent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about
[GitHub] [nifi] dan-s1 commented on a diff in pull request #7537: NIFI-11778 Initial check in of refactored Groovy tests in nifi-elasticsearch-restapi-processors to Java (and JUnit 5)
dan-s1 commented on code in PR #7537:
URL: https://github.com/apache/nifi/pull/7537#discussion_r1282327035
##
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/test/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchRecordTest.java:
##
@@ -0,0 +1,784 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License") you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.avro.Schema;
+import org.apache.nifi.avro.AvroTypeUtil;
+import org.apache.nifi.elasticsearch.IndexOperationRequest;
+import org.apache.nifi.elasticsearch.IndexOperationResponse;
+import org.apache.nifi.json.JsonRecordSetWriter;
+import org.apache.nifi.json.JsonTreeReader;
+import org.apache.nifi.processors.elasticsearch.mock.MockBulkLoadClientService;
+import org.apache.nifi.provenance.ProvenanceEventType;
+import org.apache.nifi.schema.access.SchemaAccessUtils;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.record.MockRecordParser;
+import org.apache.nifi.serialization.record.MockSchemaRegistry;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.StringUtils;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.time.format.DateTimeFormatter;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.TimeZone;
+import java.util.function.Consumer;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class PutElasticsearchRecordTest extends
AbstractPutElasticsearchTest {
+private static final int DATE_YEAR = 2020;
+private static final int DATE_MONTH = 11;
+private static final int DATE_DAY = 27;
+private static final int TIME_HOUR = 12;
+private static final int TIME_MINUTE = 55;
+private static final int TIME_SECOND = 23;
+private static final LocalDateTime LOCAL_DATE_TIME =
LocalDateTime.of(DATE_YEAR, DATE_MONTH, DATE_DAY, TIME_HOUR, TIME_MINUTE,
TIME_SECOND);
+private static final LocalDate LOCAL_DATE = LocalDate.of(DATE_YEAR,
DATE_MONTH, DATE_DAY);
+private static final LocalTime LOCAL_TIME = LocalTime.of(TIME_HOUR,
TIME_MINUTE, TIME_SECOND);
+private static final String TEST_DIR =
"src/test/resources/PutElasticsearchRecordTest";
+private static final String TEST_COMMON_DIR = "src/test/resources/common";
+private static String simpleSchema;
+private MockBulkLoadClientService clientService;
+private MockSchemaRegistry registry;
+private TestRunner runner;
+
+@Override
+public Class getTestProcessor() {
+return PutElasticsearchRecord.class;
+}
+
+@BeforeAll
+public static void setUpBeforeClass() throws Exception {
+simpleSchema = Files.readString(Paths.get(TEST_DIR,
"simpleSchema.json"));
+TimeZone.setDefault(TimeZone.getTimeZone("UTC"));
Review Comment:
@exceptionfactory I do not know why this did not surface earlier. I did not
make any changes to assertion code. I cut and pasted it from a Groovy `Closure`
to a Java `Consumer`.
Not sure if this helps or not but in the Groovy code I placed
`TimeZone.setDefault(TimeZone.getTimeZone("Europe/Paris"));`
in the `setup` method of `PutElasticsearchRecordTest` and the test did not
fail as it did when I placed it in the Java equivalent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about
[GitHub] [nifi] ChrisSamo632 commented on a diff in pull request #7537: NIFI-11778 Initial check in of refactored Groovy tests in nifi-elasticsearch-restapi-processors to Java (and JUnit 5)
ChrisSamo632 commented on code in PR #7537:
URL: https://github.com/apache/nifi/pull/7537#discussion_r1282344859
##
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/test/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchRecordTest.java:
##
@@ -0,0 +1,784 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License") you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.avro.Schema;
+import org.apache.nifi.avro.AvroTypeUtil;
+import org.apache.nifi.elasticsearch.IndexOperationRequest;
+import org.apache.nifi.elasticsearch.IndexOperationResponse;
+import org.apache.nifi.json.JsonRecordSetWriter;
+import org.apache.nifi.json.JsonTreeReader;
+import org.apache.nifi.processors.elasticsearch.mock.MockBulkLoadClientService;
+import org.apache.nifi.provenance.ProvenanceEventType;
+import org.apache.nifi.schema.access.SchemaAccessUtils;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.record.MockRecordParser;
+import org.apache.nifi.serialization.record.MockSchemaRegistry;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.StringUtils;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.time.format.DateTimeFormatter;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.TimeZone;
+import java.util.function.Consumer;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class PutElasticsearchRecordTest extends
AbstractPutElasticsearchTest {
+private static final int DATE_YEAR = 2020;
+private static final int DATE_MONTH = 11;
+private static final int DATE_DAY = 27;
+private static final int TIME_HOUR = 12;
+private static final int TIME_MINUTE = 55;
+private static final int TIME_SECOND = 23;
+private static final LocalDateTime LOCAL_DATE_TIME =
LocalDateTime.of(DATE_YEAR, DATE_MONTH, DATE_DAY, TIME_HOUR, TIME_MINUTE,
TIME_SECOND);
+private static final LocalDate LOCAL_DATE = LocalDate.of(DATE_YEAR,
DATE_MONTH, DATE_DAY);
+private static final LocalTime LOCAL_TIME = LocalTime.of(TIME_HOUR,
TIME_MINUTE, TIME_SECOND);
+private static final String TEST_DIR =
"src/test/resources/PutElasticsearchRecordTest";
+private static final String TEST_COMMON_DIR = "src/test/resources/common";
+private static String simpleSchema;
+private MockBulkLoadClientService clientService;
+private MockSchemaRegistry registry;
+private TestRunner runner;
+
+@Override
+public Class getTestProcessor() {
+return PutElasticsearchRecord.class;
+}
+
+@BeforeAll
+public static void setUpBeforeClass() throws Exception {
+simpleSchema = Files.readString(Paths.get(TEST_DIR,
"simpleSchema.json"));
+TimeZone.setDefault(TimeZone.getTimeZone("UTC"));
Review Comment:
I wonder whether this is due to the Groovy tests having converted, for
example, a `LocalDateTime` into a `Timestamp`, to an `Instant`, and finally the
`long` representation of the "epoch millis" - this would (I think) cater for
the Locale/TimeZone change in the JVM, e.g. a `Europe/Paris` TZ instead of
`UTC`, allowing the tests to use a consistent basis for creating, using, and
then asserting the timestamp/data/time
With the change to a JSON-based structure for the FlowFile contents, are the
timestamps being read from the file in one TZ then the tests and trying to
use/assert them in another?
I recall having to spend some time (no pun intended) on getting these tests
[GitHub] [nifi] dan-s1 commented on a diff in pull request #7537: NIFI-11778 Initial check in of refactored Groovy tests in nifi-elasticsearch-restapi-processors to Java (and JUnit 5)
dan-s1 commented on code in PR #7537:
URL: https://github.com/apache/nifi/pull/7537#discussion_r1282327035
##
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/test/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchRecordTest.java:
##
@@ -0,0 +1,784 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License") you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.avro.Schema;
+import org.apache.nifi.avro.AvroTypeUtil;
+import org.apache.nifi.elasticsearch.IndexOperationRequest;
+import org.apache.nifi.elasticsearch.IndexOperationResponse;
+import org.apache.nifi.json.JsonRecordSetWriter;
+import org.apache.nifi.json.JsonTreeReader;
+import org.apache.nifi.processors.elasticsearch.mock.MockBulkLoadClientService;
+import org.apache.nifi.provenance.ProvenanceEventType;
+import org.apache.nifi.schema.access.SchemaAccessUtils;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.record.MockRecordParser;
+import org.apache.nifi.serialization.record.MockSchemaRegistry;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.StringUtils;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.time.format.DateTimeFormatter;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.TimeZone;
+import java.util.function.Consumer;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class PutElasticsearchRecordTest extends
AbstractPutElasticsearchTest {
+private static final int DATE_YEAR = 2020;
+private static final int DATE_MONTH = 11;
+private static final int DATE_DAY = 27;
+private static final int TIME_HOUR = 12;
+private static final int TIME_MINUTE = 55;
+private static final int TIME_SECOND = 23;
+private static final LocalDateTime LOCAL_DATE_TIME =
LocalDateTime.of(DATE_YEAR, DATE_MONTH, DATE_DAY, TIME_HOUR, TIME_MINUTE,
TIME_SECOND);
+private static final LocalDate LOCAL_DATE = LocalDate.of(DATE_YEAR,
DATE_MONTH, DATE_DAY);
+private static final LocalTime LOCAL_TIME = LocalTime.of(TIME_HOUR,
TIME_MINUTE, TIME_SECOND);
+private static final String TEST_DIR =
"src/test/resources/PutElasticsearchRecordTest";
+private static final String TEST_COMMON_DIR = "src/test/resources/common";
+private static String simpleSchema;
+private MockBulkLoadClientService clientService;
+private MockSchemaRegistry registry;
+private TestRunner runner;
+
+@Override
+public Class getTestProcessor() {
+return PutElasticsearchRecord.class;
+}
+
+@BeforeAll
+public static void setUpBeforeClass() throws Exception {
+simpleSchema = Files.readString(Paths.get(TEST_DIR,
"simpleSchema.json"));
+TimeZone.setDefault(TimeZone.getTimeZone("UTC"));
Review Comment:
@exceptionfactory I do not know why this did not surface earlier. I did not
make any changes to assertion code. I cut and pasted it from a Groovy `Closure`
to a Java `Consumer`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] dan-s1 commented on a diff in pull request #7537: NIFI-11778 Initial check in of refactored Groovy tests in nifi-elasticsearch-restapi-processors to Java (and JUnit 5)
dan-s1 commented on code in PR #7537:
URL: https://github.com/apache/nifi/pull/7537#discussion_r1282327035
##
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/test/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchRecordTest.java:
##
@@ -0,0 +1,784 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License") you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.avro.Schema;
+import org.apache.nifi.avro.AvroTypeUtil;
+import org.apache.nifi.elasticsearch.IndexOperationRequest;
+import org.apache.nifi.elasticsearch.IndexOperationResponse;
+import org.apache.nifi.json.JsonRecordSetWriter;
+import org.apache.nifi.json.JsonTreeReader;
+import org.apache.nifi.processors.elasticsearch.mock.MockBulkLoadClientService;
+import org.apache.nifi.provenance.ProvenanceEventType;
+import org.apache.nifi.schema.access.SchemaAccessUtils;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.record.MockRecordParser;
+import org.apache.nifi.serialization.record.MockSchemaRegistry;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.StringUtils;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.time.format.DateTimeFormatter;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.TimeZone;
+import java.util.function.Consumer;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class PutElasticsearchRecordTest extends
AbstractPutElasticsearchTest {
+private static final int DATE_YEAR = 2020;
+private static final int DATE_MONTH = 11;
+private static final int DATE_DAY = 27;
+private static final int TIME_HOUR = 12;
+private static final int TIME_MINUTE = 55;
+private static final int TIME_SECOND = 23;
+private static final LocalDateTime LOCAL_DATE_TIME =
LocalDateTime.of(DATE_YEAR, DATE_MONTH, DATE_DAY, TIME_HOUR, TIME_MINUTE,
TIME_SECOND);
+private static final LocalDate LOCAL_DATE = LocalDate.of(DATE_YEAR,
DATE_MONTH, DATE_DAY);
+private static final LocalTime LOCAL_TIME = LocalTime.of(TIME_HOUR,
TIME_MINUTE, TIME_SECOND);
+private static final String TEST_DIR =
"src/test/resources/PutElasticsearchRecordTest";
+private static final String TEST_COMMON_DIR = "src/test/resources/common";
+private static String simpleSchema;
+private MockBulkLoadClientService clientService;
+private MockSchemaRegistry registry;
+private TestRunner runner;
+
+@Override
+public Class getTestProcessor() {
+return PutElasticsearchRecord.class;
+}
+
+@BeforeAll
+public static void setUpBeforeClass() throws Exception {
+simpleSchema = Files.readString(Paths.get(TEST_DIR,
"simpleSchema.json"));
+TimeZone.setDefault(TimeZone.getTimeZone("UTC"));
Review Comment:
@exceptionfactory I do not know why this did not surface earlier. I did not
make any changes to assertion code. I cut and pasted it from a Groovy closure
to a Java `Consumer`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] dan-s1 commented on a diff in pull request #7537: NIFI-11778 Initial check in of refactored Groovy tests in nifi-elasticsearch-restapi-processors to Java (and JUnit 5)
dan-s1 commented on code in PR #7537:
URL: https://github.com/apache/nifi/pull/7537#discussion_r1282327035
##
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/test/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchRecordTest.java:
##
@@ -0,0 +1,784 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License") you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.avro.Schema;
+import org.apache.nifi.avro.AvroTypeUtil;
+import org.apache.nifi.elasticsearch.IndexOperationRequest;
+import org.apache.nifi.elasticsearch.IndexOperationResponse;
+import org.apache.nifi.json.JsonRecordSetWriter;
+import org.apache.nifi.json.JsonTreeReader;
+import org.apache.nifi.processors.elasticsearch.mock.MockBulkLoadClientService;
+import org.apache.nifi.provenance.ProvenanceEventType;
+import org.apache.nifi.schema.access.SchemaAccessUtils;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.record.MockRecordParser;
+import org.apache.nifi.serialization.record.MockSchemaRegistry;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.StringUtils;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.time.format.DateTimeFormatter;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.TimeZone;
+import java.util.function.Consumer;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class PutElasticsearchRecordTest extends
AbstractPutElasticsearchTest {
+private static final int DATE_YEAR = 2020;
+private static final int DATE_MONTH = 11;
+private static final int DATE_DAY = 27;
+private static final int TIME_HOUR = 12;
+private static final int TIME_MINUTE = 55;
+private static final int TIME_SECOND = 23;
+private static final LocalDateTime LOCAL_DATE_TIME =
LocalDateTime.of(DATE_YEAR, DATE_MONTH, DATE_DAY, TIME_HOUR, TIME_MINUTE,
TIME_SECOND);
+private static final LocalDate LOCAL_DATE = LocalDate.of(DATE_YEAR,
DATE_MONTH, DATE_DAY);
+private static final LocalTime LOCAL_TIME = LocalTime.of(TIME_HOUR,
TIME_MINUTE, TIME_SECOND);
+private static final String TEST_DIR =
"src/test/resources/PutElasticsearchRecordTest";
+private static final String TEST_COMMON_DIR = "src/test/resources/common";
+private static String simpleSchema;
+private MockBulkLoadClientService clientService;
+private MockSchemaRegistry registry;
+private TestRunner runner;
+
+@Override
+public Class getTestProcessor() {
+return PutElasticsearchRecord.class;
+}
+
+@BeforeAll
+public static void setUpBeforeClass() throws Exception {
+simpleSchema = Files.readString(Paths.get(TEST_DIR,
"simpleSchema.json"));
+TimeZone.setDefault(TimeZone.getTimeZone("UTC"));
Review Comment:
@exceptionfactory I do not know why this did not surface earlier. I did not
make any changes to assertion code. I cut and pasted it from the previous
Groovy closure to a Java `Consumer`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] dan-s1 commented on a diff in pull request #7537: NIFI-11778 Initial check in of refactored Groovy tests in nifi-elasticsearch-restapi-processors to Java (and JUnit 5)
dan-s1 commented on code in PR #7537:
URL: https://github.com/apache/nifi/pull/7537#discussion_r1282327035
##
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/test/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchRecordTest.java:
##
@@ -0,0 +1,784 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License") you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.avro.Schema;
+import org.apache.nifi.avro.AvroTypeUtil;
+import org.apache.nifi.elasticsearch.IndexOperationRequest;
+import org.apache.nifi.elasticsearch.IndexOperationResponse;
+import org.apache.nifi.json.JsonRecordSetWriter;
+import org.apache.nifi.json.JsonTreeReader;
+import org.apache.nifi.processors.elasticsearch.mock.MockBulkLoadClientService;
+import org.apache.nifi.provenance.ProvenanceEventType;
+import org.apache.nifi.schema.access.SchemaAccessUtils;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.record.MockRecordParser;
+import org.apache.nifi.serialization.record.MockSchemaRegistry;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.StringUtils;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.time.format.DateTimeFormatter;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.TimeZone;
+import java.util.function.Consumer;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class PutElasticsearchRecordTest extends
AbstractPutElasticsearchTest {
+private static final int DATE_YEAR = 2020;
+private static final int DATE_MONTH = 11;
+private static final int DATE_DAY = 27;
+private static final int TIME_HOUR = 12;
+private static final int TIME_MINUTE = 55;
+private static final int TIME_SECOND = 23;
+private static final LocalDateTime LOCAL_DATE_TIME =
LocalDateTime.of(DATE_YEAR, DATE_MONTH, DATE_DAY, TIME_HOUR, TIME_MINUTE,
TIME_SECOND);
+private static final LocalDate LOCAL_DATE = LocalDate.of(DATE_YEAR,
DATE_MONTH, DATE_DAY);
+private static final LocalTime LOCAL_TIME = LocalTime.of(TIME_HOUR,
TIME_MINUTE, TIME_SECOND);
+private static final String TEST_DIR =
"src/test/resources/PutElasticsearchRecordTest";
+private static final String TEST_COMMON_DIR = "src/test/resources/common";
+private static String simpleSchema;
+private MockBulkLoadClientService clientService;
+private MockSchemaRegistry registry;
+private TestRunner runner;
+
+@Override
+public Class getTestProcessor() {
+return PutElasticsearchRecord.class;
+}
+
+@BeforeAll
+public static void setUpBeforeClass() throws Exception {
+simpleSchema = Files.readString(Paths.get(TEST_DIR,
"simpleSchema.json"));
+TimeZone.setDefault(TimeZone.getTimeZone("UTC"));
Review Comment:
@exceptionfactory I do not know why this did not surface earlier. I did not
make any changes to assertion code. I cut and pasted it from the previous
Groovy closure to a Java Consumer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (NIFI-11901) Add processor version in processor details view
[ https://issues.apache.org/jira/browse/NIFI-11901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-11901: -- Fix Version/s: 1.latest 2.latest Status: Patch Available (was: Open) > Add processor version in processor details view > --- > > Key: NIFI-11901 > URL: https://issues.apache.org/jira/browse/NIFI-11901 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Fix For: 1.latest, 2.latest > > Time Spent: 10m > Remaining Estimate: 0h > > In NIFI-8937, we added the component version in the view to configure the > component. However this version is not showing in the view of a running > processor. This should also be visible in this case. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] pvillard31 opened a new pull request, #7563: NIFI-11901 - add processor details subtitle in processor details view
pvillard31 opened a new pull request, #7563: URL: https://github.com/apache/nifi/pull/7563 # Summary [NIFI-11901](https://issues.apache.org/jira/browse/NIFI-11901) - add processor details subtitle in processor details view # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [ ] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [ ] Pull Request based on current revision of the `main` branch - [ ] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 17 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-11890) Upgrade guava to 32.0.1-jre for nifi-toolkit version 1.23.0
[ https://issues.apache.org/jira/browse/NIFI-11890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-11890: Status: Patch Available (was: Open) > Upgrade guava to 32.0.1-jre for nifi-toolkit version 1.23.0 > --- > > Key: NIFI-11890 > URL: https://issues.apache.org/jira/browse/NIFI-11890 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.23.0 >Reporter: Phil Lee >Assignee: David Handermann >Priority: Major > Fix For: 1.latest > > > Upgrade guava to 32.0.1-jre for nifi-toolkit version 1.23.0. Newest version > mitigates > [CVE-2023-2976|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-2976] > in 32.0.0-jre -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-11900) Handle IllegalStateException in ResizeImage
[ https://issues.apache.org/jira/browse/NIFI-11900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-11900. - Fix Version/s: 2.0.0 1.24.0 Resolution: Fixed > Handle IllegalStateException in ResizeImage > --- > > Key: NIFI-11900 > URL: https://issues.apache.org/jira/browse/NIFI-11900 > Project: Apache NiFi > Issue Type: Bug >Reporter: Mike Thomsen >Assignee: Mike Thomsen >Priority: Major > Fix For: 2.0.0, 1.24.0 > > Time Spent: 20m > Remaining Estimate: 0h > > We found an edge case in that processor where an IllegalStateException is > thrown on PCX files or whatever those are if they're malformed. The exception > is raised in a third party lib, and we don't currently catch it and route to > the failure relationship. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11900) Handle IllegalStateException in ResizeImage
[ https://issues.apache.org/jira/browse/NIFI-11900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17750458#comment-17750458 ] ASF subversion and git services commented on NIFI-11900: Commit a0191025585882eaaf721d8eab51466a576dc595 in nifi's branch refs/heads/support/nifi-1.x from Mike Thomsen [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=a019102558 ] NIFI-11900 Catch RuntimeException in ResizeImage failures This closes #7558 Signed-off-by: David Handermann (cherry picked from commit 49a350a765e0185ac2dd3deed5a92f58cd0534e3) > Handle IllegalStateException in ResizeImage > --- > > Key: NIFI-11900 > URL: https://issues.apache.org/jira/browse/NIFI-11900 > Project: Apache NiFi > Issue Type: Bug >Reporter: Mike Thomsen >Assignee: Mike Thomsen >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > We found an edge case in that processor where an IllegalStateException is > thrown on PCX files or whatever those are if they're malformed. The exception > is raised in a third party lib, and we don't currently catch it and route to > the failure relationship. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11900) Handle IllegalStateException in ResizeImage
[ https://issues.apache.org/jira/browse/NIFI-11900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17750455#comment-17750455 ] ASF subversion and git services commented on NIFI-11900: Commit 49a350a765e0185ac2dd3deed5a92f58cd0534e3 in nifi's branch refs/heads/main from Mike Thomsen [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=49a350a765 ] NIFI-11900 Catch RuntimeException in ResizeImage failures This closes #7558 Signed-off-by: David Handermann > Handle IllegalStateException in ResizeImage > --- > > Key: NIFI-11900 > URL: https://issues.apache.org/jira/browse/NIFI-11900 > Project: Apache NiFi > Issue Type: Bug >Reporter: Mike Thomsen >Assignee: Mike Thomsen >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > We found an edge case in that processor where an IllegalStateException is > thrown on PCX files or whatever those are if they're malformed. The exception > is raised in a third party lib, and we don't currently catch it and route to > the failure relationship. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] exceptionfactory closed pull request #7558: NIFI-11900 Added catch for IllegalStateException
exceptionfactory closed pull request #7558: NIFI-11900 Added catch for IllegalStateException URL: https://github.com/apache/nifi/pull/7558 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] ChrisSamo632 commented on a diff in pull request #7537: NIFI-11778 Initial check in of refactored Groovy tests in nifi-elasticsearch-restapi-processors to Java (and JUnit 5)
ChrisSamo632 commented on code in PR #7537:
URL: https://github.com/apache/nifi/pull/7537#discussion_r1282306029
##
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/test/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchRecordTest.java:
##
@@ -0,0 +1,784 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License") you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.avro.Schema;
+import org.apache.nifi.avro.AvroTypeUtil;
+import org.apache.nifi.elasticsearch.IndexOperationRequest;
+import org.apache.nifi.elasticsearch.IndexOperationResponse;
+import org.apache.nifi.json.JsonRecordSetWriter;
+import org.apache.nifi.json.JsonTreeReader;
+import org.apache.nifi.processors.elasticsearch.mock.MockBulkLoadClientService;
+import org.apache.nifi.provenance.ProvenanceEventType;
+import org.apache.nifi.schema.access.SchemaAccessUtils;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.record.MockRecordParser;
+import org.apache.nifi.serialization.record.MockSchemaRegistry;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.StringUtils;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.time.format.DateTimeFormatter;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.TimeZone;
+import java.util.function.Consumer;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class PutElasticsearchRecordTest extends
AbstractPutElasticsearchTest {
+private static final int DATE_YEAR = 2020;
+private static final int DATE_MONTH = 11;
+private static final int DATE_DAY = 27;
+private static final int TIME_HOUR = 12;
+private static final int TIME_MINUTE = 55;
+private static final int TIME_SECOND = 23;
+private static final LocalDateTime LOCAL_DATE_TIME =
LocalDateTime.of(DATE_YEAR, DATE_MONTH, DATE_DAY, TIME_HOUR, TIME_MINUTE,
TIME_SECOND);
+private static final LocalDate LOCAL_DATE = LocalDate.of(DATE_YEAR,
DATE_MONTH, DATE_DAY);
+private static final LocalTime LOCAL_TIME = LocalTime.of(TIME_HOUR,
TIME_MINUTE, TIME_SECOND);
+private static final String TEST_DIR =
"src/test/resources/PutElasticsearchRecordTest";
+private static final String TEST_COMMON_DIR = "src/test/resources/common";
+private static String simpleSchema;
+private MockBulkLoadClientService clientService;
+private MockSchemaRegistry registry;
+private TestRunner runner;
+
+@Override
+public Class getTestProcessor() {
+return PutElasticsearchRecord.class;
+}
+
+@BeforeAll
+public static void setUpBeforeClass() throws Exception {
+simpleSchema = Files.readString(Paths.get(TEST_DIR,
"simpleSchema.json"));
+TimeZone.setDefault(TimeZone.getTimeZone("UTC"));
Review Comment:
Needs a closer look - this doesn't seem right
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on pull request #7537: NIFI-11778 Initial check in of refactored Groovy tests in nifi-elasticsearch-restapi-processors to Java (and JUnit 5)
exceptionfactory commented on PR #7537:
URL: https://github.com/apache/nifi/pull/7537#issuecomment-1662802081
> > @ChrisSamo632 Sorry by accident closed this PR and the reopened. Would
it be okay to fix the test errors by placing in `setUpBeforeClass`
> > `TimeZone.setDefault(TimeZone.getTimeZone("UTC"));` ?
>
> I'm not sure this sounds like the right way to fix this - the Checks run
against multiple timezones for good reason, we want to be sure that processors
can handle changes in locale. Having to set the tests to use a specific
timezone to pass, seems like we might be hiding a potential issue either in the
tests or the processor code itself.
>
> Unfortunately, handling multiple timezones isn't my strong suite, although
I'll try to take a look. I wonder whether @exceptionfactory or @markap14 might
be able to provide some guidance on this one?
I will try to take a closer look at the tests soon, but I agree, changing
the default timezone in test setup code is not the right approach.
All of the automated builds run with different timezones to ensure expected
behavior in various locations, so all tests must pass without changing the
system timezone.
In general, the issue means that there is some timezone conversion
happening, which could point to a runtime issue, or simply an issue with the
sample data used for testing. I'm not sure exactly why this has not surfaced
earlier, so it points to an issue with the test-related changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (NIFI-11901) Add processor version in processor details view
[ https://issues.apache.org/jira/browse/NIFI-11901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-11901: -- Summary: Add processor version in processor details view (was: Add processor version when component is details view) > Add processor version in processor details view > --- > > Key: NIFI-11901 > URL: https://issues.apache.org/jira/browse/NIFI-11901 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > > In NIFI-8937, we added the component version in the view to configure the > component. However this version is not showing in the view of a running > processor. This should also be visible in this case. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] ChrisSamo632 commented on pull request #7537: NIFI-11778 Initial check in of refactored Groovy tests in nifi-elasticsearch-restapi-processors to Java (and JUnit 5)
ChrisSamo632 commented on PR #7537:
URL: https://github.com/apache/nifi/pull/7537#issuecomment-1662792006
> @ChrisSamo632 Sorry by accident closed this PR and the reopened. Would it
be okay to fix the test errors by placing in `setUpBeforeClass`
>
> `TimeZone.setDefault(TimeZone.getTimeZone("UTC"));` ?
I'm not sure this sounds like the right way to fix this - the Checks run
against multiple timezones for good reason, we want to be sure that processors
can handle changes in locale. Having to set the tests to use a specific
timezone to pass, seems like we might be hiding a potential issue either in the
tests or the processor code itself.
Unfortunately, handling multiple timezones isn't my strong suite, although
I'll try to take a look. I wonder whether @exceptionfactory or @markap14 might
be able to provide some guidance on this one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (NIFI-11901) Add processor version when component is details view
Pierre Villard created NIFI-11901: - Summary: Add processor version when component is details view Key: NIFI-11901 URL: https://issues.apache.org/jira/browse/NIFI-11901 Project: Apache NiFi Issue Type: Improvement Components: Core UI Reporter: Pierre Villard Assignee: Pierre Villard In NIFI-8937, we added the component version in the view to configure the component. However this version is not showing in the view of a running processor. This should also be visible in this case. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11898) Handle auto-commit and commit based on driver capabilities in PutDatabaseRecord
[ https://issues.apache.org/jira/browse/NIFI-11898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-11898: Status: Patch Available (was: In Progress) > Handle auto-commit and commit based on driver capabilities in > PutDatabaseRecord > --- > > Key: NIFI-11898 > URL: https://issues.apache.org/jira/browse/NIFI-11898 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 1.latest, 2.latest > > Time Spent: 10m > Remaining Estimate: 0h > > Some JDBC drivers throw an exception when calling setAutoCommit(false), even > if the auto-commit setting for the driver is already false. Although > unfortunate, I'm not sure it is a violation of the JDBC API. > PutDatabaseRecord should catch SQLFeatureNotSupportedException and perhaps > log a debug message but otherwise proceed. > Same goes for connection.commit() and connection.rollback(), these should not > be explicitly called if auto-commit is true for the connection. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] mattyb149 opened a new pull request, #7561: NIFI-11898: Handle auto-commit and commit based on driver capabilities in PutDatabaseRecord
mattyb149 opened a new pull request, #7561: URL: https://github.com/apache/nifi/pull/7561 # Summary [NIFI-11898](https://issues.apache.org/jira/browse/NIFI-11898) This is the `main` version of PR #7560 , please see the other PR for details. # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [x] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [x] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [x] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [x] Pull Request based on current revision of the `main` branch - [x] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [x] Build completed using `mvn clean install -P contrib-check` - [x] JDK 17 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-11860) Erroneously set decimalScale when resultSetScale is 0
[ https://issues.apache.org/jira/browse/NIFI-11860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17750402#comment-17750402 ] ASF subversion and git services commented on NIFI-11860: Commit 80e71068bec6a665486d1264bb582ed94bd84bd3 in nifi's branch refs/heads/main from Tamas Neumer [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=80e71068be ] NIFI-11860 fixed bug in ResultSetRecordSet affecting decimalScale when resultSetScale was zero. Signed-off-by: Matt Burgess This closes #7545 > Erroneously set decimalScale when resultSetScale is 0 > - > > Key: NIFI-11860 > URL: https://issues.apache.org/jira/browse/NIFI-11860 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Tamas Neumer >Assignee: Tamas Neumer >Priority: Minor > Fix For: 1.latest, 2.latest > > Time Spent: 50m > Remaining Estimate: 0h > > Hi, > In the > [ResultSetRecordSet|https://github.com/apache/nifi/blob/a61add22c2507025a901e9ba753b3a6eb976d7e1/nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/ResultSetRecordSet.java#L328] > if the resultScale is 0, then the decimal scale is set to the defaultScale. > This however is inconsistent with the [JDBC Common's > behavior|https://github.com/apache/nifi/commit/4fde9648dd8ec621ebcdf1bfd0b80786a0ff68da#diff-18c3d769d6220682bdc35a1c3136aea9ff74d323610c992daa22f7689ed3482dR585]. > > We believe that in the ResultSetRecordSet the same >= should be used. > > BR, > Tamas > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (MINIFICPP-1817) Use magic enum library instead of SMART_ENUM macros
[ https://issues.apache.org/jira/browse/MINIFICPP-1817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17750401#comment-17750401 ] Marton Szasz commented on MINIFICPP-1817: - We should add a unit test for the new enumCast > Use magic enum library instead of SMART_ENUM macros > --- > > Key: MINIFICPP-1817 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1817 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Gábor Gyimesi >Assignee: Gábor Gyimesi >Priority: Minor > Time Spent: 1.5h > Remaining Estimate: 0h > > Instead of the currently used SMART_ENUM macros we could use > [Neargye/magic_enum|https://github.com/Neargye/magic_enum] library written in > C++17. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11860) Erroneously set decimalScale when resultSetScale is 0
[ https://issues.apache.org/jira/browse/NIFI-11860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-11860: Fix Version/s: 2.0.0 1.24.0 (was: 1.latest) (was: 2.latest) Resolution: Fixed Status: Resolved (was: Patch Available) > Erroneously set decimalScale when resultSetScale is 0 > - > > Key: NIFI-11860 > URL: https://issues.apache.org/jira/browse/NIFI-11860 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Tamas Neumer >Assignee: Tamas Neumer >Priority: Minor > Fix For: 2.0.0, 1.24.0 > > Time Spent: 1h > Remaining Estimate: 0h > > Hi, > In the > [ResultSetRecordSet|https://github.com/apache/nifi/blob/a61add22c2507025a901e9ba753b3a6eb976d7e1/nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/ResultSetRecordSet.java#L328] > if the resultScale is 0, then the decimal scale is set to the defaultScale. > This however is inconsistent with the [JDBC Common's > behavior|https://github.com/apache/nifi/commit/4fde9648dd8ec621ebcdf1bfd0b80786a0ff68da#diff-18c3d769d6220682bdc35a1c3136aea9ff74d323610c992daa22f7689ed3482dR585]. > > We believe that in the ResultSetRecordSet the same >= should be used. > > BR, > Tamas > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] mattyb149 closed pull request #7545: NIFI-11860 fixed bug in ResultSetRecordSet affecting decimalScale whe…
mattyb149 closed pull request #7545: NIFI-11860 fixed bug in ResultSetRecordSet affecting decimalScale whe… URL: https://github.com/apache/nifi/pull/7545 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] mattyb149 commented on pull request #7545: NIFI-11860 fixed bug in ResultSetRecordSet affecting decimalScale whe…
mattyb149 commented on PR #7545: URL: https://github.com/apache/nifi/pull/7545#issuecomment-1662531068 +1 LGTM, thanks for the fix! Merging to main and support/nifi-1.x -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-11860) Erroneously set decimalScale when resultSetScale is 0
[ https://issues.apache.org/jira/browse/NIFI-11860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17750400#comment-17750400 ] ASF subversion and git services commented on NIFI-11860: Commit c203205b3b6ef88778b721f43b2ccc68fbafc9d9 in nifi's branch refs/heads/support/nifi-1.x from Tamas Neumer [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=c203205b3b ] NIFI-11860 fixed bug in ResultSetRecordSet affecting decimalScale when resultSetScale was zero. Signed-off-by: Matt Burgess > Erroneously set decimalScale when resultSetScale is 0 > - > > Key: NIFI-11860 > URL: https://issues.apache.org/jira/browse/NIFI-11860 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Tamas Neumer >Assignee: Tamas Neumer >Priority: Minor > Fix For: 1.latest, 2.latest > > Time Spent: 40m > Remaining Estimate: 0h > > Hi, > In the > [ResultSetRecordSet|https://github.com/apache/nifi/blob/a61add22c2507025a901e9ba753b3a6eb976d7e1/nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/ResultSetRecordSet.java#L328] > if the resultScale is 0, then the decimal scale is set to the defaultScale. > This however is inconsistent with the [JDBC Common's > behavior|https://github.com/apache/nifi/commit/4fde9648dd8ec621ebcdf1bfd0b80786a0ff68da#diff-18c3d769d6220682bdc35a1c3136aea9ff74d323610c992daa22f7689ed3482dR585]. > > We believe that in the ResultSetRecordSet the same >= should be used. > > BR, > Tamas > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11860) Erroneously set decimalScale when resultSetScale is 0
[ https://issues.apache.org/jira/browse/NIFI-11860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-11860: Fix Version/s: 1.latest 2.latest > Erroneously set decimalScale when resultSetScale is 0 > - > > Key: NIFI-11860 > URL: https://issues.apache.org/jira/browse/NIFI-11860 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Tamas Neumer >Assignee: Tamas Neumer >Priority: Minor > Fix For: 1.latest, 2.latest > > Time Spent: 40m > Remaining Estimate: 0h > > Hi, > In the > [ResultSetRecordSet|https://github.com/apache/nifi/blob/a61add22c2507025a901e9ba753b3a6eb976d7e1/nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/ResultSetRecordSet.java#L328] > if the resultScale is 0, then the decimal scale is set to the defaultScale. > This however is inconsistent with the [JDBC Common's > behavior|https://github.com/apache/nifi/commit/4fde9648dd8ec621ebcdf1bfd0b80786a0ff68da#diff-18c3d769d6220682bdc35a1c3136aea9ff74d323610c992daa22f7689ed3482dR585]. > > We believe that in the ResultSetRecordSet the same >= should be used. > > BR, > Tamas > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-8606) Add Stop & Configure button to Controller Service Details dialog
[ https://issues.apache.org/jira/browse/NIFI-8606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynaldo Rea reassigned NIFI-8606: -- Assignee: Reynaldo Rea > Add Stop & Configure button to Controller Service Details dialog > > > Key: NIFI-8606 > URL: https://issues.apache.org/jira/browse/NIFI-8606 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Affects Versions: 1.13.2 >Reporter: Mark Bean >Assignee: Reynaldo Rea >Priority: Minor > > This ticket is similar to NIFI-5986 which added the 'stop and configure' > option to a "Configure Processor" details dialog, i.e configure a processor. > The same feature should be available when in the "Controller Service > Details" dialog of a controller service. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-11898) Handle auto-commit and commit based on driver capabilities in PutDatabaseRecord
[ https://issues.apache.org/jira/browse/NIFI-11898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess reassigned NIFI-11898: --- Assignee: Matt Burgess > Handle auto-commit and commit based on driver capabilities in > PutDatabaseRecord > --- > > Key: NIFI-11898 > URL: https://issues.apache.org/jira/browse/NIFI-11898 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 1.latest, 2.latest > > > Some JDBC drivers throw an exception when calling setAutoCommit(false), even > if the auto-commit setting for the driver is already false. Although > unfortunate, I'm not sure it is a violation of the JDBC API. > PutDatabaseRecord should catch SQLFeatureNotSupportedException and perhaps > log a debug message but otherwise proceed. > Same goes for connection.commit() and connection.rollback(), these should not > be explicitly called if auto-commit is true for the connection. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (MINIFICPP-426) Update bootstrap to download libcurl-openssl on centos and rhel
[ https://issues.apache.org/jira/browse/MINIFICPP-426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz resolved MINIFICPP-426. Resolution: Invalid 3rd party libs are downloaded through cmake > Update bootstrap to download libcurl-openssl on centos and rhel > --- > > Key: MINIFICPP-426 > URL: https://issues.apache.org/jira/browse/MINIFICPP-426 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Marc Parisi >Assignee: Marc Parisi >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (MINIFICPP-420) Add max size to repository metrics
[ https://issues.apache.org/jira/browse/MINIFICPP-420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz resolved MINIFICPP-420. Resolution: Done Where there is a max size, it's part of the metrics. > Add max size to repository metrics > -- > > Key: MINIFICPP-420 > URL: https://issues.apache.org/jira/browse/MINIFICPP-420 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Marc Parisi >Priority: Major > > C2 repository metrics should include the maximum size of the repository > (prov/flow/content). The maximum size is a configuration option that can vary > by agent or agent type. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (MINIFICPP-412) Publish MiNiFi Docs to website
[ https://issues.apache.org/jira/browse/MINIFICPP-412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz updated MINIFICPP-412: --- Priority: Minor (was: Major) > Publish MiNiFi Docs to website > -- > > Key: MINIFICPP-412 > URL: https://issues.apache.org/jira/browse/MINIFICPP-412 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Aldrin Piri >Priority: Minor > > Items such as Processors, Extensions, and Expression language docs are > currently only available with the source. We should also get these at least > linked if not published to the MiNiFi site for easier consumption. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (MINIFICPP-398) Support schema config version 4
[ https://issues.apache.org/jira/browse/MINIFICPP-398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz resolved MINIFICPP-398. Resolution: Invalid The heartbeat manifest contains every configurable option. > Support schema config version 4 > --- > > Key: MINIFICPP-398 > URL: https://issues.apache.org/jira/browse/MINIFICPP-398 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Aldrin Piri >Priority: Major > > As part of MINIFI-66, and to help us carry out some initial C2 > functionalities, a closer mapping of config schema to templates will allow us > to more easily make use of registry functionality. This will require that > instance based configuration (things like repositories, security and the > like) are extracted to other files such as the conf and/or other .properties > files. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (MINIFICPP-386) Boostrap script should allow for alternative installation directories when building packages
[ https://issues.apache.org/jira/browse/MINIFICPP-386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz updated MINIFICPP-386: --- Priority: Minor (was: Major) > Boostrap script should allow for alternative installation directories when > building packages > > > Key: MINIFICPP-386 > URL: https://issues.apache.org/jira/browse/MINIFICPP-386 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Marc Parisi >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (MINIFICPP-375) Make log location configurable
[ https://issues.apache.org/jira/browse/MINIFICPP-375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz resolved MINIFICPP-375. Resolution: Done > Make log location configurable > -- > > Key: MINIFICPP-375 > URL: https://issues.apache.org/jira/browse/MINIFICPP-375 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Aldrin Piri >Priority: Major > > We should provide a property that allows us to specify the location of log > files for a given instance. Users may potentially want these logs to exist > outside of the instance directory. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (MINIFICPP-381) Update docs to reference the 0.4.0 configuration toolkit now that it is available
[ https://issues.apache.org/jira/browse/MINIFICPP-381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz resolved MINIFICPP-381. Resolution: Won't Fix > Update docs to reference the 0.4.0 configuration toolkit now that it is > available > - > > Key: MINIFICPP-381 > URL: https://issues.apache.org/jira/browse/MINIFICPP-381 > Project: Apache NiFi MiNiFi C++ > Issue Type: Task >Reporter: Aldrin Piri >Assignee: Aldrin Piri >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (MINIFICPP-360) Improve ARM build procedures
[ https://issues.apache.org/jira/browse/MINIFICPP-360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz updated MINIFICPP-360: --- Issue Type: Improvement (was: Bug) > Improve ARM build procedures > > > Key: MINIFICPP-360 > URL: https://issues.apache.org/jira/browse/MINIFICPP-360 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Marc Parisi >Priority: Trivial > > It would be nice to automate some of the cross compiling toolchain. This is > going to be difficult. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (MINIFICPP-360) Improve ARM build procedures
[ https://issues.apache.org/jira/browse/MINIFICPP-360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz updated MINIFICPP-360: --- Priority: Trivial (was: Major) > Improve ARM build procedures > > > Key: MINIFICPP-360 > URL: https://issues.apache.org/jira/browse/MINIFICPP-360 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Marc Parisi >Priority: Trivial > > It would be nice to automate some of the cross compiling toolchain. This is > going to be difficult. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (MINIFICPP-353) Re-enable TF and Bustache in bootstrap
[ https://issues.apache.org/jira/browse/MINIFICPP-353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz resolved MINIFICPP-353. Resolution: Abandoned TF is dropped in favor of python scripting, bustache is in bootstrap > Re-enable TF and Bustache in bootstrap > -- > > Key: MINIFICPP-353 > URL: https://issues.apache.org/jira/browse/MINIFICPP-353 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Marc Parisi >Assignee: Marc Parisi >Priority: Major > > Re-enable TF and Bustache in bootstrap -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (MINIFICPP-345) Explore combining unit tests to speed up build
[ https://issues.apache.org/jira/browse/MINIFICPP-345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz updated MINIFICPP-345: --- Priority: Trivial (was: Minor) > Explore combining unit tests to speed up build > -- > > Key: MINIFICPP-345 > URL: https://issues.apache.org/jira/browse/MINIFICPP-345 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Marc Parisi >Priority: Trivial > > Combining some of the unit test files should help improve build times a > little. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (MINIFICPP-345) Explore combining unit tests to speed up build
[ https://issues.apache.org/jira/browse/MINIFICPP-345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Zink updated MINIFICPP-345: -- Issue Type: Improvement (was: Bug) > Explore combining unit tests to speed up build > -- > > Key: MINIFICPP-345 > URL: https://issues.apache.org/jira/browse/MINIFICPP-345 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Marc Parisi >Priority: Minor > > Combining some of the unit test files should help improve build times a > little. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (MINIFICPP-334) Implement JSON path lookup in Expression Language
[ https://issues.apache.org/jira/browse/MINIFICPP-334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz reassigned MINIFICPP-334: -- Assignee: (was: Andrew Christianson) > Implement JSON path lookup in Expression Language > - > > Key: MINIFICPP-334 > URL: https://issues.apache.org/jira/browse/MINIFICPP-334 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Andrew Christianson >Priority: Minor > > Implement jsonPath function > https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#jsonpath -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (MINIFICPP-334) Implement JSON path lookup in Expression Language
[ https://issues.apache.org/jira/browse/MINIFICPP-334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz updated MINIFICPP-334: --- Priority: Minor (was: Major) > Implement JSON path lookup in Expression Language > - > > Key: MINIFICPP-334 > URL: https://issues.apache.org/jira/browse/MINIFICPP-334 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Andrew Christianson >Assignee: Andrew Christianson >Priority: Minor > > Implement jsonPath function > https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#jsonpath -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (MINIFICPP-345) Explore combining unit tests to speed up build
[ https://issues.apache.org/jira/browse/MINIFICPP-345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz updated MINIFICPP-345: --- Priority: Minor (was: Major) > Explore combining unit tests to speed up build > -- > > Key: MINIFICPP-345 > URL: https://issues.apache.org/jira/browse/MINIFICPP-345 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Marc Parisi >Priority: Minor > > Combining some of the unit test files should help improve build times a > little. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (MINIFICPP-309) Improve logging and institute metrics for Scheduling agents
[ https://issues.apache.org/jira/browse/MINIFICPP-309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz resolved MINIFICPP-309. Resolution: Done mostly done > Improve logging and institute metrics for Scheduling agents > --- > > Key: MINIFICPP-309 > URL: https://issues.apache.org/jira/browse/MINIFICPP-309 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Marc Parisi >Priority: Major > > Implement some low cost logging for the scheduling agents to get a better > idea of what's occurring. Metrics may be a substitute for logging if > implemented well. This ticket is borne trying to determine why ontrigger > calls were not getting called. It was not immediately clear and adding > metrics/logging can help with future debug. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (MINIFICPP-323) Create templated ExtensionFactory that uses the template argument pack to auto determine the classes.
[ https://issues.apache.org/jira/browse/MINIFICPP-323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz resolved MINIFICPP-323. Resolution: Abandoned > Create templated ExtensionFactory that uses the template argument pack to > auto determine the classes. > - > > Key: MINIFICPP-323 > URL: https://issues.apache.org/jira/browse/MINIFICPP-323 > Project: Apache NiFi MiNiFi C++ > Issue Type: Sub-task >Reporter: Marc Parisi >Assignee: Marc Parisi >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (MINIFICPP-322) Improve extension development and the documentation therein.
[ https://issues.apache.org/jira/browse/MINIFICPP-322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz resolved MINIFICPP-322. Resolution: Abandoned > Improve extension development and the documentation therein. > > > Key: MINIFICPP-322 > URL: https://issues.apache.org/jira/browse/MINIFICPP-322 > Project: Apache NiFi MiNiFi C++ > Issue Type: Epic >Reporter: Marc Parisi >Assignee: Marc Parisi >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (MINIFICPP-300) Add attribute noinline to logging
[ https://issues.apache.org/jira/browse/MINIFICPP-300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz resolved MINIFICPP-300. Resolution: Won't Fix I think we have much larger perf bottlenecks than branch mispredictions and instruction cache problems > Add attribute noinline to logging > - > > Key: MINIFICPP-300 > URL: https://issues.apache.org/jira/browse/MINIFICPP-300 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Marc Parisi >Priority: Major > > Logging functions are becoming inlined. This is causing the instruction cache > on my machine to be filled with logging code and appearing to throw off > branch prediction. We should probably not inline logging functions. This is a > micro-optimization that we should consider since this is very typical and can > usually shave milliseconds off of low latency operations. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (MINIFICPP-298) Schedulers should be able to detect if portions of the graph can immediately flow
[ https://issues.apache.org/jira/browse/MINIFICPP-298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz resolved MINIFICPP-298. Resolution: Fixed Event driven scheduling can do this > Schedulers should be able to detect if portions of the graph can immediately > flow > - > > Key: MINIFICPP-298 > URL: https://issues.apache.org/jira/browse/MINIFICPP-298 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Marc Parisi >Priority: Major > > The graph may allow us to immediately move flow files to certain points > without waiting on the scheduler, so we may be able to call onTrigger > automatically following the notification of the connection, assuming this > does not impact thread 1. We should see if this means we only do this in some > threads or all threads. The latter will likely simplify the code and minimize > problems. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (MINIFICPP-294) CURLOPT_TCP_KEEPALIVE should be removed since the base version of of libcurl does not have it on centos 6
[ https://issues.apache.org/jira/browse/MINIFICPP-294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz resolved MINIFICPP-294. Resolution: Won't Fix centos 6 is no longer supported > CURLOPT_TCP_KEEPALIVE should be removed since the base version of of libcurl > does not have it on centos 6 > - > > Key: MINIFICPP-294 > URL: https://issues.apache.org/jira/browse/MINIFICPP-294 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Marc Parisi >Assignee: Marc Parisi >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (MINIFICPP-288) Push flowfiles directly to a processor once persisted to avoid relying on spills to enforce state transition
[ https://issues.apache.org/jira/browse/MINIFICPP-288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz resolved MINIFICPP-288. Resolution: Invalid not clear > Push flowfiles directly to a processor once persisted to avoid relying on > spills to enforce state transition > > > Key: MINIFICPP-288 > URL: https://issues.apache.org/jira/browse/MINIFICPP-288 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Marc Parisi >Priority: Major > Labels: CAPI > > Currently we may rely on multiple threads that ultimately rely on state > transition through the repositories; however, we may be able to avoid this > and do some of the work serially. This should be detectable – this will also > facilitate using the C API and improve throughput and need for repositories. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (MINIFICPP-285) ExecuteProcess does not stop within linux
[ https://issues.apache.org/jira/browse/MINIFICPP-285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz updated MINIFICPP-285: --- Priority: Minor (was: Major) > ExecuteProcess does not stop within linux > - > > Key: MINIFICPP-285 > URL: https://issues.apache.org/jira/browse/MINIFICPP-285 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Marc Parisi >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
