[jira] [Resolved] (NIFI-12866) DataTypeUtils long to Timestamp conversion may not handle historical dates correctly
[
https://issues.apache.org/jira/browse/NIFI-12866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lehel Boér resolved NIFI-12866.
---
Resolution: Won't Fix
> DataTypeUtils long to Timestamp conversion may not handle historical dates
> correctly
>
>
> Key: NIFI-12866
> URL: https://issues.apache.org/jira/browse/NIFI-12866
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Lehel Boér
>Assignee: Lehel Boér
>Priority: Major
>
> * NiFi version: 1.25.0
> * Slack discussion:
> https://apachenifi.slack.com/archives/C0L9VCD47/p1709571300887339
> When inserting default time -6213559680 (0001-01-01) from NiFi to
> PostgreSQL, it results in 0001-01-03.
> The conversion happens in DataTypeUtils::toTimeStamp.
>
>
> {code:java}
> if (value instanceof Number) {
> final long longValue = ((Number) value).longValue();
> return new Timestamp(longValue);
> } {code}
>
>
> When Java created a new Timestamp, it constructs a Date object internally.
> When creating a Date object witht he same value it is also shifted.
> {code:java}
> Date date = new Date(-6213559680L);
> 1-01-02T19:00:00.000-0500" {code}
>
> This issue needs to be investigated.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (NIFI-12866) DataTypeUtils long to Timestamp conversion may not handle historical dates correctly
[
https://issues.apache.org/jira/browse/NIFI-12866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17827767#comment-17827767
]
Lehel Boér edited comment on NIFI-12866 at 3/17/24 5:01 AM:
The date conversion issue comes from how Instant represents time as epoch
millis from the first (default) epoch (1970-01-01) and how it is represented in
an actual Julian calendar in a Date object. There are day skips, leap second so
that an arbitrary point in time especially with historical dates can be
represented differently. (E.g. in 1582-10, ten days vanished because of the
switch from Julian calendar to Gregorian calendar). So Java actually shows the
correct date counting negatively from 1970-01-01 represented in the Julian
calendar.
was (Author: lehel44):
The date conversion issue comes from how Instant represents time as epoch
millis from the first (default) epoch (1970-01-01) and how it is represented in
an actual Julian calendar in a Date object. There are day skips, leap second so
that an arbitrary point in time especially with historical dates can be
represented differently. (E.g. in 1582-10, ten days vanished because of the
switch from Julian calendar to Gregorian calendar).
> DataTypeUtils long to Timestamp conversion may not handle historical dates
> correctly
>
>
> Key: NIFI-12866
> URL: https://issues.apache.org/jira/browse/NIFI-12866
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Lehel Boér
>Assignee: Lehel Boér
>Priority: Major
>
> * NiFi version: 1.25.0
> * Slack discussion:
> https://apachenifi.slack.com/archives/C0L9VCD47/p1709571300887339
> When inserting default time -6213559680 (0001-01-01) from NiFi to
> PostgreSQL, it results in 0001-01-03.
> The conversion happens in DataTypeUtils::toTimeStamp.
>
>
> {code:java}
> if (value instanceof Number) {
> final long longValue = ((Number) value).longValue();
> return new Timestamp(longValue);
> } {code}
>
>
> When Java created a new Timestamp, it constructs a Date object internally.
> When creating a Date object witht he same value it is also shifted.
> {code:java}
> Date date = new Date(-6213559680L);
> 1-01-02T19:00:00.000-0500" {code}
>
> This issue needs to be investigated.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12866) DataTypeUtils long to Timestamp conversion may not handle historical dates correctly
[
https://issues.apache.org/jira/browse/NIFI-12866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17827767#comment-17827767
]
Lehel Boér commented on NIFI-12866:
---
The date conversion issue comes from how Instant represents time as epoch
millis from the first (default) epoch (1970-01-01) and how it is represented in
an actual Julian calendar in a Date object. There are day skips, leap second so
that an arbitrary point in time especially with historical dates can be
represented differently. (E.g. in 1582-10, ten days vanished because of the
switch from Julian calendar to Gregorian calendar).
> DataTypeUtils long to Timestamp conversion may not handle historical dates
> correctly
>
>
> Key: NIFI-12866
> URL: https://issues.apache.org/jira/browse/NIFI-12866
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Lehel Boér
>Assignee: Lehel Boér
>Priority: Major
>
> * NiFi version: 1.25.0
> * Slack discussion:
> https://apachenifi.slack.com/archives/C0L9VCD47/p1709571300887339
> When inserting default time -6213559680 (0001-01-01) from NiFi to
> PostgreSQL, it results in 0001-01-03.
> The conversion happens in DataTypeUtils::toTimeStamp.
>
>
> {code:java}
> if (value instanceof Number) {
> final long longValue = ((Number) value).longValue();
> return new Timestamp(longValue);
> } {code}
>
>
> When Java created a new Timestamp, it constructs a Date object internally.
> When creating a Date object witht he same value it is also shifted.
> {code:java}
> Date date = new Date(-6213559680L);
> 1-01-02T19:00:00.000-0500" {code}
>
> This issue needs to be investigated.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12877) RestLookupService - supports Sensitive Dynamic Properties
[ https://issues.apache.org/jira/browse/NIFI-12877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lehel Boér updated NIFI-12877: -- Status: Patch Available (was: In Progress) > RestLookupService - supports Sensitive Dynamic Properties > - > > Key: NIFI-12877 > URL: https://issues.apache.org/jira/browse/NIFI-12877 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 2.0.0-M2 >Reporter: Roman >Assignee: Lehel Boér >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > The RestLookupService service does not provide the possibility to support > Sensitive Dynamic Properties: Yes. > When we add a header in InvokeHTTP processor we can set "Sensitive Value" > Yes/No, however during adding dynamic values (tokens) in RestLookupService > there is no way to make it sensitive it is grayed out. > Desired improvement: > RestLookupService > *Dynamic Properties:* > Supports Sensitive Dynamic Properties: *Yes* -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] NIFI-12877: RestLookUpService support sensitive dynamic properties [nifi]
Lehel44 opened a new pull request, #8525: URL: https://github.com/apache/nifi/pull/8525 # Summary [NIFI-12877](https://issues.apache.org/jira/browse/NIFI-12877) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [ ] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [ ] Pull Request based on current revision of the `main` branch - [ ] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12892: Use Kerberos user principal as isolation key instead of s… [nifi]
Lehel44 commented on code in PR #8497:
URL: https://github.com/apache/nifi/pull/8497#discussion_r1527391531
##
nifi-nar-bundles/nifi-iceberg-bundle/nifi-iceberg-common/src/main/java/org/apache/nifi/processors/iceberg/AbstractIcebergProcessor.java:
##
@@ -124,10 +124,16 @@ public void onTrigger(ProcessContext context,
ProcessSession session) throws Pro
@Override
public String getClassloaderIsolationKey(PropertyContext context) {
-final KerberosUserService kerberosUserService =
context.getProperty(KERBEROS_USER_SERVICE).asControllerService(KerberosUserService.class);
-if (kerberosUserService != null) {
-return kerberosUserService.getIdentifier();
+try {
+final KerberosUserService kerberosUserService =
context.getProperty(KERBEROS_USER_SERVICE).asControllerService(KerberosUserService.class);
+if (kerberosUserService != null) {
+final KerberosUser kerberosUser =
kerberosUserService.createKerberosUser();
+return kerberosUser.getPrincipal();
+}
+} catch (IllegalStateException e) {
Review Comment:
Maybe catching a wider range of Exception here would be beneficial unless we
are sure inly IllegalException can be thrown.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-12892: Use Kerberos user principal as isolation key instead of s… [nifi]
Lehel44 commented on code in PR #8497:
URL: https://github.com/apache/nifi/pull/8497#discussion_r1527391674
##
nifi-nar-bundles/nifi-iceberg-bundle/nifi-iceberg-common/src/main/java/org/apache/nifi/processors/iceberg/AbstractIcebergProcessor.java:
##
@@ -124,10 +124,16 @@ public void onTrigger(ProcessContext context,
ProcessSession session) throws Pro
@Override
public String getClassloaderIsolationKey(PropertyContext context) {
-final KerberosUserService kerberosUserService =
context.getProperty(KERBEROS_USER_SERVICE).asControllerService(KerberosUserService.class);
-if (kerberosUserService != null) {
-return kerberosUserService.getIdentifier();
+try {
+final KerberosUserService kerberosUserService =
context.getProperty(KERBEROS_USER_SERVICE).asControllerService(KerberosUserService.class);
+if (kerberosUserService != null) {
+final KerberosUser kerberosUser =
kerberosUserService.createKerberosUser();
+return kerberosUser.getPrincipal();
+}
+} catch (IllegalStateException e) {
+return null;
Review Comment:
```suggestion
getLogger().error("Failed to determine ClassLoader
Isolation Key for " + this + ". This could result in unexpected behavior by
this processor.", e);
return null;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-12889: Retry Kerberos login on auth failure in HDFS processors [nifi]
Lehel44 commented on code in PR #8495:
URL: https://github.com/apache/nifi/pull/8495#discussion_r1527388335
##
nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java:
##
@@ -372,54 +370,56 @@ public Object run() {
// Write FlowFile to temp file on HDFS
final StopWatch stopWatch = new StopWatch(true);
-session.read(putFlowFile, new InputStreamCallback() {
-
-@Override
-public void process(InputStream in) throws IOException
{
-OutputStream fos = null;
-Path createdFile = null;
-try {
-if (conflictResponse.equals(APPEND_RESOLUTION)
&& destinationExists) {
-fos = hdfs.append(copyFile, bufferSize);
-} else {
-final EnumSet cflags =
EnumSet.of(CreateFlag.CREATE, CreateFlag.OVERWRITE);
-
-if (shouldIgnoreLocality(context,
session)) {
-
cflags.add(CreateFlag.IGNORE_CLIENT_LOCALITY);
-}
+session.read(putFlowFile, in -> {
+OutputStream fos = null;
+Path createdFile = null;
+try {
+if (conflictResponse.equals(APPEND_RESOLUTION) &&
destinationExists) {
+fos = hdfs.append(copyFile, bufferSize);
+} else {
+final EnumSet cflags =
EnumSet.of(CreateFlag.CREATE, CreateFlag.OVERWRITE);
-fos = hdfs.create(actualCopyFile,
FsCreateModes.applyUMask(FsPermission.getFileDefault(),
-
FsPermission.getUMask(hdfs.getConf())), cflags, bufferSize, replication,
blockSize,
-null, null);
+if (shouldIgnoreLocality(context, session)) {
+
cflags.add(CreateFlag.IGNORE_CLIENT_LOCALITY);
}
-if (codec != null) {
-fos = codec.createOutputStream(fos);
+fos = hdfs.create(actualCopyFile,
FsCreateModes.applyUMask(FsPermission.getFileDefault(),
+
FsPermission.getUMask(hdfs.getConf())), cflags, bufferSize, replication,
blockSize,
+null, null);
+}
+
+if (codec != null) {
+fos = codec.createOutputStream(fos);
+}
+createdFile = actualCopyFile;
+BufferedInputStream bis = new
BufferedInputStream(in);
+StreamUtils.copy(bis, fos);
+bis = null;
+fos.flush();
+} catch (IOException e) {
Review Comment:
Yeah I think it's fine. If executing the mentioned lines does not cause
discrepancy, I prefer the simplicity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[PR] Bump follow-redirects from 1.15.5 to 1.15.6 in /nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-frontend/src/main/nifi [nifi]
dependabot[bot] opened a new pull request, #8524: URL: https://github.com/apache/nifi/pull/8524 Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.15.5 to 1.15.6. Commits https://github.com/follow-redirects/follow-redirects/commit/35a517c5861d79dc8bff7db8626013d20b711b06";>35a517c Release version 1.15.6 of the npm package. https://github.com/follow-redirects/follow-redirects/commit/c4f847f85176991f95ab9c88af63b1294de8649b";>c4f847f Drop Proxy-Authorization across hosts. https://github.com/follow-redirects/follow-redirects/commit/8526b4a1b2ab3a2e4044299377df623a661caa76";>8526b4a Use GitHub for disclosure. See full diff in https://github.com/follow-redirects/follow-redirects/compare/v1.15.5...v1.15.6";>compare view [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/nifi/network/alerts). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Bump follow-redirects from 1.15.4 to 1.15.6 in /nifi-registry/nifi-registry-core/nifi-registry-web-ui/src/main [nifi]
dependabot[bot] opened a new pull request, #8523: URL: https://github.com/apache/nifi/pull/8523 Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.15.4 to 1.15.6. Commits https://github.com/follow-redirects/follow-redirects/commit/35a517c5861d79dc8bff7db8626013d20b711b06";>35a517c Release version 1.15.6 of the npm package. https://github.com/follow-redirects/follow-redirects/commit/c4f847f85176991f95ab9c88af63b1294de8649b";>c4f847f Drop Proxy-Authorization across hosts. https://github.com/follow-redirects/follow-redirects/commit/8526b4a1b2ab3a2e4044299377df623a661caa76";>8526b4a Use GitHub for disclosure. https://github.com/follow-redirects/follow-redirects/commit/b1677ce00110ee50dc5da576751d39b281fc4944";>b1677ce Release version 1.15.5 of the npm package. https://github.com/follow-redirects/follow-redirects/commit/d8914f7982403ea096b39bd594a00ee9d3b7e224";>d8914f7 Preserve fragment in responseUrl. See full diff in https://github.com/follow-redirects/follow-redirects/compare/v1.15.4...v1.15.6";>compare view [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/nifi/network/alerts). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12880 Add processor DeleteFile [nifi]
EndzeitBegins commented on PR #8489: URL: https://github.com/apache/nifi/pull/8489#issuecomment-2001954793 The frontend build failed again without related changes. :( Would you be so kind to kick off the CI process once more @ChrisSamo632? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (NIFI-12893) Adjust documentation on time based properties in DBCPProperties
[
https://issues.apache.org/jira/browse/NIFI-12893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
endzeit resolved NIFI-12893.
Fix Version/s: 2.0.0
1.26.0
Resolution: Fixed
> Adjust documentation on time based properties in DBCPProperties
> ---
>
> Key: NIFI-12893
> URL: https://issues.apache.org/jira/browse/NIFI-12893
> Project: Apache NiFi
> Issue Type: Task
>Reporter: endzeit
>Assignee: endzeit
>Priority: Major
> Fix For: 2.0.0, 1.26.0
>
>
> Some of the properties in {{DBCPProperties}}, like {{MAX_CONN_LIFETIME}},
> note that the time has to be defined in milliseconds.
> However, this is not true as an time period like "5 mins" is expected.
> The documentation should be adjusted accordingly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12900) Avoid unnecessary directory listing in PutSFTP
[
https://issues.apache.org/jira/browse/NIFI-12900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
endzeit updated NIFI-12900:
---
Summary: Avoid unnecessary directory listing in PutSFTP (was: Avoid
unnecessary file listing in PutSFTP )
> Avoid unnecessary directory listing in PutSFTP
> ---
>
> Key: NIFI-12900
> URL: https://issues.apache.org/jira/browse/NIFI-12900
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: endzeit
>Assignee: endzeit
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> The processor {{PutSFTP}} is based on {{PutFileTransfer}}.
> Before an actual upload takes place, potential conflicts (e.g. existing file)
> are identified and resolved using {{identifyAndResolveConflictFile(...)}}.
> As part of this process, information on the target file is retrieved using
> {{FileTransfer.getRemoteFileInfo(...)}}.
> In case of {{PutSFTP}} this is implemented by {{SFTPTransfer}}.
> The implementation of {{getRemoteFileInfo}} executes {{ls}} on the target
> directory path. In case there are a lot of files inside the remote directory,
> e.g. >10.000 files, the listing reduces the performance of {{PutSFTP}}
> significantly.
> Instead of a listing on the directory, file information should be retrieved
> using either {{ls}} or {{stat}} on the target file directly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
