Re: [PR] NIFI-15209 JoltTransformRecord should not only take schema from first record [nifi]
pvillard31 commented on PR #10545: URL: https://github.com/apache/nifi/pull/10545#issuecomment-4217717058 @umairtum - it'll be in NiFi 2.9.0 for which the vote for the release candidate is in progress and should end tomorrow unless an issue is found by then -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] NIFI-15209 JoltTransformRecord should not only take schema from first record [nifi]
umairtum commented on PR #10545: URL: https://github.com/apache/nifi/pull/10545#issuecomment-4217514505 Is this bug fixed? if yes in which version? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] NIFI-15209 JoltTransformRecord should not only take schema from first record [nifi]
sammu97 commented on PR #10545: URL: https://github.com/apache/nifi/pull/10545#issuecomment-4053265615 > > @exceptionfactory That solution works for me to be honest. Just one thing if we go this route though, I do still believe that this behaviour with the JoltTransformRecord should be documented somewhere, so that users are aware of it. What do you think? > > Thanks for the confirmation, and for your work in this pull request @sammu97. Yes, I agree it would be helpful to document the current behavior of `JoltTransformRecord` as it stands. The documentation for the Record Reader property mentions something about the record schema, so updating that property documentation in a separate pull request seems reasonable. > > To address the primary goal, I submitted pull request #11001 for [NIFI-15712](https://issues.apache.org/jira/browse/NIFI-15712) adding support for JSON Lines/NDJSON to the `JoltTransformJSON` Processor. After some minor refactoring, it was a straightforward addition, which sounds like it should fit the use case of widely varying JSON elements, avoiding the record schema challenges. Awesome, thanks @exceptionfactory for the solution! Looking forward to having this deployed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] NIFI-15209 JoltTransformRecord should not only take schema from first record [nifi]
exceptionfactory commented on PR #10545: URL: https://github.com/apache/nifi/pull/10545#issuecomment-4051884405 > @exceptionfactory That solution works for me to be honest. Just one thing if we go this route though, I do still believe that this behaviour with the JoltTransformRecord should be documented somewhere, so that users are aware of it. What do you think? Thanks for the confirmation, and for your work in this pull request @sammu97. Yes, I agree it would be helpful to document the current behavior of `JoltTransformRecord` as it stands. The documentation for the Record Reader property mentions something about the record schema, so updating that property documentation in a separate pull request seems reasonable. To address the primary goal, I submitted pull request #11001 for [NIFI-15712](https://issues.apache.org/jira/browse/NIFI-15712) adding support for JSON Lines/NDJSON to the `JoltTransformJSON` Processor. After some minor refactoring, it was a straightforward addition, which sounds like it should fit the use case of widely varying JSON elements, avoiding the record schema challenges. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] NIFI-15209 JoltTransformRecord should not only take schema from first record [nifi]
exceptionfactory closed pull request #10545: NIFI-15209 JoltTransformRecord should not only take schema from first record URL: https://github.com/apache/nifi/pull/10545 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] NIFI-15209 JoltTransformRecord should not only take schema from first record [nifi]
exceptionfactory commented on PR #10545: URL: https://github.com/apache/nifi/pull/10545#issuecomment-4051857971 > Correct me if I am wrong but this or a similar conversation seems to have taken place on [NIFI-14309](https://issues.apache.org/jira/browse/NIFI-14309) @dan-s1, yes, this relates to the discussion on NIFI-14309, although that issue refers to the JSLT Processor, not the Jolt Processor. This discussion on NDJSON handling highlights a use case gap with record-oriented processing, so it seems best to introduce something more specific in this case for Jolt. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] NIFI-15209 JoltTransformRecord should not only take schema from first record [nifi]
dan-s1 commented on PR #10545: URL: https://github.com/apache/nifi/pull/10545#issuecomment-4050234022 Correct me if I am wrong but this or a similar conversation seems to have taken on [NIFI-14309](https://issues.apache.org/jira/browse/NIFI-14309) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] NIFI-15209 JoltTransformRecord should not only take schema from first record [nifi]
sammu97 commented on PR #10545: URL: https://github.com/apache/nifi/pull/10545#issuecomment-4042786183 @exceptionfactory That solution works for me to be honest. Just one thing if we go this route though, I do still believe that this behaviour with the JoltTransformRecord should be documented somewhere, so that users are aware of it. What do you think? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] NIFI-15209 JoltTransformRecord should not only take schema from first record [nifi]
exceptionfactory commented on PR #10545: URL: https://github.com/apache/nifi/pull/10545#issuecomment-4039337027 Thanks for the reply @sammu97. The source data being NDJSON is an important detail and may lead to a different solution. Right now, the JoltTransformJSON Processor works with either the entire FlowFile, or with Attributes, neither of which align with NDJSON. However, if that Processor were changed to support handling NDJSON, applying the configured transform to each line of NDJSON, it sounds like that might be a better fit, avoiding any kind of schema inference issues. I might be able to put something together if that seems like a potential solution. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] NIFI-15209 JoltTransformRecord should not only take schema from first record [nifi]
sammu97 commented on PR #10545: URL: https://github.com/apache/nifi/pull/10545#issuecomment-4037494145 Hi @exceptionfactory, you're right about the formatting. Will revert any unnecessary ones, apologies. Regarding it's intended use, we are specifically making use of this with the JoltTransformRecord to transform ND Json content. I agree the solution turned out a bit complex, though I am not sure on the alternatives so if you have any ideas please go ahead. My primary concern here is that this issue is not easily noticeable and can do a lot of damage before it is noticed when making use of this processor. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] NIFI-15209 JoltTransformRecord should not only take schema from first record [nifi]
sammu97 commented on PR #10545: URL: https://github.com/apache/nifi/pull/10545#issuecomment-4022593014 Hi @exceptionfactory , was wondering if you could take a look at this PR when possible please? Would love to have this fixed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] NIFI-15209 JoltTransformRecord should not only take schema from first record [nifi]
sammu97 commented on PR #10545: URL: https://github.com/apache/nifi/pull/10545#issuecomment-3730034484 Hey @exceptionfactory, I've resolved some conflicts due to the PR being stale, and also made some fixes relating to issues where records with the same schema were still being partitioned due field ordering. Let me know what you think about the change please -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] NIFI-15209 JoltTransformRecord should not only take schema from first record [nifi]
sammu97 commented on PR #10545: URL: https://github.com/apache/nifi/pull/10545#issuecomment-3570655500 @exceptionfactory Just a small note too. I've also amended some logic for the testNoRecords() test, as I have put out a small change that if the Jolt has no records to transform, in my opinion there should be no resulting flowfile as there is nothing to write. Not sure what you think about this, should I be leaving the old logic? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] NIFI-15209 JoltTransformRecord should not only take schema from first record [nifi]
sammu97 commented on PR #10545:
URL: https://github.com/apache/nifi/pull/10545#issuecomment-3569549917
> > @exceptionfactory Had to make some fixes for Windows as the checks are
usually omitted. However, any idea about the error for the Mac tests?
> > `The template is not valid. .github/workflows/ci-workflow.yml (Line:
224, Col: 16): hashFiles('**/package-lock.json') failed. Fail to hash files
under directory '/Users/runner/work/nifi/nifi'`
>
> @sammu97 the node cache issue in the build appears to have been an
intermittent problem over the weekend. I spotted other PRs with similar errors,
but then things seem to be working again this morning. I've restarted the
failed job on your PR and so far things like happier 🤞
@ChrisSamo632 Yep, seems like it's already past the step that was failing.
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] NIFI-15209 JoltTransformRecord should not only take schema from first record [nifi]
ChrisSamo632 commented on PR #10545:
URL: https://github.com/apache/nifi/pull/10545#issuecomment-3569513453
> @exceptionfactory Had to make some fixes for Windows as the checks are
usually omitted. However, any idea about the error for the Mac tests?
>
> `The template is not valid. .github/workflows/ci-workflow.yml (Line: 224,
Col: 16): hashFiles('**/package-lock.json') failed. Fail to hash files under
directory '/Users/runner/work/nifi/nifi'`
@sammu97 the node cache issue in the build appears to have been an
intermittent problem over the weekend. I spotted other PRs with similar errors,
but then things seem to be working again this morning. I've restarted the
failed job on your PR and so far things like happier 🤞
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] NIFI-15209 JoltTransformRecord should not only take schema from first record [nifi]
sammu97 commented on PR #10545:
URL: https://github.com/apache/nifi/pull/10545#issuecomment-3568033063
@exceptionfactory Had to make some fixes for Windows as the checks are
usually omitted. However, any idea about the error for the Mac tests?
`The template is not valid. .github/workflows/ci-workflow.yml (Line: 224,
Col: 16): hashFiles('**/package-lock.json') failed. Fail to hash files under
directory '/Users/runner/work/nifi/nifi'`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] NIFI-15209 JoltTransformRecord should not only take schema from first record [nifi]
sammu97 commented on PR #10545: URL: https://github.com/apache/nifi/pull/10545#issuecomment-3552559496 Looks like the build failed on some of the OSs, im suspecting a file ordering issue. Will investigate and update the PR accordingly -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] NIFI-15209 JoltTransformRecord should not only take schema from first record [nifi]
exceptionfactory commented on PR #10545: URL: https://github.com/apache/nifi/pull/10545#issuecomment-3551712550 > Yes sure @exceptionfactory , will handle this as soon as i can. > > Also, I'm seeing that some checks are failing on code checkout, is this due to the Cloudflare outage? Yes, they were due to the outage, I have restarted the checks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] NIFI-15209 JoltTransformRecord should not only take schema from first record [nifi]
sammu97 commented on PR #10545: URL: https://github.com/apache/nifi/pull/10545#issuecomment-3551696671 Yes sure @exceptionfactory , will handle this as soon as i can. Also, I'm seeing that some checks are failing on code checkout, is this due to the Cloudflare outage? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
