[GitHub] nifi issue #2082: NIFI-2829: Fixed PutSQL time unit test.
Github user yjhyjhyjh0 commented on the issue:
https://github.com/apache/nifi/pull/2082
Thanks for the detail explanation.
Indeed using local time zone is more reasonable.
Didnât notice the date will be transform into local time zone once with
value format.
Just want to point out that I first try the approach you mentioned (use
local time zone) will encounter problem with Time type.
My time zone is UTC+8, which cause time string â01:01:01â in Time type
will become negative long value.
This reasonable because Time type only consider time without epoch so it
will be negative if itâs earlier than â08:00:00â.
However this will not pass the LONG_PATTERN in PutSQL and thus consider a
negative long value to be in format "HH:mm:ss.SSSâ instead of long.
This will encounter error while parsing for sure :
java.text.ParseException: Unparseable date: â-28739000â.
If I try your commit with changing the timeStr to â01:01:01" in
testUsingDateTimeValuesWithFormatAttribute will encounter negative long parsing
problem which mentioned above.
Donât know if you will encounter similar problem with **negative long
value in Time type**, too?
I've try your commit together with LONG_PATTERN that also match negative
long value ("^-?\\d{1,19}$â) today and everything works fine now.
Thanks for any response.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1983: NiFi-2829: Add Date and Time Format Support for PutSQL
Github user yjhyjhyjh0 commented on the issue: https://github.com/apache/nifi/pull/1983 Thanks for the reply and detail review. Just update the commit title, document part and remove unnecessary if condition. Seems AppVevor fail at same part. Travis CI only fail at build 2 but pass build4,5. Not sure why. I'll keep track of it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1983: NiFi-2829: Add Date and Time Format Support for PutSQL
Github user yjhyjhyjh0 commented on the issue: https://github.com/apache/nifi/pull/1983 Thanks for the review and feedback. Iâve add your enhancement of handing time precision at milliseconds in PutSQL. Also I'm very willing to add a unit test for it. However after some experiment, I figure out Derby DB has same behavior as MySQL in Time type just like what you mentioned. Iâve tested with derby, MySQL and PostgreSQL. Only PostgreSQL will handle millisecond in Time type properly. About the experiment, it store Time type well with value "00:01:01.111" in PutSQL. But once client fetch via jdbc of Derby. It truncate the milliseconds. (actually has different long value in client side) (java jdbc client and IJ tool of derby) So even with correct format to parse, the result in client side is "00:01:01.000â. Thus, I only update document to remind this situation instead of providing a unit test for this format. Please let me know if there is any problem. Thanks for your patient. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1983: NiFi-2829: Add Date and Time Format Support for PutSQL
Github user yjhyjhyjh0 commented on the issue: https://github.com/apache/nifi/pull/1983 Well, lesson learned. Last build fail with Travis-CI is trailing whitespace problem. Just fix that and retry the PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1983: NiFi-2829: Add Date and Time Format Support for Put...
GitHub user yjhyjhyjh0 reopened a pull request: https://github.com/apache/nifi/pull/1983 NiFi-2829: Add Date and Time Format Support for PutSQL Fix unit test for Date and Time type time zone problem while testing PutSQL processor @paulgibeault made the original PR #1073, #1468 @patricker add support of **DATE** and **TIME** in Epoch format for PutSQL processor. Iâve fix the unit test in different time zone problem. The detail is list below The originally problem with unit test happens because of different time zone. Internally without specifying time zone, java.sql.Date and java.sql.Time will use local time zone to parse the time. As a result, different time zone will have different format result for a given constant time value. This is mentioned by @mattyb149 in https://github.com/apache/nifi/pull/1524 Currently solve the problem by giving time zone before insert and parse result with same time zone. (GMT) Currently build and test successfully with NiFi newest version on GitHub which is 1.4.0-SNAPSHOT. ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/yjhyjhyjh0/nifi NIFI-2829 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1983.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1983 commit 1808ddbcfd68f7197446eb566b53076593ebdde5 Author: deonhuang <deonhu...@micron.com> Date: 2017-07-03T06:00:22Z NiFi-2829: Add Date and Time Format Support for PutSQL Fix unit test for Date and Time type time zone problem while testing PutSQL processor --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1983: NiFi-2829: Add Date and Time Format Support for Put...
Github user yjhyjhyjh0 closed the pull request at: https://github.com/apache/nifi/pull/1983 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1983: NiFi-2829: Add Date and Time Format Support for Put...
GitHub user yjhyjhyjh0 opened a pull request: https://github.com/apache/nifi/pull/1983 NiFi-2829: Add Date and Time Format Support for PutSQL Fix unit test for Date and Time type time zone problem while testing PutSQL processor @paulgibeault made the original PR #1073, #1468 @patricker add support of **DATE** and **TIME** in Epoch format for PutSQL processor. Iâve fix the unit test in different time zone problem. The detail is list below The originally problem with unit test happens because of different time zone. Internally without specifying time zone, java.sql.Date and java.sql.Time will use local time zone to parse the time. As a result, different time zone will have different format result for a given constant time value. This is mentioned by @mattyb149 in https://github.com/apache/nifi/pull/1524 Currently solve the problem by giving time zone before insert and parse result with same time zone. (GMT) Currently build and test successfully with NiFi newest version on GitHub which is 1.4.0-SNAPSHOT. ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/yjhyjhyjh0/nifi NIFI-2829 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1983.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1983 commit c8e916df1ba39e6be52ae9274804740eba1d4df4 Author: deonhuang <deonhu...@micron.com> Date: 2017-07-03T06:00:22Z NiFi-2829: Add Date and Time Format Support for PutSQL Fix unit test for Date and Time type time zone problem while testing PutSQL processor --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2166: NIFI-4395 - GenerateTableFetch can't fetch column type by ...
Github user yjhyjhyjh0 commented on the issue: https://github.com/apache/nifi/pull/2166 Thanks for the detail comment and suggestion. That helps me a lot. Some fix by suggestion like. 1 - put back StringmaxValueColumnNames to GenerateTableFetch to avoid NPE unit test fail. (Originally forget to include this line) 2 - move re-cache method, calling super.setup(), before the getColumnType to provide more readable code. 3 - Remove ternary operator because evaluateAttributeExpressions already can handle null flowfile. Thanks ---
[GitHub] nifi issue #2166: NIFI-4395 - GenerateTableFetch can't fetch column type by ...
Github user yjhyjhyjh0 commented on the issue: https://github.com/apache/nifi/pull/2166 Not quite understand result from travis-ci. Seems it stuck at ANTLR Parser AttributeExpressionParser every time? ---
[GitHub] nifi pull request #2166: NIFI-4395 - GenerateTableFetch can't fetch column t...
GitHub user yjhyjhyjh0 opened a pull request: https://github.com/apache/nifi/pull/2166 NIFI-4395 - GenerateTableFetch can't fetch column type by state after⦠⦠instance reboot Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/yjhyjhyjh0/nifi NIFI-4395 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2166.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2166 commit faf6f2dfc15ef1900b6b868e19b90e39d934d07f Author: Deon Huang <yjhyjhy...@gmail.com> Date: 2017-09-21T07:34:00Z NIFI-4395 - GenerateTableFetch can't fetch column type by state after instance reboot ---
[GitHub] nifi issue #2166: NIFI-4395 - GenerateTableFetch can't fetch column type by ...
Github user yjhyjhyjh0 commented on the issue: https://github.com/apache/nifi/pull/2166 I've solved this issue by re-cache column type if needed and pass local unit and instance test. The root cause of this issue is that originally disign didn't recognize that column type cache will be clean after instance reboot. This cause later unable to find column type cache with existing state value and throw unhandle IllegalArgumentException. Now if state value exist without corresponding column type cache, we will fetch and cache it. ---
[GitHub] nifi pull request #2442: NIFI-4395 GenerateTableFetch can't fetch column typ...
GitHub user yjhyjhyjh0 opened a pull request: https://github.com/apache/nifi/pull/2442 NIFI-4395 GenerateTableFetch can't fetch column type by state after i⦠â¦nstance reboot Add multiple states recover Add support for multiple state cache recover after instance reboot. Current fix by add condition for column type not found to re-cache. Also add unit test for multiple state recover situation. ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/yjhyjhyjh0/nifi NIFI-4395 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2442.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2442 commit 4a3a212d218cf524f00f01d18f4e9d9fdf9ed403 Author: Deon Huang <yjhyjhyjh0@...> Date: 2018-01-29T15:29:21Z NIFI-4395 GenerateTableFetch can't fetch column type by state after instance reboot Add multiple states recover ---
[GitHub] nifi pull request #2530: NIFI-4800 Expose the flattenMode as property in Fla...
GitHub user yjhyjhyjh0 opened a pull request: https://github.com/apache/nifi/pull/2530 NIFI-4800 Expose the flattenMode as property in FlattenJSON processor Expose all 3 kinds of flatten mode. Documentation along with unit tests. Finish local nifi instance integration test. ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/yjhyjhyjh0/nifi NIFI-4800 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2530.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2530 commit d78cf4e088b075feef17c515cb3577b3198b1eaf Author: Deon Huang <yjhyjhyjh0@...> Date: 2018-03-11T15:01:53Z NIFI-4800 Expose the flattenMode as property in FlattenJSON processor ---
[GitHub] nifi issue #2530: NIFI-4800 Expose the flattenMode as property in FlattenJSO...
Github user yjhyjhyjh0 commented on the issue: https://github.com/apache/nifi/pull/2530 Thanks for the review. I've added display name as most processor does and removed unnecessary line at unit tests. ---
[GitHub] nifi pull request #2538: NIFI-4962 FlattenJson processor add unexpected back...
GitHub user yjhyjhyjh0 opened a pull request: https://github.com/apache/nifi/pull/2538 NIFI-4962 FlattenJson processor add unexpected backslash after flatten Upgrade json flatten library version to newest one to deal with StringEscapePolicy. Add unit test code to verify solution. ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/yjhyjhyjh0/nifi NIFI-4962 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2538.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2538 commit b6215f866e93d4065277a8e763bfa9127be972ed Author: Deon Huang <yjhyjhyjh0@...> Date: 2018-03-13T16:04:29Z NIFI-4962 FlattenJson processor add unexpected backslash after flatten ---
[GitHub] nifi pull request #3107: NIFI-5744: Put exception message to attribute while...
GitHub user yjhyjhyjh0 opened a pull request: https://github.com/apache/nifi/pull/3107 NIFI-5744: Put exception message to attribute while ExecuteSQL fail Put exception message to flowfile attribute if exception occur. Add unit test to ExecuteSQL, ExecuteSQLRecord. Update document annotation. Similar exception catching also implement in other processors like InvokeHTTP, GenerateTableFetch. ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/yjhyjhyjh0/nifi NIFI-5744 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3107.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3107 ---
[GitHub] nifi pull request #3107: NIFI-5744: Put exception message to attribute while...
Github user yjhyjhyjh0 commented on a diff in the pull request: https://github.com/apache/nifi/pull/3107#discussion_r230031051 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteSQL.java --- @@ -83,6 +83,8 @@ @WritesAttribute(attribute = "executesql.query.fetchtime", description = "Duration of the result set fetch time in milliseconds"), @WritesAttribute(attribute = "executesql.resultset.index", description = "Assuming multiple result sets are returned, " + "the zero based index of this result set."), +@WritesAttribute(attribute = "executesql.error.message", description = "If processing an incoming flow file causes " ++ "an Exception, the flow file is routed to failure and this attribute is set to the exception message."), --- End diff -- Sure, thanks for the correction. ---
[GitHub] nifi issue #3107: NIFI-5744: Put exception message to attribute while Execut...
Github user yjhyjhyjh0 commented on the issue: https://github.com/apache/nifi/pull/3107 Thanks for the remind, there are some nice discussions in the mailing list. I'll fix capitalization after we have consensus. ---
[GitHub] nifi pull request #3156: NIFI-5780 Add pre and post statements to ExecuteSQL...
Github user yjhyjhyjh0 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3156#discussion_r231754034
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestExecuteSQLRecord.java
---
@@ -350,6 +352,141 @@ public void invokeOnTriggerRecords(final Integer
queryTimeout, final String quer
assertEquals(durationTime, fetchTime + executionTime);
}
+@Test
+public void testPreQuery() throws Exception {
+// remove previous test database, if any
+final File dbLocation = new File(DB_LOCATION);
+dbLocation.delete();
+
+// load test data to database
+final Connection con = ((DBCPService)
runner.getControllerService("dbcp")).getConnection();
+Statement stmt = con.createStatement();
+
+try {
+stmt.execute("drop table TEST_NULL_INT");
+} catch (final SQLException sqle) {
+}
+
+stmt.execute("create table TEST_NULL_INT (id integer not null,
val1 integer, val2 integer, constraint my_pk primary key (id))");
+
+runner.setIncomingConnection(true);
+runner.setProperty(ExecuteSQL.SQL_PRE_QUERY, "insert into
TEST_NULL_INT values(1,2,3);insert into TEST_NULL_INT values(4,5,6)");
--- End diff --
That's indeed great use case, I'll update test cases.
Thanks for the sharing the information.
---
[GitHub] nifi pull request #3156: NIFI-5780 Add pre and post statements to ExecuteSQL...
Github user yjhyjhyjh0 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3156#discussion_r231753838
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractExecuteSQL.java
---
@@ -82,6 +84,16 @@
.identifiesControllerService(DBCPService.class)
.build();
+public static final PropertyDescriptor SQL_PRE_QUERY = new
PropertyDescriptor.Builder()
+.name("sql-pre-query")
+.displayName("SQL pre-query")
--- End diff --
Hi Peter, thanks for the feedback.
I intentionally type it in 'SQL pre-query' to align with SQL_SELECT_QUERY
property displayName 'SQL select query'.
For example currently,
SQL_PRE_QUERY, display '**SQL pre-query**'
SQL_SELECT_QUERY, display '**SQL select query**'
SQL_POST_QUERY, display '**SQL post-query**'
If modified as you mention (I actually try this version in the first try)
For example,
SQL_PRE_QUERY, display '**SQL Pre-Query**'
SQL_SELECT_QUERY, display '**SQL select query**'
SQL_POST_QUERY, display '**SQL Post-Query**'
Please let me know your though on this alignment, thanks
---
[GitHub] nifi pull request #3156: NIFI-5780 Add pre and post statements to ExecuteSQL...
Github user yjhyjhyjh0 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3156#discussion_r232293044
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractExecuteSQL.java
---
@@ -82,6 +84,16 @@
.identifiesControllerService(DBCPService.class)
.build();
+public static final PropertyDescriptor SQL_PRE_QUERY = new
PropertyDescriptor.Builder()
+.name("sql-pre-query")
+.displayName("SQL Pre-Query")
+.description("SQL pre-query to execute. Semicolon-delimited
list of queries. "
++ "Note, the results/outputs of these queries will be
suppressed if successfully executed.")
--- End diff --
Thanks for the suggestion, will omit it at next commit.
---
[GitHub] nifi pull request #3156: NIFI-5780 Add pre and post statements to ExecuteSQL...
Github user yjhyjhyjh0 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3156#discussion_r232292846
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractExecuteSQL.java
---
@@ -82,6 +84,16 @@
.identifiesControllerService(DBCPService.class)
.build();
+public static final PropertyDescriptor SQL_PRE_QUERY = new
PropertyDescriptor.Builder()
+.name("sql-pre-query")
+.displayName("SQL Pre-Query")
+.description("SQL pre-query to execute. Semicolon-delimited
list of queries. "
--- End diff --
Sounds great to me.
I'll also modify Hive description and squash the commit.
---
[GitHub] nifi issue #3156: NIFI-5780 Add pre and post statements to ExecuteSQL and Ex...
Github user yjhyjhyjh0 commented on the issue: https://github.com/apache/nifi/pull/3156 Thanks for the suggestion. Update description to be more precise in ExecuteSQL, ExecuteSQLRecord, SelectHiveQL. Squash the this commit. ---
[GitHub] nifi pull request #3156: NIFI-5780 Add pre and post statements to ExecuteSQL...
GitHub user yjhyjhyjh0 opened a pull request: https://github.com/apache/nifi/pull/3156 NIFI-5780 Add pre and post statements to ExecuteSQL and ExecuteSQLRecord Add pre, post query property to AbstractExecuteSQL. Most of implementation comes from SelectHiveQL. Add unit test to pre, post query. Finish local nifi integration test. ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [x] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/yjhyjhyjh0/nifi NIFI-5780 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3156.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3156 commit e5ec707036b04607217c276c616c145aad54df4c Author: yjhyjhyjh0 Date: 2018-11-07T16:25:50Z NIFI-5780 Add pre and post statements to ExecuteSQL and ExecuteSQLRecord ---
[GitHub] nifi issue #3107: NIFI-5744: Put exception message to attribute while Execut...
Github user yjhyjhyjh0 commented on the issue: https://github.com/apache/nifi/pull/3107 Thanks for the suggestion. Rebase against master, solve conflicts and squash into single commit. ---
[GitHub] nifi pull request #3107: NIFI-5744: Put exception message to attribute while...
Github user yjhyjhyjh0 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3107#discussion_r234407245
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestExecuteSQLRecord.java
---
@@ -350,6 +357,37 @@ public void invokeOnTriggerRecords(final Integer
queryTimeout, final String quer
assertEquals(durationTime, fetchTime + executionTime);
}
+@SuppressWarnings("unchecked")
+@Test
+public void testWithSqlExceptionErrorProcessingResultSet() throws
Exception {
+DBCPService dbcp = mock(DBCPService.class);
+Connection conn = mock(Connection.class);
+when(dbcp.getConnection(any(Map.class))).thenReturn(conn);
+when(dbcp.getIdentifier()).thenReturn("mockdbcp");
+PreparedStatement statement = mock(PreparedStatement.class);
+when(conn.prepareStatement(anyString())).thenReturn(statement);
+when(statement.execute()).thenReturn(true);
+ResultSet rs = mock(ResultSet.class);
+when(statement.getResultSet()).thenReturn(rs);
+// Throw an exception the first time you access the ResultSet,
this is after the flow file to hold the results has been created.
+when(rs.getMetaData()).thenThrow(new SQLException("test execute
statement failed"));
+
--- End diff --
You're right, thanks for point out and suggestion.
I should be more cautious while resolving commit.
I've add lines of mock record writer back and test successfully.
Squashed and pushed.
---
