[jira] [Commented] (BEAM-1310) Add running integration tests for JdbcIO
[ https://issues.apache.org/jira/browse/BEAM-1310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15902206#comment-15902206 ] ASF GitHub Bot commented on BEAM-1310: -- Github user asfgit closed the pull request at: https://github.com/apache/beam/pull/2174 > Add running integration tests for JdbcIO > > > Key: BEAM-1310 > URL: https://issues.apache.org/jira/browse/BEAM-1310 > Project: Beam > Issue Type: Bug > Components: sdk-java-extensions >Reporter: Stephen Sisk >Assignee: Stephen Sisk > Fix For: 0.6.0 > > > Jdbc IO could use some integration tests! We'd like to have them run against > a real list instance of postgres. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (BEAM-1310) Add running integration tests for JdbcIO
[ https://issues.apache.org/jira/browse/BEAM-1310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898457#comment-15898457 ] ASF GitHub Bot commented on BEAM-1310: -- GitHub user ssisk opened a pull request: https://github.com/apache/beam/pull/2174 [BEAM-1310] updates to JdbcIO k8 scripts & data loading Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [X] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [X] Replace `` in the title with the actual Jira issue number, if there is one. - [X] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- We have started setting up the k8 cluster & postgres instances, and I had a few small updates as I got things working. R @jbonofre cc @jasonkuster You can merge this pull request into a Git repository by running: $ git pull https://github.com/ssisk/beam jdbc-it-profiles Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/2174.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2174 commit a59aebad8481044714e0ed8532812de263793843 Author: Stephen Sisk Date: 2017-03-06T23:59:31Z Jdbc k8 & data loading: add teardown and update names/docs commit 156018bbdc7769de6095425504d38703a8e81762 Author: Stephen Sisk Date: 2017-03-07T00:01:26Z Jdbc k8 script: postgres data store only accessible inside test project > Add running integration tests for JdbcIO > > > Key: BEAM-1310 > URL: https://issues.apache.org/jira/browse/BEAM-1310 > Project: Beam > Issue Type: Bug > Components: sdk-java-extensions >Reporter: Stephen Sisk >Assignee: Stephen Sisk > Fix For: 0.6.0 > > > Jdbc IO could use some integration tests! We'd like to have them run against > a real list instance of postgres. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (BEAM-1310) Add running integration tests for JdbcIO
[ https://issues.apache.org/jira/browse/BEAM-1310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15893861#comment-15893861 ] ASF GitHub Bot commented on BEAM-1310: -- Github user asfgit closed the pull request at: https://github.com/apache/beam/pull/2090 > Add running integration tests for JdbcIO > > > Key: BEAM-1310 > URL: https://issues.apache.org/jira/browse/BEAM-1310 > Project: Beam > Issue Type: Bug > Components: sdk-java-extensions >Reporter: Stephen Sisk >Assignee: Stephen Sisk > > Jdbc IO could use some integration tests! We'd like to have them run against > a real list instance of postgres. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (BEAM-1310) Add running integration tests for JdbcIO
[ https://issues.apache.org/jira/browse/BEAM-1310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881455#comment-15881455 ] ASF GitHub Bot commented on BEAM-1310: -- GitHub user ssisk opened a pull request: https://github.com/apache/beam/pull/2090 [BEAM-1310] Jdbc IO module now can run ITs against spark & dataflow runners Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [X] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [X] Replace `` in the title with the actual Jira issue number, if there is one. - [X] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- I have added some changes that will allow us to start running the JdbcIO ITs with spark & dataflow. These 2 commits are deliberately separate since they're doing different things. I don't think it's worth separating out into different commits since both are moving towards the same goal of enabling JdbcIOIT to actually run in jenkins, and can be logically reviewed together. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ssisk/beam jdbc-it-profiles Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/2090.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2090 > Add running integration tests for JdbcIO > > > Key: BEAM-1310 > URL: https://issues.apache.org/jira/browse/BEAM-1310 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Stephen Sisk >Assignee: Stephen Sisk > > Jdbc IO could use some integration tests! We'd like to have them run against > a real list instance of postgres. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (BEAM-1310) Add running integration tests for JdbcIO
[ https://issues.apache.org/jira/browse/BEAM-1310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15864887#comment-15864887 ] ASF GitHub Bot commented on BEAM-1310: -- Github user asfgit closed the pull request at: https://github.com/apache/beam/pull/1841 > Add running integration tests for JdbcIO > > > Key: BEAM-1310 > URL: https://issues.apache.org/jira/browse/BEAM-1310 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Stephen Sisk >Assignee: Stephen Sisk > > Jdbc IO could use some integration tests! We'd like to have them run against > a real list instance of postgres. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (BEAM-1310) Add running integration tests for JdbcIO
[ https://issues.apache.org/jira/browse/BEAM-1310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15837042#comment-15837042 ] Stephen Sisk commented on BEAM-1310: Pull request: https://github.com/apache/beam/pull/1841 > Add running integration tests for JdbcIO > > > Key: BEAM-1310 > URL: https://issues.apache.org/jira/browse/BEAM-1310 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Stephen Sisk >Assignee: Stephen Sisk > > Jdbc IO could use some integration tests! We'd like to have them run against > a real list instance of postgres. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-1310) Add running integration tests for JdbcIO
[ https://issues.apache.org/jira/browse/BEAM-1310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15837034#comment-15837034 ] ASF GitHub Bot commented on BEAM-1310: -- GitHub user ssisk opened a pull request: https://github.com/apache/beam/pull/1841 BEAM-1310 Add integration tests for JdbcIO Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [X] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [X] Replace `` in the title with the actual Jira issue number, if there is one. - [X] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- Hi @tgroh can you please take a look? (2nd time!) This adds an integration test & data loading script for Jdbc IO You'll note there are TODOs for verifying the contents of the rows in the tests - I will get to those, but wanted to get a first, useful test in so that we can continue working on the end to end integration test infrastructure. This IT demonstrates: * Having a separate load script for read tests * Using TestPipelineOptions that are not test based, but rather data source based. * Writing kubernetes scripts for instantiating an instance of the data source. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ssisk/beam jdbc-it Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/1841.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1841 commit e6007397e7d5508a4dfe39cf95ad4e1ceaaa2d6e Author: Stephen Sisk Date: 2017-01-25T01:56:35Z Add pg IT, load script and k8 script for JdbcIO > Add running integration tests for JdbcIO > > > Key: BEAM-1310 > URL: https://issues.apache.org/jira/browse/BEAM-1310 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Stephen Sisk >Assignee: Stephen Sisk > > Jdbc IO could use some integration tests! We'd like to have them run against > a real list instance of postgres. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-1310) Add running integration tests for JdbcIO
[ https://issues.apache.org/jira/browse/BEAM-1310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15837024#comment-15837024 ] ASF GitHub Bot commented on BEAM-1310: -- Github user ssisk closed the pull request at: https://github.com/apache/beam/pull/1840 > Add running integration tests for JdbcIO > > > Key: BEAM-1310 > URL: https://issues.apache.org/jira/browse/BEAM-1310 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Stephen Sisk >Assignee: Stephen Sisk > > Jdbc IO could use some integration tests! We'd like to have them run against > a real list instance of postgres. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-1310) Add running integration tests for JdbcIO
[ https://issues.apache.org/jira/browse/BEAM-1310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15837006#comment-15837006 ] ASF GitHub Bot commented on BEAM-1310: -- GitHub user ssisk opened a pull request: https://github.com/apache/beam/pull/1840 BEAM-1310 Add integration tests for JdbcIO Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [X ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [X ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [X ] Replace `` in the title with the actual Jira issue number, if there is one. - [X ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- Hi @tgroh can you please take a look? This adds an integration test & data loading script for Jdbc IO You'll note there are TODOs for verifying the contents of the rows in the tests - I will get to those, but wanted to get a first, useful test in so that we can continue working on the end to end integration test infrastructure. This IT demonstrates: * Having a separate load script for read tests * Using TestPipelineOptions that are not test based, but rather data source based. * Writing kubernetes scripts for instantiating an instance of the data source. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ssisk/beam io-testing Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/1840.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1840 commit 21561ce64db15e2ba6b13186cc0092ba419b253e Author: Stephen Sisk Date: 2016-12-23T01:35:40Z Add an example IT that relies on an external data store. commit 83fcafcec6ababa664bbb2098fb440d86873ba20 Author: Stephen Sisk Date: 2017-01-11T19:21:47Z Update JdbcIOIT example so it uses real pipeline options & demonstrates a bug in JdbcIO commit ba57d38e92fc6b79a192738f9f40925d2f3ae6f9 Author: Stephen Sisk Date: 2017-01-12T19:48:37Z JdbcIOIT now runs successfully. Fixed errors in Serialization & tableName It is intentional that we use static inner classes rather than anonymous classes - anonymous classes pull in their containing class, and thus the containing class must be serializable. There was also a small error in using an outdated table name in verification. Also updated a few comments. commit 82b35cf42387f39380145c77b9f15725fe9fff06 Author: Stephen Sisk Date: 2016-12-23T01:35:40Z Add an example IT that relies on an external data store. commit b3456996ad2363de6d27181b03cdd08634a32aad Author: Stephen Sisk Date: 2017-01-11T19:21:47Z Update JdbcIOIT example so it uses real pipeline options & demonstrates a bug in JdbcIO commit 12963c6d657821b50491138b577f9ae0f7286eb6 Author: Stephen Sisk Date: 2017-01-12T19:48:37Z JdbcIOIT now runs successfully. Fixed errors in Serialization & tableName It is intentional that we use static inner classes rather than anonymous classes - anonymous classes pull in their containing class, and thus the containing class must be serializable. There was also a small error in using an outdated table name in verification. Also updated a few comments. commit e59949ae41ea60698490bf4c836382bcd1e05bcc Author: Stephen Sisk Date: 2017-01-13T01:27:17Z Merge branch 'io-testing' of github.com:ssisk/beam into io-testing commit 1e95f9253ad9cd6295f1377850f064edcd317032 Author: Stephen Sisk Date: 2016-12-23T01:35:40Z Add an example IT that relies on an external data store. commit 760413fe328b0f7feed637e39bfe2990168b9cbf Author: Stephen Sisk Date: 2017-01-11T19:21:47Z Update JdbcIOIT example so it uses real pipeline options & demonstrates a bug in JdbcIO commit 8720379930b6fdc772d7e32facb55c9ded5b55ae Author: Stephen Sisk Date: 2017-01-12T19:48:37Z JdbcIOIT now runs successfully. Fixed errors in Serialization & tableName It is intentional that we use static inner classes rather than anonymous classes - anonymous classes pull in their containing class, and thus the containing class must be serializable. There was also a small error in using an outdated table name in verification. Also updated a few comments. commit 8c364ff6cfbc1d34f6f2b283be6ee7beff61c773 Author: Stephen Sisk Date: 2017-01-13T02:16:40Z Switch JdbcTestOptions -> PostgresTestOptions & add comments explaining use. commit 35ee94279b37
