Hi, thanks for the help. I'm able to run the test locally by running the sql postCommit :sdks:java:extensions:sql:postCommit. However my problem comes when trying to change something in the query in order to inject an error, for example trying to change the table name in a query in order to test the error path. The problem is that in the integration tests when making the call to the datastore read and write transforms the query is provided by creating a BeamRelNode created with the query sqlEnv.parseQuery(query) <https://github.com/apache/beam/blob/ce406c69d2d06e5c5b659fa259253d5cc64249e7/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datastore/DataStoreReadWriteIT.java#L105> which will throw a ParseException if I try to use a query with a non existent table name. Is there a way to execute the integration test with a query that will cause an error?
On Wed, Aug 18, 2021 at 1:28 PM Kyle Weaver <kcwea...@google.com> wrote: > There's a hard-coded "include" clause in the source. You can change that. > https://github.com/apache/beam/blob/a0fbe00ef12b72ec89672ab32ccc6d6331ca5edd/sdks/java/extensions/sql/build.gradle#L198 > > On Wed, Aug 18, 2021 at 11:19 AM Andrew Pilloud <apill...@google.com> > wrote: > >> You might be able to use the '--tests' flag. (This works for unit tests.) >> For example: >> >> ./gradlew :sdks:java:extensions:sql:postCommit >> --tests >> org.apache.beam.sdk.extensions.sql.meta.provider.datastore.DataStoreReadWriteIT.testDataStoreV1SqlWriteRead >> >> On Wed, Aug 18, 2021 at 11:14 AM Chamikara Jayalath <chamik...@google.com> >> wrote: >> >>> I don't have the Gradle command to run an individual IT unfortunately. I >>> usually invoke the corresponding Jenkins test suite (PostCommit SQL in this >>> case). >>> Note that there are additional Datastore read/write IT tests that do not >>> involve SQL (and run as a part of Java PostCommit). >>> >>> https://ci-beam.apache.org/job/beam_PostCommit_Java/7917/testReport/org.apache.beam.sdk.io.gcp.datastore/ >>> >>> Thanks, >>> Cham >>> >>> On Tue, Aug 17, 2021 at 5:33 PM Alex Amato <ajam...@google.com> wrote: >>> >>>> +Chamikara, +Heejong, >>>> >>>> Do you have any ideas here? I want to verify the library does indeed >>>> produce exceptions as we expect and that we can catch them as done here: >>>> https://github.com/apache/beam/pull/15183 >>>> >>>> On Mon, Aug 16, 2021 at 3:26 PM Miguel Anzo Palomo < >>>> miguel.a...@wizeline.com> wrote: >>>> >>>>> Hi, I was able to run the integration test locally, but I'm having >>>>> some troubles injecting an error to inject an error here [1] to trigger a >>>>> DatastoreException here [2]. The problem is that if I try to change the >>>>> table or fields in the select query, an exception is raised in [1] when >>>>> parsing the sql to create the BeamRelNode. Would anyone know a way to >>>>> inject an error in the integration test to cause a DatastoreException in >>>>> [2]? >>>>> >>>>> [1] >>>>> https://github.com/apache/beam/blob/ce406c69d2d06e5c5b659fa259253d5cc64249e7/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datastore/DataStoreReadWriteIT.java#L105 >>>>> [2] >>>>> https://github.com/apache/beam/blob/089ca7a62be86ad7865308f2cbd676bb18d7d648/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java#L927 >>>>> >>>>> On Tue, Aug 10, 2021 at 3:25 PM Miguel Anzo Palomo < >>>>> miguel.a...@wizeline.com> wrote: >>>>> >>>>>> Hi Alex, >>>>>> My problem trying to locally run that test was that, since I don't >>>>>> have access to the apache-beam-testing GCP project I was having problems >>>>>> getting a datastore instance to read/write from. >>>>>> >>>>>> On Tue, Aug 10, 2021 at 1:08 PM Alex Amato <ajam...@google.com> >>>>>> wrote: >>>>>> >>>>>>> Miguel, I believe you were having trouble running this test, you >>>>>>> mentioned that you needed a Datastore instance. >>>>>>> Would you mind sharing the errors or any issue you had when you >>>>>>> tried Ke's command? >>>>>>> >>>>>>> Ke, Do we need to specify a datastore instance to run this test? >>>>>>> Where will it read/write from? I assume it will it default to something >>>>>>> in >>>>>>> the apache-beam-testing GCP project, as other tests do. >>>>>>> >>>>>>> >>>>>>> On Thu, Jul 29, 2021 at 6:16 PM Ke Wu <ke.wu...@gmail.com> wrote: >>>>>>> >>>>>>>> “Run SQL PostCommit” is essentially running >>>>>>>> “./gradlew :sqlPostCommit” [1] >>>>>>>> >>>>>>>> In your case where you would like to run DataStoreReadWriteIT only, >>>>>>>> it can be simplified [2] to >>>>>>>> >>>>>>>> "./gradlew :sdks:java:extensions:sql:postCommit” >>>>>>>> >>>>>>>> Best, >>>>>>>> Ke >>>>>>>> >>>>>>>> [1] >>>>>>>> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PostCommit_SQL.groovy#L41 >>>>>>>> >>>>>>>> [2] >>>>>>>> https://github.com/apache/beam/blob/master/build.gradle.kts#L190 >>>>>>>> >>>>>>>> >>>>>>>> On Jul 29, 2021, at 9:58 AM, Alex Amato <ajam...@google.com> wrote: >>>>>>>> >>>>>>>> I was hoping for the command line to run it. So that the test could >>>>>>>> be tweaked to inject an error, and ensure the error handling code >>>>>>>> works as >>>>>>>> expected >>>>>>>> >>>>>>>> On Wed, Jul 28, 2021 at 8:34 PM Ke Wu <ke.wu...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Comment the PR with "Run SQL PostCommit” would trigger the post >>>>>>>>> commit integration tests for SQL, which I suppose includes >>>>>>>>> DataStoreReadWriteIT >>>>>>>>> >>>>>>>>> Let me know if whether or not this is sufficient. >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Ke >>>>>>>>> >>>>>>>>> On Jul 28, 2021, at 12:20 PM, Alex Amato <ajam...@google.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Is it possible to run a Datastore IO integration test to test this >>>>>>>>> PR? >>>>>>>>> >>>>>>>>> https://github.com/apache/beam/pull/15183/files >>>>>>>>> >>>>>>>>> Probably this test can be ran somehow. Though I don't know the >>>>>>>>> gradle command to run it >>>>>>>>> >>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datastore/DataStoreReadWriteIT.java >>>>>>>>> >>>>>>>>> Does anyone know how to run this test? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Miguel Angel Anzo Palomo | WIZELINE >>>>>> >>>>>> Software Engineer >>>>>> >>>>>> miguel.a...@wizeline.com >>>>>> >>>>>> Remote Office >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Miguel Angel Anzo Palomo | WIZELINE >>>>> >>>>> Software Engineer >>>>> >>>>> miguel.a...@wizeline.com >>>>> >>>>> Remote Office >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> *This email and its contents (including any attachments) are being >>>>> sent toyou on the condition of confidentiality and may be protected by >>>>> legalprivilege. Access to this email by anyone other than the intended >>>>> recipientis unauthorized. If you are not the intended recipient, please >>>>> immediatelynotify the sender by replying to this message and delete the >>>>> materialimmediately from your system. Any further use, dissemination, >>>>> distributionor reproduction of this email is strictly prohibited. Further, >>>>> norepresentation is made with respect to any content contained in this >>>>> email.* >>>> >>>> -- Miguel Angel Anzo Palomo | WIZELINE Software Engineer miguel.a...@wizeline.com Remote Office -- *This email and its contents (including any attachments) are being sent to you on the condition of confidentiality and may be protected by legal privilege. Access to this email by anyone other than the intended recipient is unauthorized. If you are not the intended recipient, please immediately notify the sender by replying to this message and delete the material immediately from your system. Any further use, dissemination, distribution or reproduction of this email is strictly prohibited. Further, no representation is made with respect to any content contained in this email.*
