Re: Command line to run DatstoreIO integration tests for java

2021-08-20 Thread Miguel Anzo Palomo 
Hi, thanks for the help. I'm able to run the test locally by running the
sql postCommit :sdks:java:extensions:sql:postCommit. However my problem
comes when trying to change something in the query in order to inject an
error, for example trying to change the table name in a query in order to
test the error path. The problem is that in the integration tests when
making the call to the datastore read and write transforms the query is
provided by creating a BeamRelNode created with the query
sqlEnv.parseQuery(query)

which will throw a ParseException if I try to use a query with a non
existent table name. Is there a way to execute the integration test with a
query that will cause an error?


On Wed, Aug 18, 2021 at 1:28 PM Kyle Weaver  wrote:

> There's a hard-coded "include" clause in the source. You can change that.
> https://github.com/apache/beam/blob/a0fbe00ef12b72ec89672ab32ccc6d6331ca5edd/sdks/java/extensions/sql/build.gradle#L198
>
> On Wed, Aug 18, 2021 at 11:19 AM Andrew Pilloud 
> wrote:
>
>> You might be able to use the '--tests' flag. (This works for unit tests.)
>> For example:
>>
>> ./gradlew :sdks:java:extensions:sql:postCommit
>> --tests 
>> org.apache.beam.sdk.extensions.sql.meta.provider.datastore.DataStoreReadWriteIT.testDataStoreV1SqlWriteRead
>>
>> On Wed, Aug 18, 2021 at 11:14 AM Chamikara Jayalath 
>> wrote:
>>
>>> I don't have the Gradle command to run an individual IT unfortunately. I
>>> usually invoke the corresponding Jenkins test suite (PostCommit SQL in this
>>> case).
>>> Note that there are additional Datastore read/write IT tests that do not
>>> involve SQL (and run as a part of Java PostCommit).
>>>
>>> https://ci-beam.apache.org/job/beam_PostCommit_Java/7917/testReport/org.apache.beam.sdk.io.gcp.datastore/
>>>
>>> Thanks,
>>> Cham
>>>
>>> On Tue, Aug 17, 2021 at 5:33 PM Alex Amato  wrote:
>>>
 +Chamikara, +Heejong,

 Do you have any ideas here? I want to verify the library does indeed
 produce exceptions as we expect and that we can catch them as done here:
 https://github.com/apache/beam/pull/15183

 On Mon, Aug 16, 2021 at 3:26 PM Miguel Anzo Palomo <
 miguel.a...@wizeline.com> wrote:

> Hi, I was able to run the integration test locally, but I'm having
> some troubles injecting an error to inject an error here [1] to trigger a
> DatastoreException here [2]. The problem is that if I try to change the
> table or fields in the select query, an exception is raised in [1] when
> parsing the sql to create the BeamRelNode. Would anyone know a way to
> inject an error in the integration test to cause a DatastoreException in
> [2]?
>
> [1]
> https://github.com/apache/beam/blob/ce406c69d2d06e5c5b659fa259253d5cc64249e7/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datastore/DataStoreReadWriteIT.java#L105
> [2]
> https://github.com/apache/beam/blob/089ca7a62be86ad7865308f2cbd676bb18d7d648/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java#L927
>
> On Tue, Aug 10, 2021 at 3:25 PM Miguel Anzo Palomo <
> miguel.a...@wizeline.com> wrote:
>
>> Hi Alex,
>> My problem trying to locally run that test was that, since I don't
>> have access to the apache-beam-testing GCP project I was having problems
>> getting a datastore instance to read/write from.
>>
>> On Tue, Aug 10, 2021 at 1:08 PM Alex Amato 
>> wrote:
>>
>>> Miguel, I believe you were having trouble running this test, you
>>> mentioned that you needed a Datastore instance.
>>> Would you mind sharing the errors or any issue you had when you
>>> tried Ke's command?
>>>
>>> Ke, Do we need to specify a datastore instance to run this test?
>>> Where will it read/write from? I assume it will it default to something 
>>> in
>>> the apache-beam-testing GCP project, as other tests do.
>>>
>>>
>>> On Thu, Jul 29, 2021 at 6:16 PM Ke Wu  wrote:
>>>
 “Run SQL PostCommit” is essentially running
 “./gradlew :sqlPostCommit” [1]

 In your case where you would like to run DataStoreReadWriteIT only,
 it can be simplified [2] to

 "./gradlew :sdks:java:extensions:sql:postCommit”

 Best,
 Ke

 [1]
 https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PostCommit_SQL.groovy#L41

 [2]
 https://github.com/apache/beam/blob/master/build.gradle.kts#L190


 On Jul 29, 2021, at 9:58 AM, Alex Amato  wrote:

 I was hoping for the command line to run it. So that the test could
 be tweaked to inject an

Re: Command line to run DatstoreIO integration tests for java

2021-08-16 Thread Miguel Anzo Palomo 
Hi, I was able to run the integration test locally, but I'm having some
troubles injecting an error to inject an error here [1] to trigger a
DatastoreException here [2]. The problem is that if I try to change the
table or fields in the select query, an exception is raised in [1] when
parsing the sql to create the BeamRelNode. Would anyone know a way to
inject an error in the integration test to cause a DatastoreException in
[2]?

[1]
https://github.com/apache/beam/blob/ce406c69d2d06e5c5b659fa259253d5cc64249e7/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datastore/DataStoreReadWriteIT.java#L105
[2]
https://github.com/apache/beam/blob/089ca7a62be86ad7865308f2cbd676bb18d7d648/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java#L927

On Tue, Aug 10, 2021 at 3:25 PM Miguel Anzo Palomo 
wrote:

> Hi Alex,
> My problem trying to locally run that test was that, since I don't have
> access to the apache-beam-testing GCP project I was having problems getting
> a datastore instance to read/write from.
>
> On Tue, Aug 10, 2021 at 1:08 PM Alex Amato  wrote:
>
>> Miguel, I believe you were having trouble running this test, you
>> mentioned that you needed a Datastore instance.
>> Would you mind sharing the errors or any issue you had when you tried
>> Ke's command?
>>
>> Ke, Do we need to specify a datastore instance to run this test? Where
>> will it read/write from? I assume it will it default to something in
>> the apache-beam-testing GCP project, as other tests do.
>>
>>
>> On Thu, Jul 29, 2021 at 6:16 PM Ke Wu  wrote:
>>
>>> “Run SQL PostCommit” is essentially running “./gradlew :sqlPostCommit”
>>> [1]
>>>
>>> In your case where you would like to run DataStoreReadWriteIT only, it
>>> can be simplified [2] to
>>>
>>> "./gradlew :sdks:java:extensions:sql:postCommit”
>>>
>>> Best,
>>> Ke
>>>
>>> [1]
>>> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PostCommit_SQL.groovy#L41
>>>
>>> [2] https://github.com/apache/beam/blob/master/build.gradle.kts#L190
>>>
>>>
>>> On Jul 29, 2021, at 9:58 AM, Alex Amato  wrote:
>>>
>>> I was hoping for the command line to run it. So that the test could be
>>> tweaked to inject an error, and ensure the error handling code works as
>>> expected
>>>
>>> On Wed, Jul 28, 2021 at 8:34 PM Ke Wu  wrote:
>>>
 Comment the PR with "Run SQL PostCommit” would trigger the post commit
 integration tests for SQL, which I suppose includes DataStoreReadWriteIT

 Let me know if whether or not this is sufficient.

 Best,
 Ke

 On Jul 28, 2021, at 12:20 PM, Alex Amato  wrote:

 Is it possible to run a Datastore IO integration test to test this PR?

 https://github.com/apache/beam/pull/15183/files

 Probably this test can be ran somehow. Though I don't know the gradle
 command to run it

 https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datastore/DataStoreReadWriteIT.java

 Does anyone know how to run this test?



>>>
>
> --
>
> Miguel Angel Anzo Palomo | WIZELINE
>
> Software Engineer
>
> miguel.a...@wizeline.com
>
> Remote Office
>


-- 

Miguel Angel Anzo Palomo | WIZELINE

Software Engineer

miguel.a...@wizeline.com

Remote Office

-- 
*This email and its contents (including any attachments) are being sent to
you on the condition of confidentiality and may be protected by legal
privilege. Access to this email by anyone other than the intended recipient
is unauthorized. If you are not the intended recipient, please immediately
notify the sender by replying to this message and delete the material
immediately from your system. Any further use, dissemination, distribution
or reproduction of this email is strictly prohibited. Further, no
representation is made with respect to any content contained in this email.*

Re: Command line to run DatstoreIO integration tests for java

2021-08-10 Thread Miguel Anzo Palomo 
Hi Alex,
My problem trying to locally run that test was that, since I don't have
access to the apache-beam-testing GCP project I was having problems getting
a datastore instance to read/write from.

On Tue, Aug 10, 2021 at 1:08 PM Alex Amato  wrote:

> Miguel, I believe you were having trouble running this test, you mentioned
> that you needed a Datastore instance.
> Would you mind sharing the errors or any issue you had when you tried Ke's
> command?
>
> Ke, Do we need to specify a datastore instance to run this test? Where
> will it read/write from? I assume it will it default to something in
> the apache-beam-testing GCP project, as other tests do.
>
>
> On Thu, Jul 29, 2021 at 6:16 PM Ke Wu  wrote:
>
>> “Run SQL PostCommit” is essentially running “./gradlew :sqlPostCommit” [1]
>>
>> In your case where you would like to run DataStoreReadWriteIT only, it
>> can be simplified [2] to
>>
>> "./gradlew :sdks:java:extensions:sql:postCommit”
>>
>> Best,
>> Ke
>>
>> [1]
>> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PostCommit_SQL.groovy#L41
>>
>> [2] https://github.com/apache/beam/blob/master/build.gradle.kts#L190
>>
>>
>> On Jul 29, 2021, at 9:58 AM, Alex Amato  wrote:
>>
>> I was hoping for the command line to run it. So that the test could be
>> tweaked to inject an error, and ensure the error handling code works as
>> expected
>>
>> On Wed, Jul 28, 2021 at 8:34 PM Ke Wu  wrote:
>>
>>> Comment the PR with "Run SQL PostCommit” would trigger the post commit
>>> integration tests for SQL, which I suppose includes DataStoreReadWriteIT
>>>
>>> Let me know if whether or not this is sufficient.
>>>
>>> Best,
>>> Ke
>>>
>>> On Jul 28, 2021, at 12:20 PM, Alex Amato  wrote:
>>>
>>> Is it possible to run a Datastore IO integration test to test this PR?
>>>
>>> https://github.com/apache/beam/pull/15183/files
>>>
>>> Probably this test can be ran somehow. Though I don't know the gradle
>>> command to run it
>>>
>>> https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datastore/DataStoreReadWriteIT.java
>>>
>>> Does anyone know how to run this test?
>>>
>>>
>>>
>>

-- 

Miguel Angel Anzo Palomo | WIZELINE

Software Engineer

miguel.a...@wizeline.com

Remote Office

-- 
*This email and its contents (including any attachments) are being sent to
you on the condition of confidentiality and may be protected by legal
privilege. Access to this email by anyone other than the intended recipient
is unauthorized. If you are not the intended recipient, please immediately
notify the sender by replying to this message and delete the material
immediately from your system. Any further use, dissemination, distribution
or reproduction of this email is strictly prohibited. Further, no
representation is made with respect to any content contained in this email.*

Re: Command line to run DatstoreIO integration tests for java

2021-07-29 Thread Ke Wu 
“Run SQL PostCommit” is essentially running “./gradlew :sqlPostCommit” [1]

In your case where you would like to run DataStoreReadWriteIT only, it can be 
simplified [2] to 

"./gradlew :sdks:java:extensions:sql:postCommit”

Best,
Ke

[1] 
https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PostCommit_SQL.groovy#L41
 

 
[2] https://github.com/apache/beam/blob/master/build.gradle.kts#L190 
 


> On Jul 29, 2021, at 9:58 AM, Alex Amato  wrote:
> 
> I was hoping for the command line to run it. So that the test could be 
> tweaked to inject an error, and ensure the error handling code works as 
> expected
> 
> On Wed, Jul 28, 2021 at 8:34 PM Ke Wu  > wrote:
> Comment the PR with "Run SQL PostCommit” would trigger the post commit 
> integration tests for SQL, which I suppose includes DataStoreReadWriteIT
> 
> Let me know if whether or not this is sufficient.
> 
> Best,
> Ke
> 
>> On Jul 28, 2021, at 12:20 PM, Alex Amato > > wrote:
>> 
>> Is it possible to run a Datastore IO integration test to test this PR?
>> 
>> https://github.com/apache/beam/pull/15183/files 
>> 
>> 
>> Probably this test can be ran somehow. Though I don't know the gradle 
>> command to run it
>> https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datastore/DataStoreReadWriteIT.java
>>  
>> 
>> 
>> Does anyone know how to run this test?
>

Re: Command line to run DatstoreIO integration tests for java

2021-07-29 Thread Alex Amato 
I was hoping for the command line to run it. So that the test could be
tweaked to inject an error, and ensure the error handling code works as
expected

On Wed, Jul 28, 2021 at 8:34 PM Ke Wu  wrote:

> Comment the PR with "Run SQL PostCommit” would trigger the post commit
> integration tests for SQL, which I suppose includes DataStoreReadWriteIT
>
> Let me know if whether or not this is sufficient.
>
> Best,
> Ke
>
> On Jul 28, 2021, at 12:20 PM, Alex Amato  wrote:
>
> Is it possible to run a Datastore IO integration test to test this PR?
>
> https://github.com/apache/beam/pull/15183/files
>
> Probably this test can be ran somehow. Though I don't know the gradle
> command to run it
>
> https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datastore/DataStoreReadWriteIT.java
>
> Does anyone know how to run this test?
>
>
>

Re: Command line to run DatstoreIO integration tests for java

2021-07-28 Thread Ke Wu 
Comment the PR with "Run SQL PostCommit” would trigger the post commit 
integration tests for SQL, which I suppose includes DataStoreReadWriteIT

Let me know if whether or not this is sufficient.

Best,
Ke

> On Jul 28, 2021, at 12:20 PM, Alex Amato  wrote:
> 
> Is it possible to run a Datastore IO integration test to test this PR?
> 
> https://github.com/apache/beam/pull/15183/files 
> 
> 
> Probably this test can be ran somehow. Though I don't know the gradle command 
> to run it
> https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datastore/DataStoreReadWriteIT.java
>  
> 
> 
> Does anyone know how to run this test?