Seems to be green now. Just had to wait for another run after the schema update.
On Thu, Feb 6, 2020 at 9:47 AM Tomo Suzuki <suzt...@google.com> wrote: > Ismaël, > > The latest postcommit failure > https://builds.apache.org/job/beam_PostCommit_SQL/3951/ was 4:15:45 PM > Brian's successful case > https://builds.apache.org/job/beam_PostCommit_SQL_PR/243/ started > 4:29:57 PM. > I hope the next SQL postcommit should succeed. > > On Thu, Feb 6, 2020 at 11:57 AM Ismaël Mejía <ieme...@gmail.com> wrote: > > > > This one is still broken, maybe there are two different data sources, > one for the '_PR' version and the normal one, can you please confirm? > > https://builds.apache.org/job/beam_PostCommit_SQL/ > > > > > > On Thu, Feb 6, 2020 at 5:44 PM Brian Hulette <bhule...@google.com> > wrote: > >> > >> Sorry for the delay. I had some issues updating the schema, I ended up > having to drop it and re-create for some reason. Looks like SQL PostCommit > is green on https://github.com/apache/beam/pull/10765 now. > >> > >> > setting up from scratch is a good idea. > >> +1, I filed https://issues.apache.org/jira/browse/BEAM-9260 for this > and assigned it to Kenn for now. > >> > >> > I’m still feeling it’s ok to read an Integer as Long. > >> In this case the issue is that we were reading data with a schema (id: > INT64), and then passing it to a PAssert that checked against Rows with a > different schema (id: INT32). So I think it's a legitimate error because we > were asserting that a PCollection contained rows with one schema but it > actually had a different schema. The message should definitely be better in > this case though. > >> > >> Brian > >> > >> On Wed, Feb 5, 2020 at 9:28 PM Tomo Suzuki <suzt...@google.com> wrote: > >>> > >>> (My understanding) > >>> The test ensures the CSV data stored in GCS should be readable through > Datacatalog. It fails because an Integer value in the CSV was read as Long > as per Datacatalog. > >>> > >>> > >>> > setting up from scratch is a good idea. > >>> > >>> I agree. Furthermore, it would be nice if it can test different > type-cast behaviors. I’m still feeling it’s ok to read an Integer as Long. > (If this is the case, how about Long to Integer? What if the long is small > enough to fit in 32 bits? and so on) > >>> > >>> On Wed, Feb 5, 2020 at 23:15 Kenneth Knowles <k...@apache.org> wrote: > >>>> > >>>> I think that was me... sorry! > >>>> > >>>> Is this a test where it is important that the data is pre-existing? > Otherwise I would say that setting up from scratch is a good idea. Does > anyone have context on it? I am happy to take on the small bit of coding, > since I broke it. > >>>> > >>>> Kenn > >>>> > >>>> On Wed, Feb 5, 2020 at 1:22 PM Brian Hulette <bhule...@google.com> > wrote: > >>>>> > >>>>> So it looks like the schema for `integ_test_small_csv_test_1` was > updated yesterday around the same time that PR#10563 went in, and it no > longer matches the schema we expect in the test. > >>>>> > >>>>> I'm just going to change it back for now. I am curious who changed > it and why, if the perpetrator is on this list please let us know :) > >>>>> > >>>>> > >>>>> Note the updateTime: > >>>>> ``` > >>>>> ❯ gcloud beta data-catalog entries lookup > '`datacatalog`.`entry`.`apache-beam-testing`.`us-central1`.`samples`.`integ_test_small_csv_test_1`'`` > >>>>> gcsFilesetSpec: > >>>>> filePatterns: > >>>>> - gs://apache-beam-samples/integration_test_small_csv/test.csv > >>>>> linkedResource: // > datacatalog.googleapis.com/projects/apache-beam-testing/locations/us-central1/entryGroups/samples/entries/integ_test_small_csv_test_1 > >>>>> name: > projects/apache-beam-testing/locations/us-central1/entryGroups/samples/entries/integ_test_small_csv_test_1 > >>>>> schema: > >>>>> columns: > >>>>> - column: id > >>>>> mode: NULLABLE > >>>>> type: INT64 > >>>>> - column: name > >>>>> mode: NULLABLE > >>>>> type: STRING > >>>>> - column: type > >>>>> mode: NULLABLE > >>>>> type: STRING > >>>>> sourceSystemTimestamps: > >>>>> createTime: '2019-08-16T01:49:06.235Z' > >>>>> updateTime: '2020-02-04T17:18:17.671Z' > >>>>> type: FILESET > >>>>> ``` > >>> > >>> -- > >>> Regards, > >>> Tomo > > > > -- > Regards, > Tomo >
