Hi Kamil,
I did not code nor scheduled the spark performance tests but I see in the log
(https://builds.apache.org/blue/organizati
ons/jenkins/beam_PerformanceTests_Spark/detail/beam_PerformanceTests_Spark/1628/pipeline)
that it uses PerfKit and also
that the job that is run is this one: dpb_wordcount_benchmark
Best
Etienne
Le mardi 24 avril 2018 à 11:05 +0200, Kamil Szewczyk a écrit :
> Hi Etienne,
>
> I was recently playing a lot with BigQuerry when working on anomaly detection
> tool and noticed that in db schema
> timestamp is defined as FLOAT.
> Perfkit also produces it as a float
> 'timestamp': 1524485484.41655,
> so the upload passes.
>
> Probably it was defined as float from the beginning because of how perfkit
> produces it.
>
> Are you using Perfkit when running performance tests job for spark ?
>
> Kind Regards
> Kamil Szewczyk
>
> 2018-04-23 10:17 GMT+02:00 Etienne Chauchot <[email protected]>:
> > Hi guys,
> >
> > I noticed a failure in the performance tests job for spark (I did not take
> > a look at the others): it seems to be
> > related to a schema update in the bigQuery output.
> >
> > BigQuery error in load operation: Error processing job
> > 'apache-beam-testing:bqjob_r2527a0e444514f2b_00000162f128db2b_1': Invalid
> > schema
> > update. Field timestamp has changed type from TIMESTAMP to FLOAT
> >
> > I opened a ticket to track the issue
> > https://issues.apache.org/jira/browse/BEAM-4153
> >
> > Best
> >
> > Etienne
> >
> >
