Awesome !!! Great news !
Thanks guys for that !
I started to implement IT in JMS, MQTT, Redis, Cassandra IOs. I keep you posted.
Regards
JB
On 03/21/2017 11:01 PM, Stephen Sisk wrote:
I'm really excited to see these tests are running!
These Jdbc tests are testing against a postgres instance - that instance is
running on the kubernetes cluster I've set up for beam IO ITs as discussed
in the "Hosting data stores for IO transform testing" thread[0]. I set up
that postgres instance using the kubernetes scripts for Jdbc[1]. Anyone can
run their own kubernetes cluster and do the same thing for themselves to
run the ITs. (I'd actually to love to hear about that if anyone does it.)
I'm excited to get a few more ITs using this infrastructure so we can test
it out/smooth out the remaining rough edges in creating ITs. I'm happy to
answer questions about that on the mailing list, but we obviously have to
have the process written down - the Testing IO Transforms in Apache Beam
doc [2] covers how to do this, but is still rough. I'm working on getting
that up on the website and ironing out the rough edges [3], but generally
reading that doc plus checking out how the JdbcIO or ElasticsearchIO tests
work should give you a sense of how to get it working. I'm also thinking we
might want to simplify the way we do data loading, so I don't consider this
process fully stabilized, but I'll port code written according to the
current standards to the new standards if we make changes.
ElasticsearchIO has all the prerequisites, so I'd like to get them going in
the near future. I know JB has started on this in his RedisIO PR, and the
HadoopInputFormatIO also has ITs & k8 scripts, so there's more in the pipe.
For now, each datastore has to be manually set up, but I'd like to automate
that process - I'll file a JIRA ticket shortly for that.
Thanks,
Stephen
[0] Hosting data stores for IO transform testing -
https://lists.apache.org/thread.html/9fd3c51cb679706efa4d0df2111a6ac438b851818b639aba644607af@%3Cdev.beam.apache.org%3E
[1] Postgres k8 scripts -
https://github.com/apache/beam/tree/master/sdks/java/io/jdbc/src/test/resources/kubernetes
[2] IO testing guide -
https://docs.google.com/document/d/153J9jPQhMCNi_eBzJfhAg-NprQ7vbf1jNVRgdqeEE8I/edit?usp=sharing
[3] Jira for IO guide - https://issues.apache.org/jira/browse/BEAM-1025
On Tue, Mar 21, 2017 at 2:28 PM Jason Kuster <jasonkus...@google.com.invalid>
wrote:
Hi all,
Exciting news! As of yesterday, we have checked in the Jenkins
configuration for our first continuously running IO Integration Test! You
can check it out in Jenkins here[1]. We’re also publishing results to a
database, and we’ve turned up a basic dashboarding system where you can see
the results here[2]. Caveat: there are only two runs, and we’ll be tweaking
the underlying system still, so don’t panic that we’re up and to the right
currently. ;)
This is the first test running continuously on top of the performance / IO
testing infrastructure described in this doc[3]. Initial support for Beam
is now present in PerfKit Benchmarker; given what they had already, it was
easiest to add support for Dataflow and Java. We need your help to add
additional support! The doc lists a number of JIRA issues to build out
support for other systems. I’m happy to work with people to help them
understand what is necessary for these tasks; just send an email to the
list if you need help and I’ll help you move forwards.
Looking forward to it!
Jason
[1] https://builds.apache.org/job/beam_PerformanceTests_JDBC/
[2]
https://apache-beam-testing.appspot.com/explore?dashboard=5714163003293696
[3]
https://docs.google.com/document/d/1PsjGPSN6FuorEEPrKEP3u3m16tyOzph5FnL2DhaRDz0/edit?ts=58a78e73
--
-------
Jason Kuster
Apache Beam / Google Cloud Dataflow
--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
