Re: Review Request 23666: REST service for aurora client commands, built with Tornado Web framework
> On July 22, 2014, 12:54 a.m., Bill Farner wrote: > > Do you think this should be obviated by https://reviews.apache.org/r/23741/ > > ? > > Misho Krastev wrote: > This looks like elegant (declarative) solution to export REST variant of > the native thrift interface. I am not familiar with the Aurora scheduler code > so I can not read very well the diffs. From the review request description > and what I understand from the diffs, I would vote for that (your) solution. > > However, I don't think this REST service can be replaced by the other > one. The reason is the REST service that I submitted for review is an > interface to the capabilities provided by the Aurora client, and this client > is functionality-rich. It adds more features on top of those offered by the > scheduler. In particular, what we needed was the ability to do rolling > upgrades where instances (shards) are upgraded in batches and not en bloc. > There are other examples but this is the one that is of great importance to > us. > > Bill Farner wrote: > Ah, yes - good point. We're actually embarking on pulling update > orchestration into the scheduler [1] this week, after which we will remove > the updating logic from the client. > > > [1] > http://mail-archives.apache.org/mod_mbox/incubator-aurora-dev/201407.mbox/browser > > Bill Farner wrote: > Err, here's the permalink: > http://mail-archives.apache.org/mod_mbox/incubator-aurora-dev/201407.mbox/%3CCAGRA8uMxwVDokp_iHXhNru2gd-x_nM%2BDYAurpfAO6wuX7%3DnHFw%40mail.gmail.com%3E > > Misho Krastev wrote: > Very well. You are adding REST interface to the scheduler, and when you > have the additional features that today are offered by the client it will be > the complete solution. Hope it will be available soon, until that time we'll > be using our version of the REST service. > > I will also look into the option to change this code into standalone > application that can be built with the aurora client libraries but outside of > the codebase. It will be good exercise and it may be useful to someone. > > I will remove this review request. Thanks for the comments. Thanks for your understanding. We hope to have these features ready to try out in ~2 weeks, tracking at AURORA-610 [1]. [1] https://issues.apache.org/jira/browse/AURORA-610 - Bill --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23666/#review48322 --- On July 21, 2014, 10:58 p.m., Misho Krastev wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/23666/ > --- > > (Updated July 21, 2014, 10:58 p.m.) > > > Review request for Aurora. > > > Repository: aurora > > > Description > --- > > REST service for aurora client commands, built using Tornado Web framework > > This is a draft implementation of a REST server exposing APIs to call Aurora > client commands. It is a wrapper around the aurora command-line client and > aurora client code. It implements some but not all of the commands available > with the aurora command-line client. Comments on the code as well suggestions > for improvements and enhancements are requested. > > This work is a eBay project that we would like to contribute to the Aurora > project. > > Note: development is still ongoing to add unittests and to resolve the issues > discovered with the stress tests. > Note(2): GitHub repo -- https://github.com/misho-kr/incubator-aurora > > > Diffs > - > > 3rdparty/python/BUILD 641d610deb58fd102ec584e3deaa388b51a6f221 > docs/client-rest-api.md PRE-CREATION > src/main/python/apache/aurora/client/bin/BUILD > 43d747956df0611b0880f64df9955d5f5806901c > src/main/python/apache/aurora/client/bin/aurora_rest.py PRE-CREATION > src/main/python/apache/aurora/client/rest/BUILD PRE-CREATION > src/main/python/apache/aurora/client/rest/application.py PRE-CREATION > src/main/python/apache/aurora/client/rest/application_async.py PRE-CREATION > src/main/python/apache/aurora/client/rest/coroutine_executor.py > PRE-CREATION > src/main/python/apache/aurora/client/rest/external_executor.py PRE-CREATION > src/main/python/apache/aurora/client/rest/internal_executor.py PRE-CREATION > src/main/python/apache/aurora/client/rest/mp_executor.py PRE-CREATION > src/main/python/apache/aurora/client/rest/mt_executor.py PRE-CREATION > > Diff: https://reviews.apache.org/r/23666/diff/ > > > Testing > --- > > Stress testing with Locust to verify the REST service can handle multiple > requests simultaneously. Not all of them passed, particularly the ones with > the internal executor that calls Aurora client api. > > > Thanks, > > Misho
Re: Review Request 23666: REST service for aurora client commands, built with Tornado Web framework
> On July 22, 2014, 12:54 a.m., Bill Farner wrote: > > Do you think this should be obviated by https://reviews.apache.org/r/23741/ > > ? > > Misho Krastev wrote: > This looks like elegant (declarative) solution to export REST variant of > the native thrift interface. I am not familiar with the Aurora scheduler code > so I can not read very well the diffs. From the review request description > and what I understand from the diffs, I would vote for that (your) solution. > > However, I don't think this REST service can be replaced by the other > one. The reason is the REST service that I submitted for review is an > interface to the capabilities provided by the Aurora client, and this client > is functionality-rich. It adds more features on top of those offered by the > scheduler. In particular, what we needed was the ability to do rolling > upgrades where instances (shards) are upgraded in batches and not en bloc. > There are other examples but this is the one that is of great importance to > us. > > Bill Farner wrote: > Ah, yes - good point. We're actually embarking on pulling update > orchestration into the scheduler [1] this week, after which we will remove > the updating logic from the client. > > > [1] > http://mail-archives.apache.org/mod_mbox/incubator-aurora-dev/201407.mbox/browser > > Bill Farner wrote: > Err, here's the permalink: > http://mail-archives.apache.org/mod_mbox/incubator-aurora-dev/201407.mbox/%3CCAGRA8uMxwVDokp_iHXhNru2gd-x_nM%2BDYAurpfAO6wuX7%3DnHFw%40mail.gmail.com%3E Very well. You are adding REST interface to the scheduler, and when you have the additional features that today are offered by the client it will be the complete solution. Hope it will be available soon, until that time we'll be using our version of the REST service. I will also look into the option to change this code into standalone application that can be built with the aurora client libraries but outside of the codebase. It will be good exercise and it may be useful to someone. I will remove this review request. Thanks for the comments. - Misho --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23666/#review48322 --- On July 21, 2014, 10:58 p.m., Misho Krastev wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/23666/ > --- > > (Updated July 21, 2014, 10:58 p.m.) > > > Review request for Aurora. > > > Repository: aurora > > > Description > --- > > REST service for aurora client commands, built using Tornado Web framework > > This is a draft implementation of a REST server exposing APIs to call Aurora > client commands. It is a wrapper around the aurora command-line client and > aurora client code. It implements some but not all of the commands available > with the aurora command-line client. Comments on the code as well suggestions > for improvements and enhancements are requested. > > This work is a eBay project that we would like to contribute to the Aurora > project. > > Note: development is still ongoing to add unittests and to resolve the issues > discovered with the stress tests. > Note(2): GitHub repo -- https://github.com/misho-kr/incubator-aurora > > > Diffs > - > > 3rdparty/python/BUILD 641d610deb58fd102ec584e3deaa388b51a6f221 > docs/client-rest-api.md PRE-CREATION > src/main/python/apache/aurora/client/bin/BUILD > 43d747956df0611b0880f64df9955d5f5806901c > src/main/python/apache/aurora/client/bin/aurora_rest.py PRE-CREATION > src/main/python/apache/aurora/client/rest/BUILD PRE-CREATION > src/main/python/apache/aurora/client/rest/application.py PRE-CREATION > src/main/python/apache/aurora/client/rest/application_async.py PRE-CREATION > src/main/python/apache/aurora/client/rest/coroutine_executor.py > PRE-CREATION > src/main/python/apache/aurora/client/rest/external_executor.py PRE-CREATION > src/main/python/apache/aurora/client/rest/internal_executor.py PRE-CREATION > src/main/python/apache/aurora/client/rest/mp_executor.py PRE-CREATION > src/main/python/apache/aurora/client/rest/mt_executor.py PRE-CREATION > > Diff: https://reviews.apache.org/r/23666/diff/ > > > Testing > --- > > Stress testing with Locust to verify the REST service can handle multiple > requests simultaneously. Not all of them passed, particularly the ones with > the internal executor that calls Aurora client api. > > > Thanks, > > Misho Krastev > >
Re: Review Request 23666: REST service for aurora client commands, built with Tornado Web framework
> On July 22, 2014, 12:54 a.m., Bill Farner wrote: > > Do you think this should be obviated by https://reviews.apache.org/r/23741/ > > ? > > Misho Krastev wrote: > This looks like elegant (declarative) solution to export REST variant of > the native thrift interface. I am not familiar with the Aurora scheduler code > so I can not read very well the diffs. From the review request description > and what I understand from the diffs, I would vote for that (your) solution. > > However, I don't think this REST service can be replaced by the other > one. The reason is the REST service that I submitted for review is an > interface to the capabilities provided by the Aurora client, and this client > is functionality-rich. It adds more features on top of those offered by the > scheduler. In particular, what we needed was the ability to do rolling > upgrades where instances (shards) are upgraded in batches and not en bloc. > There are other examples but this is the one that is of great importance to > us. > > Bill Farner wrote: > Ah, yes - good point. We're actually embarking on pulling update > orchestration into the scheduler [1] this week, after which we will remove > the updating logic from the client. > > > [1] > http://mail-archives.apache.org/mod_mbox/incubator-aurora-dev/201407.mbox/browser Err, here's the permalink: http://mail-archives.apache.org/mod_mbox/incubator-aurora-dev/201407.mbox/%3CCAGRA8uMxwVDokp_iHXhNru2gd-x_nM%2BDYAurpfAO6wuX7%3DnHFw%40mail.gmail.com%3E - Bill --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23666/#review48322 --- On July 21, 2014, 10:58 p.m., Misho Krastev wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/23666/ > --- > > (Updated July 21, 2014, 10:58 p.m.) > > > Review request for Aurora. > > > Repository: aurora > > > Description > --- > > REST service for aurora client commands, built using Tornado Web framework > > This is a draft implementation of a REST server exposing APIs to call Aurora > client commands. It is a wrapper around the aurora command-line client and > aurora client code. It implements some but not all of the commands available > with the aurora command-line client. Comments on the code as well suggestions > for improvements and enhancements are requested. > > This work is a eBay project that we would like to contribute to the Aurora > project. > > Note: development is still ongoing to add unittests and to resolve the issues > discovered with the stress tests. > Note(2): GitHub repo -- https://github.com/misho-kr/incubator-aurora > > > Diffs > - > > 3rdparty/python/BUILD 641d610deb58fd102ec584e3deaa388b51a6f221 > docs/client-rest-api.md PRE-CREATION > src/main/python/apache/aurora/client/bin/BUILD > 43d747956df0611b0880f64df9955d5f5806901c > src/main/python/apache/aurora/client/bin/aurora_rest.py PRE-CREATION > src/main/python/apache/aurora/client/rest/BUILD PRE-CREATION > src/main/python/apache/aurora/client/rest/application.py PRE-CREATION > src/main/python/apache/aurora/client/rest/application_async.py PRE-CREATION > src/main/python/apache/aurora/client/rest/coroutine_executor.py > PRE-CREATION > src/main/python/apache/aurora/client/rest/external_executor.py PRE-CREATION > src/main/python/apache/aurora/client/rest/internal_executor.py PRE-CREATION > src/main/python/apache/aurora/client/rest/mp_executor.py PRE-CREATION > src/main/python/apache/aurora/client/rest/mt_executor.py PRE-CREATION > > Diff: https://reviews.apache.org/r/23666/diff/ > > > Testing > --- > > Stress testing with Locust to verify the REST service can handle multiple > requests simultaneously. Not all of them passed, particularly the ones with > the internal executor that calls Aurora client api. > > > Thanks, > > Misho Krastev > >
Re: Review Request 23666: REST service for aurora client commands, built with Tornado Web framework
> On July 22, 2014, 12:54 a.m., Bill Farner wrote: > > Do you think this should be obviated by https://reviews.apache.org/r/23741/ > > ? > > Misho Krastev wrote: > This looks like elegant (declarative) solution to export REST variant of > the native thrift interface. I am not familiar with the Aurora scheduler code > so I can not read very well the diffs. From the review request description > and what I understand from the diffs, I would vote for that (your) solution. > > However, I don't think this REST service can be replaced by the other > one. The reason is the REST service that I submitted for review is an > interface to the capabilities provided by the Aurora client, and this client > is functionality-rich. It adds more features on top of those offered by the > scheduler. In particular, what we needed was the ability to do rolling > upgrades where instances (shards) are upgraded in batches and not en bloc. > There are other examples but this is the one that is of great importance to > us. Ah, yes - good point. We're actually embarking on pulling update orchestration into the scheduler [1] this week, after which we will remove the updating logic from the client. [1] http://mail-archives.apache.org/mod_mbox/incubator-aurora-dev/201407.mbox/browser - Bill --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23666/#review48322 --- On July 21, 2014, 10:58 p.m., Misho Krastev wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/23666/ > --- > > (Updated July 21, 2014, 10:58 p.m.) > > > Review request for Aurora. > > > Repository: aurora > > > Description > --- > > REST service for aurora client commands, built using Tornado Web framework > > This is a draft implementation of a REST server exposing APIs to call Aurora > client commands. It is a wrapper around the aurora command-line client and > aurora client code. It implements some but not all of the commands available > with the aurora command-line client. Comments on the code as well suggestions > for improvements and enhancements are requested. > > This work is a eBay project that we would like to contribute to the Aurora > project. > > Note: development is still ongoing to add unittests and to resolve the issues > discovered with the stress tests. > Note(2): GitHub repo -- https://github.com/misho-kr/incubator-aurora > > > Diffs > - > > 3rdparty/python/BUILD 641d610deb58fd102ec584e3deaa388b51a6f221 > docs/client-rest-api.md PRE-CREATION > src/main/python/apache/aurora/client/bin/BUILD > 43d747956df0611b0880f64df9955d5f5806901c > src/main/python/apache/aurora/client/bin/aurora_rest.py PRE-CREATION > src/main/python/apache/aurora/client/rest/BUILD PRE-CREATION > src/main/python/apache/aurora/client/rest/application.py PRE-CREATION > src/main/python/apache/aurora/client/rest/application_async.py PRE-CREATION > src/main/python/apache/aurora/client/rest/coroutine_executor.py > PRE-CREATION > src/main/python/apache/aurora/client/rest/external_executor.py PRE-CREATION > src/main/python/apache/aurora/client/rest/internal_executor.py PRE-CREATION > src/main/python/apache/aurora/client/rest/mp_executor.py PRE-CREATION > src/main/python/apache/aurora/client/rest/mt_executor.py PRE-CREATION > > Diff: https://reviews.apache.org/r/23666/diff/ > > > Testing > --- > > Stress testing with Locust to verify the REST service can handle multiple > requests simultaneously. Not all of them passed, particularly the ones with > the internal executor that calls Aurora client api. > > > Thanks, > > Misho Krastev > >
Re: Review Request 23666: REST service for aurora client commands, built with Tornado Web framework
> On July 22, 2014, 12:54 a.m., Bill Farner wrote: > > Do you think this should be obviated by https://reviews.apache.org/r/23741/ > > ? This looks like elegant (declarative) solution to export REST variant of the native thrift interface. I am not familiar with the Aurora scheduler code so I can not read very well the diffs. From the review request description and what I understand from the diffs, I would vote for that (your) solution. However, I don't think this REST service can be replaced by the other one. The reason is the REST service that I submitted for review is an interface to the capabilities provided by the Aurora client, and this client is functionality-rich. It adds more features on top of those offered by the scheduler. In particular, what we needed was the ability to do rolling upgrades where instances (shards) are upgraded in batches and not en bloc. There are other examples but this is the one that is of great importance to us. - Misho --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23666/#review48322 --- On July 21, 2014, 10:58 p.m., Misho Krastev wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/23666/ > --- > > (Updated July 21, 2014, 10:58 p.m.) > > > Review request for Aurora. > > > Repository: aurora > > > Description > --- > > REST service for aurora client commands, built using Tornado Web framework > > This is a draft implementation of a REST server exposing APIs to call Aurora > client commands. It is a wrapper around the aurora command-line client and > aurora client code. It implements some but not all of the commands available > with the aurora command-line client. Comments on the code as well suggestions > for improvements and enhancements are requested. > > This work is a eBay project that we would like to contribute to the Aurora > project. > > Note: development is still ongoing to add unittests and to resolve the issues > discovered with the stress tests. > Note(2): GitHub repo -- https://github.com/misho-kr/incubator-aurora > > > Diffs > - > > 3rdparty/python/BUILD 641d610deb58fd102ec584e3deaa388b51a6f221 > docs/client-rest-api.md PRE-CREATION > src/main/python/apache/aurora/client/bin/BUILD > 43d747956df0611b0880f64df9955d5f5806901c > src/main/python/apache/aurora/client/bin/aurora_rest.py PRE-CREATION > src/main/python/apache/aurora/client/rest/BUILD PRE-CREATION > src/main/python/apache/aurora/client/rest/application.py PRE-CREATION > src/main/python/apache/aurora/client/rest/application_async.py PRE-CREATION > src/main/python/apache/aurora/client/rest/coroutine_executor.py > PRE-CREATION > src/main/python/apache/aurora/client/rest/external_executor.py PRE-CREATION > src/main/python/apache/aurora/client/rest/internal_executor.py PRE-CREATION > src/main/python/apache/aurora/client/rest/mp_executor.py PRE-CREATION > src/main/python/apache/aurora/client/rest/mt_executor.py PRE-CREATION > > Diff: https://reviews.apache.org/r/23666/diff/ > > > Testing > --- > > Stress testing with Locust to verify the REST service can handle multiple > requests simultaneously. Not all of them passed, particularly the ones with > the internal executor that calls Aurora client api. > > > Thanks, > > Misho Krastev > >
Re: Review Request 23666: REST service for aurora client commands, built with Tornado Web framework
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23666/#review48322 --- Do you think this should be obviated by https://reviews.apache.org/r/23741/ ? - Bill Farner On July 21, 2014, 10:58 p.m., Misho Krastev wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/23666/ > --- > > (Updated July 21, 2014, 10:58 p.m.) > > > Review request for Aurora. > > > Repository: aurora > > > Description > --- > > REST service for aurora client commands, built using Tornado Web framework > > This is a draft implementation of a REST server exposing APIs to call Aurora > client commands. It is a wrapper around the aurora command-line client and > aurora client code. It implements some but not all of the commands available > with the aurora command-line client. Comments on the code as well suggestions > for improvements and enhancements are requested. > > This work is a eBay project that we would like to contribute to the Aurora > project. > > Note: development is still ongoing to add unittests and to resolve the issues > discovered with the stress tests. > Note(2): GitHub repo -- https://github.com/misho-kr/incubator-aurora > > > Diffs > - > > 3rdparty/python/BUILD 641d610deb58fd102ec584e3deaa388b51a6f221 > docs/client-rest-api.md PRE-CREATION > src/main/python/apache/aurora/client/bin/BUILD > 43d747956df0611b0880f64df9955d5f5806901c > src/main/python/apache/aurora/client/bin/aurora_rest.py PRE-CREATION > src/main/python/apache/aurora/client/rest/BUILD PRE-CREATION > src/main/python/apache/aurora/client/rest/application.py PRE-CREATION > src/main/python/apache/aurora/client/rest/application_async.py PRE-CREATION > src/main/python/apache/aurora/client/rest/coroutine_executor.py > PRE-CREATION > src/main/python/apache/aurora/client/rest/external_executor.py PRE-CREATION > src/main/python/apache/aurora/client/rest/internal_executor.py PRE-CREATION > src/main/python/apache/aurora/client/rest/mp_executor.py PRE-CREATION > src/main/python/apache/aurora/client/rest/mt_executor.py PRE-CREATION > > Diff: https://reviews.apache.org/r/23666/diff/ > > > Testing > --- > > Stress testing with Locust to verify the REST service can handle multiple > requests simultaneously. Not all of them passed, particularly the ones with > the internal executor that calls Aurora client api. > > > Thanks, > > Misho Krastev > >
Re: Review Request 23666: REST service for aurora client commands, built with Tornado Web framework
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23666/ --- (Updated July 21, 2014, 10:58 p.m.) Review request for Aurora. Changes --- Added link to Github repo -- https://github.com/misho-kr/incubator-aurora Repository: aurora Description (updated) --- REST service for aurora client commands, built using Tornado Web framework This is a draft implementation of a REST server exposing APIs to call Aurora client commands. It is a wrapper around the aurora command-line client and aurora client code. It implements some but not all of the commands available with the aurora command-line client. Comments on the code as well suggestions for improvements and enhancements are requested. This work is a eBay project that we would like to contribute to the Aurora project. Note: development is still ongoing to add unittests and to resolve the issues discovered with the stress tests. Note(2): GitHub repo -- https://github.com/misho-kr/incubator-aurora Diffs - 3rdparty/python/BUILD 641d610deb58fd102ec584e3deaa388b51a6f221 docs/client-rest-api.md PRE-CREATION src/main/python/apache/aurora/client/bin/BUILD 43d747956df0611b0880f64df9955d5f5806901c src/main/python/apache/aurora/client/bin/aurora_rest.py PRE-CREATION src/main/python/apache/aurora/client/rest/BUILD PRE-CREATION src/main/python/apache/aurora/client/rest/application.py PRE-CREATION src/main/python/apache/aurora/client/rest/application_async.py PRE-CREATION src/main/python/apache/aurora/client/rest/coroutine_executor.py PRE-CREATION src/main/python/apache/aurora/client/rest/external_executor.py PRE-CREATION src/main/python/apache/aurora/client/rest/internal_executor.py PRE-CREATION src/main/python/apache/aurora/client/rest/mp_executor.py PRE-CREATION src/main/python/apache/aurora/client/rest/mt_executor.py PRE-CREATION Diff: https://reviews.apache.org/r/23666/diff/ Testing --- Stress testing with Locust to verify the REST service can handle multiple requests simultaneously. Not all of them passed, particularly the ones with the internal executor that calls Aurora client api. Thanks, Misho Krastev
Re: Review Request 23666: REST service for aurora client commands, built with Tornado Web framework
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23666/ --- (Updated July 17, 2014, 11:16 p.m.) Review request for Aurora. Repository: aurora Description (updated) --- REST service for aurora client commands, built using Tornado Web framework This is a draft implementation of a REST server exposing APIs to call Aurora client commands. It is a wrapper around the aurora command-line client and aurora client code. It implements some but not all of the commands available with the aurora command-line client. Comments on the code as well suggestions for improvements and enhancements are requested. This work is a eBay project that we would like to contribute to the Aurora project. Note: development is still ongoing to add unittests and to resolve the issues discovered with the stress tests. Diffs - 3rdparty/python/BUILD 641d610deb58fd102ec584e3deaa388b51a6f221 docs/client-rest-api.md PRE-CREATION src/main/python/apache/aurora/client/bin/BUILD 43d747956df0611b0880f64df9955d5f5806901c src/main/python/apache/aurora/client/bin/aurora_rest.py PRE-CREATION src/main/python/apache/aurora/client/rest/BUILD PRE-CREATION src/main/python/apache/aurora/client/rest/application.py PRE-CREATION src/main/python/apache/aurora/client/rest/application_async.py PRE-CREATION src/main/python/apache/aurora/client/rest/coroutine_executor.py PRE-CREATION src/main/python/apache/aurora/client/rest/external_executor.py PRE-CREATION src/main/python/apache/aurora/client/rest/internal_executor.py PRE-CREATION src/main/python/apache/aurora/client/rest/mp_executor.py PRE-CREATION src/main/python/apache/aurora/client/rest/mt_executor.py PRE-CREATION Diff: https://reviews.apache.org/r/23666/diff/ Testing --- Stress testing with Locust to verify the REST service can handle multiple requests simultaneously. Not all of them passed, particularly the ones with the internal executor that calls Aurora client api. Thanks, Misho Krastev
