Re: New scheduling algorithm proposal.

2018-06-24 Thread TzuChiao Yeh
re Kafka > > > nodes, > > > > the number of partitions which is managed by one Kafka node would be > > > > unchanged. > > > > So if we can support 30K TPS with 1000 topics using 3 nodes, then we > > can > > > > still get 60K TPS with

Re: New scheduling algorithm proposal.

2018-06-08 Thread Dominic Kim
page > 121) > > - > > >> are these using different namespaces? I ask because I think the > current > > >> impl isolates the container per namespace, so if you are limited to > 180 > > >> containers, I can see how there will be container removal/restar

Re: New scheduling algorithm proposal.

2018-06-07 Thread TzuChiao Yeh
if there are any kafka experts here that can > >> provide more comments on the topics/partition handling question? I will > >> also ask for some additional feedback from other colleagues. > >> > >> > >> I will gather some more comments to share, bu

Re: New scheduling algorithm proposal.

2018-06-06 Thread Dominic Kim
ner get reused for many different >> namespaces?) >> >> >> I'm interested to know if there are any kafka experts here that can >> provide more comments on the topics/partition handling question? I will >> also ask for some additional feedback from other colleag

Re: New scheduling algorithm proposal.

2018-06-06 Thread Dominic Kim
wanted to start off with > these. Will continue next week after the long (US holiday) weekend. > > > Thanks > > Tyson > > > ____ > From: Dominic Kim > Sent: Friday, May 25, 2018 1:58:55 AM > To: dev@openwhisk.apache.org > Subject: Re

Re: New scheduling algorithm proposal.

2018-05-25 Thread Tyson Norris
nks Tyson From: Dominic Kim <style9...@gmail.com> Sent: Friday, May 25, 2018 1:58:55 AM To: dev@openwhisk.apache.org Subject: Re: New scheduling algorithm proposal. Dear Whiskers. I uploaded the material that I used to give a speech at last biweekly meetin

Re: New scheduling algorithm proposal.

2018-05-25 Thread Dominic Kim
with current implementation 3. New scheduling algorithm proposal: Autonomous Container Scheduling 4. Review previous issues with new scheduling algorithm 5. Pros and cons of new scheduling algorithm 6. Performance evaluation with prototyping I hope this is helpful for many Whiskers to understand the issues

Re: New scheduling algorithm proposal.

2018-05-18 Thread Dominic Kim
Hi Tyson Thank you for comments. First, total data size in Kafka would not be changed, since the number of activation requests will be same though we activate more topics. Same amount of data will be just split into different topics, so there would be no need for more disk space in Kafka. But it

Re: New scheduling algorithm proposal.

2018-05-16 Thread Dominic Kim
Dear all. Does anyone have any comments on this? Any comments or opinions would be greatly welcome : ) I think we need around following changes to take this in. 1. SPI supports for ContainerProxy and ContainerPool ( I already opened PR for this:

Re: New scheduling algorithm proposal.

2018-05-08 Thread Dominic Kim
Thank you for the response Markus and Christian. Yes I agree that we need to discuss this proposal in abstract way instead in conjunction it with any specific technology because we can take better software stack if possible. Let me answer your questions line by line. ## Does not wait for

Re: New scheduling algorithm proposal.

2018-05-08 Thread Markus Thoemmes
Hey Dominic, Thank you for the very detailed writeup. Since there is a lot in here, please allow me to rephrase some of your proposals to see if I understood correctly. I'll go through point-by-point to try to keep it close to your proposal. **Note:** This is a result of an extensive