Re: New scheduling algorithm proposal.

2018-06-24 Thread TzuChiao Yeh
re Kafka > > > nodes, > > > > the number of partitions which is managed by one Kafka node would be > > > > unchanged. > > > > So if we can support 30K TPS with 1000 topics using 3 nodes, then we > > can > > > > still get 60K TPS with

Re: New scheduling algorithm proposal.

2018-06-08 Thread Dominic Kim
Dominic - > > >> > > >> Thanks for the detailed presentation! It is helpful to go through to > > >> understand your points - well done. > > >> > > >> > > >> A couple of comments: > > >> > > >> - I'm

Re: New scheduling algorithm proposal.

2018-06-07 Thread TzuChiao Yeh
g different namespaces? I ask because I think the current > >> impl isolates the container per namespace, so if you are limited to 180 > >> containers, I can see how there will be container removal/restarts in > the > >> case where the number of users greatly outnumbers t

Re: New scheduling algorithm proposal.

2018-06-06 Thread Dominic Kim
ner get reused for many different >> namespaces?) >> >> >> I'm interested to know if there are any kafka experts here that can >> provide more comments on the topics/partition handling question? I will >> also ask for some additional feedback from other colleag

Re: New scheduling algorithm proposal.

2018-06-06 Thread Dominic Kim
wanted to start off with > these. Will continue next week after the long (US holiday) weekend. > > > Thanks > > Tyson > > > ____ > From: Dominic Kim > Sent: Friday, May 25, 2018 1:58:55 AM > To: dev@openwhisk.apache.org > Subject: Re

Re: New scheduling algorithm proposal.

2018-05-25 Thread Tyson Norris
nks Tyson From: Dominic Kim <style9...@gmail.com> Sent: Friday, May 25, 2018 1:58:55 AM To: dev@openwhisk.apache.org Subject: Re: New scheduling algorithm proposal. Dear Whiskers. I uploaded the material that I used to give a speech at last biweekly meetin

Re: New scheduling algorithm proposal.

2018-05-25 Thread Dominic Kim
Dear Whiskers. I uploaded the material that I used to give a speech at last biweekly meeting. https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduling This document mainly describes following things: 1. Current implementation details 2. Potential issues with

Re: New scheduling algorithm proposal.

2018-05-18 Thread Dominic Kim
Hi Tyson Thank you for comments. First, total data size in Kafka would not be changed, since the number of activation requests will be same though we activate more topics. Same amount of data will be just split into different topics, so there would be no need for more disk space in Kafka. But it

Re: New scheduling algorithm proposal.

2018-05-16 Thread Dominic Kim
Dear all. Does anyone have any comments on this? Any comments or opinions would be greatly welcome : ) I think we need around following changes to take this in. 1. SPI supports for ContainerProxy and ContainerPool ( I already opened PR for this:

Re: New scheduling algorithm proposal.

2018-05-08 Thread Dominic Kim
Thank you for the response Markus and Christian. Yes I agree that we need to discuss this proposal in abstract way instead in conjunction it with any specific technology because we can take better software stack if possible. Let me answer your questions line by line. ## Does not wait for

Re: New scheduling algorithm proposal.

2018-05-08 Thread Markus Thoemmes
Hey Dominic, Thank you for the very detailed writeup. Since there is a lot in here, please allow me to rephrase some of your proposals to see if I understood correctly. I'll go through point-by-point to try to keep it close to your proposal. **Note:** This is a result of an extensive