Re: Minio to copresent , quorum for next gen bigtop ?
Hiya youngwoo. Ok will do! I’ll make a PR into it today. > On Sep 23, 2019, at 1:16 AM, Youngwoo Kim (김영우) wrote: > > Hi, > > Let's make a step forward... I created an umbrella Jira to manage the > development issues https://issues.apache.org/jira/browse/BIGTOP-3225 > and also created the branch 'cnb' > https://gitbox.apache.org/repos/asf?p=bigtop.git;a=shortlog;h=refs/heads/cnb > > @Jay, > Please push your work into 'cnb' branch and then, let's find out what we > can improve. > > Thanks, > Youngwoo > > On Fri, Sep 20, 2019 at 8:39 PM Jay Vyas > wrote: > >> Btw ... kind or minikube is even easier than kubespray for small local >> testing . >> >>> On Sep 20, 2019, at 12:57 AM, Evans Ye wrote: >>> >>> +1 >>> >>> Youngwoo Kim (김영우) 於 2019年9月20日 週五 上午11:12寫道： >>> I did work through the snippet@Jay's repo on my end in using >> Kubespray[1] and vagrant VM. There are minor nits but overall it works well as expected... In my opinion, it's breaking changes for us because the tooling that >> we've used is not integrated(or replaced) yet. so I think it would be good to start with a development branch based on Jay's work... and also new >> branch is harmless on current development. I'm proposing a new branch for **incubating** cloud-native distribution which is based on Jay's ground work! What do you think? Thanks, Youngwoo 1. https://github.com/kubernetes-sigs/kubespray On Mon, Sep 16, 2019 at 8:02 PM Jay Vyas wrote: > - No need for our own k8s. Let’s just test with nfs volume provisioned > and note what version of nfs, networking, helm, and cluster were using to > test releases . > > - I think since we are using helm charts , we can add anything and >> curate > and test. Cost of packaging with helm and docker is very low. > > - If you go to the repo there is a 20 line snippet we can start with on > the readme that sets up Kafka spark NiFi zk minio and presto. > >> On Sep 15, 2019, at 10:10 PM, Youngwoo Kim (김영우) > wrote: >> >> Good job Jay & Sid. Nice work! very informative and also thanks Jay >> for sharing the stuffs. >> >> I have a couple of questions and suggestions: >> - Do we need our own k8s distribution? (pros vs. cons) >> - k8s is complex and also people love (easy)UI :-) How can we reduce the >> gap? >> - How can we test the distribution? (integrity and performance?) >> - Rest of the elements?, Such as logging, metrics & security. >> - How do we manage base container images? official images or github? >> >> As a distribution for big data and analytics, it would be nice to have > such >> components(already... there are nice projects in ASF :-)): >> - Jupyter Notebook and jupyterhub >> - More choices on stream data processing: Flink, Pulsar >> - Task scheduling: Airflow >> - Interactive analytics and dashboard: Superset >> - Functions(serverless): OpenWhisk or KNative >> - Purpose-built data store for big data: Redis, Cassandra and Druid. >> - and more... >> >> WDYT? >> >> Thanks, >> Youngwoo >> >> On Sun, Sep 15, 2019 at 2:07 AM jay vyas >> >> wrote: >> >>> Hi everyone ! >>> >>> The contents for this that were finally presented are here... >>> https://github.com/jayunit100/apachecon-2019-bigtop3x >>> >>> and the video is here ! https://youtu.be/LUCE63qPMyM >>> >>> Let us know what you think >>> >>> >>> On Thu, Sep 12, 2019 at 4:35 PM jay vyas < >> jayunit100.apa...@gmail.com > >>> wrote: >>> hi folks, am in the conference lobby area now. have 2 laptops out >> so >>> easy to find :) . . On Thu, Sep 12, 2019 at 2:41 PM Konstantin Boudnik >>> wrote: > Well, here's an interesting piece of news: > > > >>> > >> https://www.theregister.co.uk/2019/09/10/google_cloud_lights_spark_for_kubernetes/ > > Mind you, these nice fellas [/sarc] promised on two different > occasions > that > they will be contributing back to Apache Bigtop - the foundation of > the > Dataproc - and yet they never did. But I guess this is fine ... >> not > the > first > company taking this kind of advantage :) > > -- > Cos > > >> On Wed, Sep 04, 2019 at 05:54AM, Jay Vyas wrote: >> Hi folks. Couple of updateys in the next gen brainstorming we’ve >>> been > doing... >> >> Sid Mani from minio is going to copresent with me at our bigtop talk >>> at >> apachecon next week, around the internals of minio integrations etc. >> >> Spoke with cos and sounds
Re: Minio to copresent , quorum for next gen bigtop ?
Hi, Let's make a step forward... I created an umbrella Jira to manage the development issues https://issues.apache.org/jira/browse/BIGTOP-3225 and also created the branch 'cnb' https://gitbox.apache.org/repos/asf?p=bigtop.git;a=shortlog;h=refs/heads/cnb @Jay, Please push your work into 'cnb' branch and then, let's find out what we can improve. Thanks, Youngwoo On Fri, Sep 20, 2019 at 8:39 PM Jay Vyas wrote: > Btw ... kind or minikube is even easier than kubespray for small local > testing . > > > On Sep 20, 2019, at 12:57 AM, Evans Ye wrote: > > > > +1 > > > > Youngwoo Kim (김영우) 於 2019年9月20日 週五 上午11:12寫道： > > > >> I did work through the snippet@Jay's repo on my end in using > Kubespray[1] > >> and vagrant VM. There are minor nits but overall it works well as > >> expected... > >> In my opinion, it's breaking changes for us because the tooling that > we've > >> used is not integrated(or replaced) yet. so I think it would be good to > >> start with a development branch based on Jay's work... and also new > branch > >> is harmless on current development. > >> > >> I'm proposing a new branch for **incubating** cloud-native distribution > >> which is based on Jay's ground work! > >> > >> What do you think? > >> > >> Thanks, > >> Youngwoo > >> > >> 1. https://github.com/kubernetes-sigs/kubespray > >> > >> On Mon, Sep 16, 2019 at 8:02 PM Jay Vyas > >> wrote: > >> > >>> - No need for our own k8s. Let’s just test with nfs volume provisioned > >>> and note what version of nfs, networking, helm, and cluster were using > >> to > >>> test releases . > >>> > >>> - I think since we are using helm charts , we can add anything and > curate > >>> and test. Cost of packaging with helm and docker is very low. > >>> > >>> - If you go to the repo there is a 20 line snippet we can start with on > >>> the readme that sets up Kafka spark NiFi zk minio and presto. > >>> > On Sep 15, 2019, at 10:10 PM, Youngwoo Kim (김영우) > >>> wrote: > > Good job Jay & Sid. Nice work! very informative and also thanks Jay > for sharing the stuffs. > > I have a couple of questions and suggestions: > - Do we need our own k8s distribution? (pros vs. cons) > - k8s is complex and also people love (easy)UI :-) How can we reduce > >> the > gap? > - How can we test the distribution? (integrity and performance?) > - Rest of the elements?, Such as logging, metrics & security. > - How do we manage base container images? official images or github? > > As a distribution for big data and analytics, it would be nice to have > >>> such > components(already... there are nice projects in ASF :-)): > - Jupyter Notebook and jupyterhub > - More choices on stream data processing: Flink, Pulsar > - Task scheduling: Airflow > - Interactive analytics and dashboard: Superset > - Functions(serverless): OpenWhisk or KNative > - Purpose-built data store for big data: Redis, Cassandra and Druid. > - and more... > > WDYT? > > Thanks, > Youngwoo > > On Sun, Sep 15, 2019 at 2:07 AM jay vyas > > wrote: > > > Hi everyone ! > > > > The contents for this that were finally presented are here... > > https://github.com/jayunit100/apachecon-2019-bigtop3x > > > > and the video is here ! https://youtu.be/LUCE63qPMyM > > > > Let us know what you think > > > > > > On Thu, Sep 12, 2019 at 4:35 PM jay vyas < > jayunit100.apa...@gmail.com > >>> > > wrote: > > > >> hi folks, am in the conference lobby area now. have 2 laptops out > so > > easy > >> to find :) . . > >> > >> > >> On Thu, Sep 12, 2019 at 2:41 PM Konstantin Boudnik > > wrote: > >> > >>> Well, here's an interesting piece of news: > >>> > >>> > >>> > > > >>> > >> > https://www.theregister.co.uk/2019/09/10/google_cloud_lights_spark_for_kubernetes/ > >>> > >>> Mind you, these nice fellas [/sarc] promised on two different > >>> occasions > >>> that > >>> they will be contributing back to Apache Bigtop - the foundation of > >>> the > >>> Dataproc - and yet they never did. But I guess this is fine ... > not > >>> the > >>> first > >>> company taking this kind of advantage :) > >>> > >>> -- > >>> Cos > >>> > >>> > On Wed, Sep 04, 2019 at 05:54AM, Jay Vyas wrote: > Hi folks. Couple of updateys in the next gen brainstorming we’ve > > been > >>> doing... > > Sid Mani from minio is going to copresent with me at our bigtop > >> talk > > at > apachecon next week, around the internals of minio integrations > >> etc. > > Spoke with cos and sounds like we definetly have a quorum for > >> hacking > >>> on this next gen implementation. > > Whose in on hacking on it, and who will be in Vegas on the 12th? > > I’ll
Re: Minio to copresent , quorum for next gen bigtop ?
Btw ... kind or minikube is even easier than kubespray for small local testing . > On Sep 20, 2019, at 12:57 AM, Evans Ye wrote: > > +1 > > Youngwoo Kim (김영우) 於 2019年9月20日 週五 上午11:12寫道： > >> I did work through the snippet@Jay's repo on my end in using Kubespray[1] >> and vagrant VM. There are minor nits but overall it works well as >> expected... >> In my opinion, it's breaking changes for us because the tooling that we've >> used is not integrated(or replaced) yet. so I think it would be good to >> start with a development branch based on Jay's work... and also new branch >> is harmless on current development. >> >> I'm proposing a new branch for **incubating** cloud-native distribution >> which is based on Jay's ground work! >> >> What do you think? >> >> Thanks, >> Youngwoo >> >> 1. https://github.com/kubernetes-sigs/kubespray >> >> On Mon, Sep 16, 2019 at 8:02 PM Jay Vyas >> wrote: >> >>> - No need for our own k8s. Let’s just test with nfs volume provisioned >>> and note what version of nfs, networking, helm, and cluster were using >> to >>> test releases . >>> >>> - I think since we are using helm charts , we can add anything and curate >>> and test. Cost of packaging with helm and docker is very low. >>> >>> - If you go to the repo there is a 20 line snippet we can start with on >>> the readme that sets up Kafka spark NiFi zk minio and presto. >>> On Sep 15, 2019, at 10:10 PM, Youngwoo Kim (김영우) >>> wrote: Good job Jay & Sid. Nice work! very informative and also thanks Jay for sharing the stuffs. I have a couple of questions and suggestions: - Do we need our own k8s distribution? (pros vs. cons) - k8s is complex and also people love (easy)UI :-) How can we reduce >> the gap? - How can we test the distribution? (integrity and performance?) - Rest of the elements?, Such as logging, metrics & security. - How do we manage base container images? official images or github? As a distribution for big data and analytics, it would be nice to have >>> such components(already... there are nice projects in ASF :-)): - Jupyter Notebook and jupyterhub - More choices on stream data processing: Flink, Pulsar - Task scheduling: Airflow - Interactive analytics and dashboard: Superset - Functions(serverless): OpenWhisk or KNative - Purpose-built data store for big data: Redis, Cassandra and Druid. - and more... WDYT? Thanks, Youngwoo On Sun, Sep 15, 2019 at 2:07 AM jay vyas wrote: > Hi everyone ! > > The contents for this that were finally presented are here... > https://github.com/jayunit100/apachecon-2019-bigtop3x > > and the video is here ! https://youtu.be/LUCE63qPMyM > > Let us know what you think > > > On Thu, Sep 12, 2019 at 4:35 PM jay vyas >> > wrote: > >> hi folks, am in the conference lobby area now. have 2 laptops out so > easy >> to find :) . . >> >> >> On Thu, Sep 12, 2019 at 2:41 PM Konstantin Boudnik > wrote: >> >>> Well, here's an interesting piece of news: >>> >>> >>> > >>> >> https://www.theregister.co.uk/2019/09/10/google_cloud_lights_spark_for_kubernetes/ >>> >>> Mind you, these nice fellas [/sarc] promised on two different >>> occasions >>> that >>> they will be contributing back to Apache Bigtop - the foundation of >>> the >>> Dataproc - and yet they never did. But I guess this is fine ... not >>> the >>> first >>> company taking this kind of advantage :) >>> >>> -- >>> Cos >>> >>> On Wed, Sep 04, 2019 at 05:54AM, Jay Vyas wrote: Hi folks. Couple of updateys in the next gen brainstorming we’ve > been >>> doing... Sid Mani from minio is going to copresent with me at our bigtop >> talk > at apachecon next week, around the internals of minio integrations >> etc. Spoke with cos and sounds like we definetly have a quorum for >> hacking >>> on this next gen implementation. Whose in on hacking on it, and who will be in Vegas on the 12th? I’ll help w the k8s stuff so don’t be intimidated if that’s your >> weak >>> spot. >>> >> >> >> -- >> jay vyas >> > > > -- > jay vyas > >>> >>
Re: Minio to copresent , quorum for next gen bigtop ?
Thanks youngwoo! Ya let’s just set a branch called “cnb” for “ cloud native bigtop” . That will give a nice ring to it when we tell people about it at (try bigtop CNB if you have k8s). And we just should just push to it for now I think. Thanks so much for testing that’s awesome . > On Sep 20, 2019, at 12:57 AM, Evans Ye wrote: > > +1 > > Youngwoo Kim (김영우) 於 2019年9月20日 週五 上午11:12寫道： > >> I did work through the snippet@Jay's repo on my end in using Kubespray[1] >> and vagrant VM. There are minor nits but overall it works well as >> expected... >> In my opinion, it's breaking changes for us because the tooling that we've >> used is not integrated(or replaced) yet. so I think it would be good to >> start with a development branch based on Jay's work... and also new branch >> is harmless on current development. >> >> I'm proposing a new branch for **incubating** cloud-native distribution >> which is based on Jay's ground work! >> >> What do you think? >> >> Thanks, >> Youngwoo >> >> 1. https://github.com/kubernetes-sigs/kubespray >> >> On Mon, Sep 16, 2019 at 8:02 PM Jay Vyas >> wrote: >> >>> - No need for our own k8s. Let’s just test with nfs volume provisioned >>> and note what version of nfs, networking, helm, and cluster were using >> to >>> test releases . >>> >>> - I think since we are using helm charts , we can add anything and curate >>> and test. Cost of packaging with helm and docker is very low. >>> >>> - If you go to the repo there is a 20 line snippet we can start with on >>> the readme that sets up Kafka spark NiFi zk minio and presto. >>> On Sep 15, 2019, at 10:10 PM, Youngwoo Kim (김영우) >>> wrote: Good job Jay & Sid. Nice work! very informative and also thanks Jay for sharing the stuffs. I have a couple of questions and suggestions: - Do we need our own k8s distribution? (pros vs. cons) - k8s is complex and also people love (easy)UI :-) How can we reduce >> the gap? - How can we test the distribution? (integrity and performance?) - Rest of the elements?, Such as logging, metrics & security. - How do we manage base container images? official images or github? As a distribution for big data and analytics, it would be nice to have >>> such components(already... there are nice projects in ASF :-)): - Jupyter Notebook and jupyterhub - More choices on stream data processing: Flink, Pulsar - Task scheduling: Airflow - Interactive analytics and dashboard: Superset - Functions(serverless): OpenWhisk or KNative - Purpose-built data store for big data: Redis, Cassandra and Druid. - and more... WDYT? Thanks, Youngwoo On Sun, Sep 15, 2019 at 2:07 AM jay vyas wrote: > Hi everyone ! > > The contents for this that were finally presented are here... > https://github.com/jayunit100/apachecon-2019-bigtop3x > > and the video is here ! https://youtu.be/LUCE63qPMyM > > Let us know what you think > > > On Thu, Sep 12, 2019 at 4:35 PM jay vyas >> > wrote: > >> hi folks, am in the conference lobby area now. have 2 laptops out so > easy >> to find :) . . >> >> >> On Thu, Sep 12, 2019 at 2:41 PM Konstantin Boudnik > wrote: >> >>> Well, here's an interesting piece of news: >>> >>> >>> > >>> >> https://www.theregister.co.uk/2019/09/10/google_cloud_lights_spark_for_kubernetes/ >>> >>> Mind you, these nice fellas [/sarc] promised on two different >>> occasions >>> that >>> they will be contributing back to Apache Bigtop - the foundation of >>> the >>> Dataproc - and yet they never did. But I guess this is fine ... not >>> the >>> first >>> company taking this kind of advantage :) >>> >>> -- >>> Cos >>> >>> On Wed, Sep 04, 2019 at 05:54AM, Jay Vyas wrote: Hi folks. Couple of updateys in the next gen brainstorming we’ve > been >>> doing... Sid Mani from minio is going to copresent with me at our bigtop >> talk > at apachecon next week, around the internals of minio integrations >> etc. Spoke with cos and sounds like we definetly have a quorum for >> hacking >>> on this next gen implementation. Whose in on hacking on it, and who will be in Vegas on the 12th? I’ll help w the k8s stuff so don’t be intimidated if that’s your >> weak >>> spot. >>> >> >> >> -- >> jay vyas >> > > > -- > jay vyas > >>> >>
Re: Minio to copresent , quorum for next gen bigtop ?
+1 Youngwoo Kim (김영우) 於 2019年9月20日 週五 上午11:12寫道： > I did work through the snippet@Jay's repo on my end in using Kubespray[1] > and vagrant VM. There are minor nits but overall it works well as > expected... > In my opinion, it's breaking changes for us because the tooling that we've > used is not integrated(or replaced) yet. so I think it would be good to > start with a development branch based on Jay's work... and also new branch > is harmless on current development. > > I'm proposing a new branch for **incubating** cloud-native distribution > which is based on Jay's ground work! > > What do you think? > > Thanks, > Youngwoo > > 1. https://github.com/kubernetes-sigs/kubespray > > On Mon, Sep 16, 2019 at 8:02 PM Jay Vyas > wrote: > > > - No need for our own k8s. Let’s just test with nfs volume provisioned > > and note what version of nfs, networking, helm, and cluster were using > to > > test releases . > > > > - I think since we are using helm charts , we can add anything and curate > > and test. Cost of packaging with helm and docker is very low. > > > > - If you go to the repo there is a 20 line snippet we can start with on > > the readme that sets up Kafka spark NiFi zk minio and presto. > > > > > On Sep 15, 2019, at 10:10 PM, Youngwoo Kim (김영우) > > wrote: > > > > > > Good job Jay & Sid. Nice work! very informative and also thanks Jay > > > for sharing the stuffs. > > > > > > I have a couple of questions and suggestions: > > > - Do we need our own k8s distribution? (pros vs. cons) > > > - k8s is complex and also people love (easy)UI :-) How can we reduce > the > > > gap? > > > - How can we test the distribution? (integrity and performance?) > > > - Rest of the elements?, Such as logging, metrics & security. > > > - How do we manage base container images? official images or github? > > > > > > As a distribution for big data and analytics, it would be nice to have > > such > > > components(already... there are nice projects in ASF :-)): > > > - Jupyter Notebook and jupyterhub > > > - More choices on stream data processing: Flink, Pulsar > > > - Task scheduling: Airflow > > > - Interactive analytics and dashboard: Superset > > > - Functions(serverless): OpenWhisk or KNative > > > - Purpose-built data store for big data: Redis, Cassandra and Druid. > > > - and more... > > > > > > WDYT? > > > > > > Thanks, > > > Youngwoo > > > > > > On Sun, Sep 15, 2019 at 2:07 AM jay vyas > > > wrote: > > > > > >> Hi everyone ! > > >> > > >> The contents for this that were finally presented are here... > > >> https://github.com/jayunit100/apachecon-2019-bigtop3x > > >> > > >> and the video is here ! https://youtu.be/LUCE63qPMyM > > >> > > >> Let us know what you think > > >> > > >> > > >> On Thu, Sep 12, 2019 at 4:35 PM jay vyas > > > >> wrote: > > >> > > >>> hi folks, am in the conference lobby area now. have 2 laptops out so > > >> easy > > >>> to find :) . . > > >>> > > >>> > > >>> On Thu, Sep 12, 2019 at 2:41 PM Konstantin Boudnik > > >> wrote: > > >>> > > Well, here's an interesting piece of news: > > > > > > > > >> > > > https://www.theregister.co.uk/2019/09/10/google_cloud_lights_spark_for_kubernetes/ > > > > Mind you, these nice fellas [/sarc] promised on two different > > occasions > > that > > they will be contributing back to Apache Bigtop - the foundation of > > the > > Dataproc - and yet they never did. But I guess this is fine ... not > > the > > first > > company taking this kind of advantage :) > > > > -- > > Cos > > > > > > > On Wed, Sep 04, 2019 at 05:54AM, Jay Vyas wrote: > > > Hi folks. Couple of updateys in the next gen brainstorming we’ve > > >> been > > doing... > > > > > > Sid Mani from minio is going to copresent with me at our bigtop > talk > > >> at > > > apachecon next week, around the internals of minio integrations > etc. > > > > > > Spoke with cos and sounds like we definetly have a quorum for > hacking > > on this next gen implementation. > > > > > > Whose in on hacking on it, and who will be in Vegas on the 12th? > > > > > > I’ll help w the k8s stuff so don’t be intimidated if that’s your > weak > > spot. > > > > >>> > > >>> > > >>> -- > > >>> jay vyas > > >>> > > >> > > >> > > >> -- > > >> jay vyas > > >> > > >
Re: Minio to copresent , quorum for next gen bigtop ?
I did work through the snippet@Jay's repo on my end in using Kubespray[1] and vagrant VM. There are minor nits but overall it works well as expected... In my opinion, it's breaking changes for us because the tooling that we've used is not integrated(or replaced) yet. so I think it would be good to start with a development branch based on Jay's work... and also new branch is harmless on current development. I'm proposing a new branch for **incubating** cloud-native distribution which is based on Jay's ground work! What do you think? Thanks, Youngwoo 1. https://github.com/kubernetes-sigs/kubespray On Mon, Sep 16, 2019 at 8:02 PM Jay Vyas wrote: > - No need for our own k8s. Let’s just test with nfs volume provisioned > and note what version of nfs, networking, helm, and cluster were using to > test releases . > > - I think since we are using helm charts , we can add anything and curate > and test. Cost of packaging with helm and docker is very low. > > - If you go to the repo there is a 20 line snippet we can start with on > the readme that sets up Kafka spark NiFi zk minio and presto. > > > On Sep 15, 2019, at 10:10 PM, Youngwoo Kim (김영우) > wrote: > > > > Good job Jay & Sid. Nice work! very informative and also thanks Jay > > for sharing the stuffs. > > > > I have a couple of questions and suggestions: > > - Do we need our own k8s distribution? (pros vs. cons) > > - k8s is complex and also people love (easy)UI :-) How can we reduce the > > gap? > > - How can we test the distribution? (integrity and performance?) > > - Rest of the elements?, Such as logging, metrics & security. > > - How do we manage base container images? official images or github? > > > > As a distribution for big data and analytics, it would be nice to have > such > > components(already... there are nice projects in ASF :-)): > > - Jupyter Notebook and jupyterhub > > - More choices on stream data processing: Flink, Pulsar > > - Task scheduling: Airflow > > - Interactive analytics and dashboard: Superset > > - Functions(serverless): OpenWhisk or KNative > > - Purpose-built data store for big data: Redis, Cassandra and Druid. > > - and more... > > > > WDYT? > > > > Thanks, > > Youngwoo > > > > On Sun, Sep 15, 2019 at 2:07 AM jay vyas > > wrote: > > > >> Hi everyone ! > >> > >> The contents for this that were finally presented are here... > >> https://github.com/jayunit100/apachecon-2019-bigtop3x > >> > >> and the video is here ! https://youtu.be/LUCE63qPMyM > >> > >> Let us know what you think > >> > >> > >> On Thu, Sep 12, 2019 at 4:35 PM jay vyas > >> wrote: > >> > >>> hi folks, am in the conference lobby area now. have 2 laptops out so > >> easy > >>> to find :) . . > >>> > >>> > >>> On Thu, Sep 12, 2019 at 2:41 PM Konstantin Boudnik > >> wrote: > >>> > Well, here's an interesting piece of news: > > > > >> > https://www.theregister.co.uk/2019/09/10/google_cloud_lights_spark_for_kubernetes/ > > Mind you, these nice fellas [/sarc] promised on two different > occasions > that > they will be contributing back to Apache Bigtop - the foundation of > the > Dataproc - and yet they never did. But I guess this is fine ... not > the > first > company taking this kind of advantage :) > > -- > Cos > > > > On Wed, Sep 04, 2019 at 05:54AM, Jay Vyas wrote: > > Hi folks. Couple of updateys in the next gen brainstorming we’ve > >> been > doing... > > > > Sid Mani from minio is going to copresent with me at our bigtop talk > >> at > > apachecon next week, around the internals of minio integrations etc. > > > > Spoke with cos and sounds like we definetly have a quorum for hacking > on this next gen implementation. > > > > Whose in on hacking on it, and who will be in Vegas on the 12th? > > > > I’ll help w the k8s stuff so don’t be intimidated if that’s your weak > spot. > > >>> > >>> > >>> -- > >>> jay vyas > >>> > >> > >> > >> -- > >> jay vyas > >> >
Re: Minio to copresent , quorum for next gen bigtop ?
- No need for our own k8s. Let’s just test with nfs volume provisioned and note what version of nfs, networking, helm, and cluster were using to test releases . - I think since we are using helm charts , we can add anything and curate and test. Cost of packaging with helm and docker is very low. - If you go to the repo there is a 20 line snippet we can start with on the readme that sets up Kafka spark NiFi zk minio and presto. > On Sep 15, 2019, at 10:10 PM, Youngwoo Kim (김영우) wrote: > > Good job Jay & Sid. Nice work! very informative and also thanks Jay > for sharing the stuffs. > > I have a couple of questions and suggestions: > - Do we need our own k8s distribution? (pros vs. cons) > - k8s is complex and also people love (easy)UI :-) How can we reduce the > gap? > - How can we test the distribution? (integrity and performance?) > - Rest of the elements?, Such as logging, metrics & security. > - How do we manage base container images? official images or github? > > As a distribution for big data and analytics, it would be nice to have such > components(already... there are nice projects in ASF :-)): > - Jupyter Notebook and jupyterhub > - More choices on stream data processing: Flink, Pulsar > - Task scheduling: Airflow > - Interactive analytics and dashboard: Superset > - Functions(serverless): OpenWhisk or KNative > - Purpose-built data store for big data: Redis, Cassandra and Druid. > - and more... > > WDYT? > > Thanks, > Youngwoo > > On Sun, Sep 15, 2019 at 2:07 AM jay vyas > wrote: > >> Hi everyone ! >> >> The contents for this that were finally presented are here... >> https://github.com/jayunit100/apachecon-2019-bigtop3x >> >> and the video is here ! https://youtu.be/LUCE63qPMyM >> >> Let us know what you think >> >> >> On Thu, Sep 12, 2019 at 4:35 PM jay vyas >> wrote: >> >>> hi folks, am in the conference lobby area now. have 2 laptops out so >> easy >>> to find :) . . >>> >>> >>> On Thu, Sep 12, 2019 at 2:41 PM Konstantin Boudnik >> wrote: >>> Well, here's an interesting piece of news: >> https://www.theregister.co.uk/2019/09/10/google_cloud_lights_spark_for_kubernetes/ Mind you, these nice fellas [/sarc] promised on two different occasions that they will be contributing back to Apache Bigtop - the foundation of the Dataproc - and yet they never did. But I guess this is fine ... not the first company taking this kind of advantage :) -- Cos > On Wed, Sep 04, 2019 at 05:54AM, Jay Vyas wrote: > Hi folks. Couple of updateys in the next gen brainstorming we’ve >> been doing... > > Sid Mani from minio is going to copresent with me at our bigtop talk >> at > apachecon next week, around the internals of minio integrations etc. > > Spoke with cos and sounds like we definetly have a quorum for hacking on this next gen implementation. > > Whose in on hacking on it, and who will be in Vegas on the 12th? > > I’ll help w the k8s stuff so don’t be intimidated if that’s your weak spot. >>> >>> >>> -- >>> jay vyas >>> >> >> >> -- >> jay vyas >>
Re: Minio to copresent , quorum for next gen bigtop ?
Good job Jay & Sid. Nice work! very informative and also thanks Jay for sharing the stuffs. I have a couple of questions and suggestions: - Do we need our own k8s distribution? (pros vs. cons) - k8s is complex and also people love (easy)UI :-) How can we reduce the gap? - How can we test the distribution? (integrity and performance?) - Rest of the elements?, Such as logging, metrics & security. - How do we manage base container images? official images or github? As a distribution for big data and analytics, it would be nice to have such components(already... there are nice projects in ASF :-)): - Jupyter Notebook and jupyterhub - More choices on stream data processing: Flink, Pulsar - Task scheduling: Airflow - Interactive analytics and dashboard: Superset - Functions(serverless): OpenWhisk or KNative - Purpose-built data store for big data: Redis, Cassandra and Druid. - and more... WDYT? Thanks, Youngwoo On Sun, Sep 15, 2019 at 2:07 AM jay vyas wrote: > Hi everyone ! > > The contents for this that were finally presented are here... > https://github.com/jayunit100/apachecon-2019-bigtop3x > > and the video is here ! https://youtu.be/LUCE63qPMyM > > Let us know what you think > > > On Thu, Sep 12, 2019 at 4:35 PM jay vyas > wrote: > > > hi folks, am in the conference lobby area now. have 2 laptops out so > easy > > to find :) . . > > > > > > On Thu, Sep 12, 2019 at 2:41 PM Konstantin Boudnik > wrote: > > > >> Well, here's an interesting piece of news: > >> > >> > >> > https://www.theregister.co.uk/2019/09/10/google_cloud_lights_spark_for_kubernetes/ > >> > >> Mind you, these nice fellas [/sarc] promised on two different occasions > >> that > >> they will be contributing back to Apache Bigtop - the foundation of the > >> Dataproc - and yet they never did. But I guess this is fine ... not the > >> first > >> company taking this kind of advantage :) > >> > >> -- > >> Cos > >> > >> > >> On Wed, Sep 04, 2019 at 05:54AM, Jay Vyas wrote: > >> > Hi folks. Couple of updateys in the next gen brainstorming we’ve > been > >> doing... > >> > > >> > Sid Mani from minio is going to copresent with me at our bigtop talk > at > >> > apachecon next week, around the internals of minio integrations etc. > >> > > >> > Spoke with cos and sounds like we definetly have a quorum for hacking > >> on this next gen implementation. > >> > > >> > Whose in on hacking on it, and who will be in Vegas on the 12th? > >> > > >> > I’ll help w the k8s stuff so don’t be intimidated if that’s your weak > >> spot. > >> > > > > > > -- > > jay vyas > > > > > -- > jay vyas >
Re: Minio to copresent , quorum for next gen bigtop ?
Hi everyone ! The contents for this that were finally presented are here... https://github.com/jayunit100/apachecon-2019-bigtop3x and the video is here ! https://youtu.be/LUCE63qPMyM Let us know what you think On Thu, Sep 12, 2019 at 4:35 PM jay vyas wrote: > hi folks, am in the conference lobby area now. have 2 laptops out so easy > to find :) . . > > > On Thu, Sep 12, 2019 at 2:41 PM Konstantin Boudnik wrote: > >> Well, here's an interesting piece of news: >> >> >> https://www.theregister.co.uk/2019/09/10/google_cloud_lights_spark_for_kubernetes/ >> >> Mind you, these nice fellas [/sarc] promised on two different occasions >> that >> they will be contributing back to Apache Bigtop - the foundation of the >> Dataproc - and yet they never did. But I guess this is fine ... not the >> first >> company taking this kind of advantage :) >> >> -- >> Cos >> >> >> On Wed, Sep 04, 2019 at 05:54AM, Jay Vyas wrote: >> > Hi folks. Couple of updateys in the next gen brainstorming we’ve been >> doing... >> > >> > Sid Mani from minio is going to copresent with me at our bigtop talk at >> > apachecon next week, around the internals of minio integrations etc. >> > >> > Spoke with cos and sounds like we definetly have a quorum for hacking >> on this next gen implementation. >> > >> > Whose in on hacking on it, and who will be in Vegas on the 12th? >> > >> > I’ll help w the k8s stuff so don’t be intimidated if that’s your weak >> spot. >> > > > -- > jay vyas > -- jay vyas
Re: Minio to copresent , quorum for next gen bigtop ?
hi folks, am in the conference lobby area now. have 2 laptops out so easy to find :) . . On Thu, Sep 12, 2019 at 2:41 PM Konstantin Boudnik wrote: > Well, here's an interesting piece of news: > > > https://www.theregister.co.uk/2019/09/10/google_cloud_lights_spark_for_kubernetes/ > > Mind you, these nice fellas [/sarc] promised on two different occasions > that > they will be contributing back to Apache Bigtop - the foundation of the > Dataproc - and yet they never did. But I guess this is fine ... not the > first > company taking this kind of advantage :) > > -- > Cos > > > On Wed, Sep 04, 2019 at 05:54AM, Jay Vyas wrote: > > Hi folks. Couple of updateys in the next gen brainstorming we’ve been > doing... > > > > Sid Mani from minio is going to copresent with me at our bigtop talk at > > apachecon next week, around the internals of minio integrations etc. > > > > Spoke with cos and sounds like we definetly have a quorum for hacking on > this next gen implementation. > > > > Whose in on hacking on it, and who will be in Vegas on the 12th? > > > > I’ll help w the k8s stuff so don’t be intimidated if that’s your weak > spot. > -- jay vyas
Re: Minio to copresent , quorum for next gen bigtop ?
Well, here's an interesting piece of news: https://www.theregister.co.uk/2019/09/10/google_cloud_lights_spark_for_kubernetes/ Mind you, these nice fellas [/sarc] promised on two different occasions that they will be contributing back to Apache Bigtop - the foundation of the Dataproc - and yet they never did. But I guess this is fine ... not the first company taking this kind of advantage :) -- Cos On Wed, Sep 04, 2019 at 05:54AM, Jay Vyas wrote: > Hi folks. Couple of updateys in the next gen brainstorming we’ve been > doing... > > Sid Mani from minio is going to copresent with me at our bigtop talk at > apachecon next week, around the internals of minio integrations etc. > > Spoke with cos and sounds like we definetly have a quorum for hacking on this > next gen implementation. > > Whose in on hacking on it, and who will be in Vegas on the 12th? > > I’ll help w the k8s stuff so don’t be intimidated if that’s your weak spot. signature.asc Description: Digital signature
Re: Minio to copresent , quorum for next gen bigtop ?
I will be there. Ganesh Sent from my iPhone > On Sep 8, 2019, at 9:20 AM, Evans Ye wrote: > > huge +1. I'd be there. Count me in. > > Youngwoo Kim (김영우) 於 2019年9月5日 週四 上午10:46寫道： > >> Jay, >> I believe It would be a great collaboration! I've been using Minio for last >> 6 months to do my day job, It's great and reliable. >> >> Unfortunately, I can't be there because my trip to Vegas was cancelled. :-( >> I'll spend time with my family during Korean Thanksgiving Day. >> >> Happy hacking! >> >> Thanks, >> Youngwoo >> >> >> On Wed, Sep 4, 2019 at 6:55 PM Jay Vyas >> wrote: >> >>> Hi folks. Couple of updateys in the next gen brainstorming we’ve been >>> doing... >>> >>> >>> Sid Mani from minio is going to copresent with me at our bigtop talk at >>> apachecon next week, around the internals of minio integrations etc. >>> >>> Spoke with cos and sounds like we definetly have a quorum for hacking on >>> this next gen implementation. >>> >>> Whose in on hacking on it, and who will be in Vegas on the 12th? >>> >>> I’ll help w the k8s stuff so don’t be intimidated if that’s your weak >> spot. >>
Re: Minio to copresent , quorum for next gen bigtop ?
huge +1. I'd be there. Count me in. Youngwoo Kim (김영우) 於 2019年9月5日 週四 上午10:46寫道： > Jay, > I believe It would be a great collaboration! I've been using Minio for last > 6 months to do my day job, It's great and reliable. > > Unfortunately, I can't be there because my trip to Vegas was cancelled. :-( > I'll spend time with my family during Korean Thanksgiving Day. > > Happy hacking! > > Thanks, > Youngwoo > > > On Wed, Sep 4, 2019 at 6:55 PM Jay Vyas > wrote: > > > Hi folks. Couple of updateys in the next gen brainstorming we’ve been > > doing... > > > > > > Sid Mani from minio is going to copresent with me at our bigtop talk at > > apachecon next week, around the internals of minio integrations etc. > > > > Spoke with cos and sounds like we definetly have a quorum for hacking on > > this next gen implementation. > > > > Whose in on hacking on it, and who will be in Vegas on the 12th? > > > > I’ll help w the k8s stuff so don’t be intimidated if that’s your weak > spot. >
Re: Minio to copresent , quorum for next gen bigtop ?
Jay, I believe It would be a great collaboration! I've been using Minio for last 6 months to do my day job, It's great and reliable. Unfortunately, I can't be there because my trip to Vegas was cancelled. :-( I'll spend time with my family during Korean Thanksgiving Day. Happy hacking! Thanks, Youngwoo On Wed, Sep 4, 2019 at 6:55 PM Jay Vyas wrote: > Hi folks. Couple of updateys in the next gen brainstorming we’ve been > doing... > > > Sid Mani from minio is going to copresent with me at our bigtop talk at > apachecon next week, around the internals of minio integrations etc. > > Spoke with cos and sounds like we definetly have a quorum for hacking on > this next gen implementation. > > Whose in on hacking on it, and who will be in Vegas on the 12th? > > I’ll help w the k8s stuff so don’t be intimidated if that’s your weak spot.
Minio to copresent , quorum for next gen bigtop ?
Hi folks. Couple of updateys in the next gen brainstorming we’ve been doing... Sid Mani from minio is going to copresent with me at our bigtop talk at apachecon next week, around the internals of minio integrations etc. Spoke with cos and sounds like we definetly have a quorum for hacking on this next gen implementation. Whose in on hacking on it, and who will be in Vegas on the 12th? I’ll help w the k8s stuff so don’t be intimidated if that’s your weak spot.
