Re: [DISCUSSION] Questions about docker images support
Do we want to try alpine base image to minimize the image size? Or we want to stay with one of our existings? On Tue, Dec 10, 2019 at 6:47 PM Simon Weng wrote: > System process, in the form of container, can still work side by side with > the spout and bolt container, because they are in the same pod, sharing the > same Linux kernel and networking space, they’re seeing each other on > localhost. They can even share the same file system as long as they share > the same volume. > > On Tue, Dec 10, 2019 at 6:37 PM Ning Wang wrote: > >> One image sounds good to me. >> >> The two ideas are interesting. Heron system processes are designed to work >> in the same container of worker instances so is might be tricky to >> separate. That being said, there are definitely things to improve. >> >> On Tue, Dec 10, 2019 at 1:07 PM SiMing Weng >> wrote: >> >> > Hi, guys: >> > >> > I think it indeed makes sense to narrow down to a single base image. >> > Regarding which variant to choose, it boils down to any >> platform-dependent >> > component in Heron, I can only think of native StreamManager as far as I >> > know. I’m sure others on the team knows better than I. >> > >> > Also, it may sounds a long shot, but I’d like to throw the idea here. >> > Currently, we containerize Heron as an all-in-one image, which is huge >> and >> > serves as something like a VM image. Ideally, we should containerize >> each >> > component and make them slim, then it becomes easier to contribute >> > to/maintain/release. >> > >> > Then even more ambitiously, on k8s cluster, what if we make it possible >> to >> > deploy individual Spout/Bolt also as container inside each pod, with a >> > bunch of side-car container, such as StreamManger, etc, rather than >> > managing the individual process manually in our Executor. This k8s >> native >> > approach would open a whole lot possibility, such as easier log >> aggregation >> > by existing open source tools, auto scaling/self-tuning with k8s >> operator >> > pattern. I think the process-based stream processing nature of Heron >> gives >> > itself a unique advantage to be used in containerized environment, >> compared >> > to thread-based peers. >> > >> > Cheers, >> > >> > SiMing Weng >> > >> > > On Dec 10, 2019, at 2:11 PM, Josh Fischer >> wrote: >> > > >> > > I think we should settle on one "official" version for the community >> to >> > > support. I would like to hear from other users on the list as to >> which >> > > image it is we keep supporting moving forward. I don't think we have >> > > enough bandwidth or resources to support 5 different versions of >> Heron >> > in >> > > a container. >> > > >> > > On Tue, Dec 10, 2019 at 1:53 AM Ning Wang >> wrote: >> > > >> > >> Hi, everyone, >> > >> >> > >> When I was working on this PR ( >> > >> https://github.com/apache/incubator-heron/pull/3411), Josh raised >> up a >> > >> question about the docker images supported by Heron. There are 5 >> images >> > >> configured: centos7, debian9, ubuntu 14.04, 16.04 and 18.04. >> > >> >> > >> Questions to discuss: >> > >> - It could be a burden to maintain all of them. Do you feel it is >> > necessary >> > >> to support all of them? Or is there another one we should have? >> > >> - For ubuntu, do we really need 3 versions? >> > >> >> > >> Regards, >> > >> --ning >> > >> >> > >> > >> > -- > Sent from Gmail Mobile > -- Sent from Gmail Mobile
Re: [DISCUSSION] Questions about docker images support
System process, in the form of container, can still work side by side with the spout and bolt container, because they are in the same pod, sharing the same Linux kernel and networking space, they’re seeing each other on localhost. They can even share the same file system as long as they share the same volume. On Tue, Dec 10, 2019 at 6:37 PM Ning Wang wrote: > One image sounds good to me. > > The two ideas are interesting. Heron system processes are designed to work > in the same container of worker instances so is might be tricky to > separate. That being said, there are definitely things to improve. > > On Tue, Dec 10, 2019 at 1:07 PM SiMing Weng wrote: > > > Hi, guys: > > > > I think it indeed makes sense to narrow down to a single base image. > > Regarding which variant to choose, it boils down to any > platform-dependent > > component in Heron, I can only think of native StreamManager as far as I > > know. I’m sure others on the team knows better than I. > > > > Also, it may sounds a long shot, but I’d like to throw the idea here. > > Currently, we containerize Heron as an all-in-one image, which is huge > and > > serves as something like a VM image. Ideally, we should containerize each > > component and make them slim, then it becomes easier to contribute > > to/maintain/release. > > > > Then even more ambitiously, on k8s cluster, what if we make it possible > to > > deploy individual Spout/Bolt also as container inside each pod, with a > > bunch of side-car container, such as StreamManger, etc, rather than > > managing the individual process manually in our Executor. This k8s native > > approach would open a whole lot possibility, such as easier log > aggregation > > by existing open source tools, auto scaling/self-tuning with k8s operator > > pattern. I think the process-based stream processing nature of Heron > gives > > itself a unique advantage to be used in containerized environment, > compared > > to thread-based peers. > > > > Cheers, > > > > SiMing Weng > > > > > On Dec 10, 2019, at 2:11 PM, Josh Fischer wrote: > > > > > > I think we should settle on one "official" version for the community to > > > support. I would like to hear from other users on the list as to > which > > > image it is we keep supporting moving forward. I don't think we have > > > enough bandwidth or resources to support 5 different versions of Heron > > in > > > a container. > > > > > > On Tue, Dec 10, 2019 at 1:53 AM Ning Wang > wrote: > > > > > >> Hi, everyone, > > >> > > >> When I was working on this PR ( > > >> https://github.com/apache/incubator-heron/pull/3411), Josh raised up > a > > >> question about the docker images supported by Heron. There are 5 > images > > >> configured: centos7, debian9, ubuntu 14.04, 16.04 and 18.04. > > >> > > >> Questions to discuss: > > >> - It could be a burden to maintain all of them. Do you feel it is > > necessary > > >> to support all of them? Or is there another one we should have? > > >> - For ubuntu, do we really need 3 versions? > > >> > > >> Regards, > > >> --ning > > >> > > > > > -- Sent from Gmail Mobile
Re: [DISCUSSION] Questions about docker images support
One image sounds good to me. The two ideas are interesting. Heron system processes are designed to work in the same container of worker instances so is might be tricky to separate. That being said, there are definitely things to improve. On Tue, Dec 10, 2019 at 1:07 PM SiMing Weng wrote: > Hi, guys: > > I think it indeed makes sense to narrow down to a single base image. > Regarding which variant to choose, it boils down to any platform-dependent > component in Heron, I can only think of native StreamManager as far as I > know. I’m sure others on the team knows better than I. > > Also, it may sounds a long shot, but I’d like to throw the idea here. > Currently, we containerize Heron as an all-in-one image, which is huge and > serves as something like a VM image. Ideally, we should containerize each > component and make them slim, then it becomes easier to contribute > to/maintain/release. > > Then even more ambitiously, on k8s cluster, what if we make it possible to > deploy individual Spout/Bolt also as container inside each pod, with a > bunch of side-car container, such as StreamManger, etc, rather than > managing the individual process manually in our Executor. This k8s native > approach would open a whole lot possibility, such as easier log aggregation > by existing open source tools, auto scaling/self-tuning with k8s operator > pattern. I think the process-based stream processing nature of Heron gives > itself a unique advantage to be used in containerized environment, compared > to thread-based peers. > > Cheers, > > SiMing Weng > > > On Dec 10, 2019, at 2:11 PM, Josh Fischer wrote: > > > > I think we should settle on one "official" version for the community to > > support. I would like to hear from other users on the list as to which > > image it is we keep supporting moving forward. I don't think we have > > enough bandwidth or resources to support 5 different versions of Heron > in > > a container. > > > > On Tue, Dec 10, 2019 at 1:53 AM Ning Wang wrote: > > > >> Hi, everyone, > >> > >> When I was working on this PR ( > >> https://github.com/apache/incubator-heron/pull/3411), Josh raised up a > >> question about the docker images supported by Heron. There are 5 images > >> configured: centos7, debian9, ubuntu 14.04, 16.04 and 18.04. > >> > >> Questions to discuss: > >> - It could be a burden to maintain all of them. Do you feel it is > necessary > >> to support all of them? Or is there another one we should have? > >> - For ubuntu, do we really need 3 versions? > >> > >> Regards, > >> --ning > >> > >
Re: [DISCUSSION] Questions about docker images support
Hi, guys: I think it indeed makes sense to narrow down to a single base image. Regarding which variant to choose, it boils down to any platform-dependent component in Heron, I can only think of native StreamManager as far as I know. I’m sure others on the team knows better than I. Also, it may sounds a long shot, but I’d like to throw the idea here. Currently, we containerize Heron as an all-in-one image, which is huge and serves as something like a VM image. Ideally, we should containerize each component and make them slim, then it becomes easier to contribute to/maintain/release. Then even more ambitiously, on k8s cluster, what if we make it possible to deploy individual Spout/Bolt also as container inside each pod, with a bunch of side-car container, such as StreamManger, etc, rather than managing the individual process manually in our Executor. This k8s native approach would open a whole lot possibility, such as easier log aggregation by existing open source tools, auto scaling/self-tuning with k8s operator pattern. I think the process-based stream processing nature of Heron gives itself a unique advantage to be used in containerized environment, compared to thread-based peers. Cheers, SiMing Weng > On Dec 10, 2019, at 2:11 PM, Josh Fischer wrote: > > I think we should settle on one "official" version for the community to > support. I would like to hear from other users on the list as to which > image it is we keep supporting moving forward. I don't think we have > enough bandwidth or resources to support 5 different versions of Heron in > a container. > > On Tue, Dec 10, 2019 at 1:53 AM Ning Wang wrote: > >> Hi, everyone, >> >> When I was working on this PR ( >> https://github.com/apache/incubator-heron/pull/3411), Josh raised up a >> question about the docker images supported by Heron. There are 5 images >> configured: centos7, debian9, ubuntu 14.04, 16.04 and 18.04. >> >> Questions to discuss: >> - It could be a burden to maintain all of them. Do you feel it is necessary >> to support all of them? Or is there another one we should have? >> - For ubuntu, do we really need 3 versions? >> >> Regards, >> --ning >>
Re: [DISCUSSION] Questions about docker images support
I think we should settle on one "official" version for the community to support. I would like to hear from other users on the list as to which image it is we keep supporting moving forward. I don't think we have enough bandwidth or resources to support 5 different versions of Heron in a container. On Tue, Dec 10, 2019 at 1:53 AM Ning Wang wrote: > Hi, everyone, > > When I was working on this PR ( > https://github.com/apache/incubator-heron/pull/3411), Josh raised up a > question about the docker images supported by Heron. There are 5 images > configured: centos7, debian9, ubuntu 14.04, 16.04 and 18.04. > > Questions to discuss: > - It could be a burden to maintain all of them. Do you feel it is necessary > to support all of them? Or is there another one we should have? > - For ubuntu, do we really need 3 versions? > > Regards, > --ning >
