Re: Thoughts about moving submarine to a separate git repo?
> > One major point from my end is that the commits to new repo ideally should > happen from committers or branch committers. Definitely agree! On Thu, Aug 22, 2019 at 12:32 PM Sunil Govindan wrote: > Thanks Wangda for sharing the thoughts. > > I agree with the idea of new repo for more cleaner code base and remove > additional dependencies, jenkins etc. > One major point from my end is that the commits to new repo ideally should > happen from committers or branch committers. > > As hadoop community, we could help in this case. @Wangda Tan > your thoughts? > > Thanks > Sunil > > On Wed, Aug 21, 2019 at 9:47 AM Wangda Tan wrote: > >> Hi Xun, >> >> Thanks for starting this thread. I'm glad to see the existing momentum >> made >> by Submarine community, and I like the proposal to make it be a separate >> Git repo. >> >> Some suggestions: >> >> 1) For options mentioned by Sunil: I think it's better to be a separate >> Git >> repo instead of a branch. To me, the branch is targeted for a >> diverged codebase instead of a new codebase. Since Submarine needs a clean >> root source code directory. I think moving to a new Git repo makes more >> sense. >> >> 2) For the Submarine external code, when we pulling them in, I think we >> need to make sure license, iCLA, code comply with Apache standard. Which >> means we need to do some additional reviews, etc. for patches being pulled >> in. (instead of as-is). >> >> 3) Can we address comments from @Wei-Chiu Chuang , >> to >> give some extra time for existing Hadoop committers/contributors who have >> interests to review the code? Waiting for at least 1 day for a big patch >> and 6 hours for a minor fix might be a good rule to follow. And @Wei-Chiu >> Chuang please let Submarine community know if you >> have >> anything interested to review so developers can ping you when they have >> any >> patches. >> >> Best, >> Wangda >> >> On Tue, Aug 20, 2019 at 9:54 PM Wei-Chiu Chuang >> wrote: >> >> > > >> > > >> > > >> > > Submarine dev community has a total of 8 developers and submits an >> > average >> > > of 4 to 5 PR per day. >> > > But there are a limited number of Hadoop committer actively help >> review >> > and >> > > merge patches, which causes development progress delays. >> > > >> > > I just want to point this out that this is concerning -- I wanted to >> help >> > review patches, but it wasn't obvious the patches were raised as PRs in >> a >> > non-apache git repo. >> > Please be more inclusive. >> > >> >
Re: Thoughts about moving submarine to a separate git repo?
Thanks Wangda for sharing the thoughts. I agree with the idea of new repo for more cleaner code base and remove additional dependencies, jenkins etc. One major point from my end is that the commits to new repo ideally should happen from committers or branch committers. As hadoop community, we could help in this case. @Wangda Tan your thoughts? Thanks Sunil On Wed, Aug 21, 2019 at 9:47 AM Wangda Tan wrote: > Hi Xun, > > Thanks for starting this thread. I'm glad to see the existing momentum made > by Submarine community, and I like the proposal to make it be a separate > Git repo. > > Some suggestions: > > 1) For options mentioned by Sunil: I think it's better to be a separate Git > repo instead of a branch. To me, the branch is targeted for a > diverged codebase instead of a new codebase. Since Submarine needs a clean > root source code directory. I think moving to a new Git repo makes more > sense. > > 2) For the Submarine external code, when we pulling them in, I think we > need to make sure license, iCLA, code comply with Apache standard. Which > means we need to do some additional reviews, etc. for patches being pulled > in. (instead of as-is). > > 3) Can we address comments from @Wei-Chiu Chuang , to > give some extra time for existing Hadoop committers/contributors who have > interests to review the code? Waiting for at least 1 day for a big patch > and 6 hours for a minor fix might be a good rule to follow. And @Wei-Chiu > Chuang please let Submarine community know if you > have > anything interested to review so developers can ping you when they have any > patches. > > Best, > Wangda > > On Tue, Aug 20, 2019 at 9:54 PM Wei-Chiu Chuang > wrote: > > > > > > > > > > > > > Submarine dev community has a total of 8 developers and submits an > > average > > > of 4 to 5 PR per day. > > > But there are a limited number of Hadoop committer actively help review > > and > > > merge patches, which causes development progress delays. > > > > > > I just want to point this out that this is concerning -- I wanted to > help > > review patches, but it wasn't obvious the patches were raised as PRs in a > > non-apache git repo. > > Please be more inclusive. > > >
Re: Thoughts about moving submarine to a separate git repo?
Hi Xun, Thanks for starting this thread. I'm glad to see the existing momentum made by Submarine community, and I like the proposal to make it be a separate Git repo. Some suggestions: 1) For options mentioned by Sunil: I think it's better to be a separate Git repo instead of a branch. To me, the branch is targeted for a diverged codebase instead of a new codebase. Since Submarine needs a clean root source code directory. I think moving to a new Git repo makes more sense. 2) For the Submarine external code, when we pulling them in, I think we need to make sure license, iCLA, code comply with Apache standard. Which means we need to do some additional reviews, etc. for patches being pulled in. (instead of as-is). 3) Can we address comments from @Wei-Chiu Chuang , to give some extra time for existing Hadoop committers/contributors who have interests to review the code? Waiting for at least 1 day for a big patch and 6 hours for a minor fix might be a good rule to follow. And @Wei-Chiu Chuang please let Submarine community know if you have anything interested to review so developers can ping you when they have any patches. Best, Wangda On Tue, Aug 20, 2019 at 9:54 PM Wei-Chiu Chuang wrote: > > > > > > > > Submarine dev community has a total of 8 developers and submits an > average > > of 4 to 5 PR per day. > > But there are a limited number of Hadoop committer actively help review > and > > merge patches, which causes development progress delays. > > > > I just want to point this out that this is concerning -- I wanted to help > review patches, but it wasn't obvious the patches were raised as PRs in a > non-apache git repo. > Please be more inclusive. >
Re: Thoughts about moving submarine to a separate git repo?
> > > > Submarine dev community has a total of 8 developers and submits an average > of 4 to 5 PR per day. > But there are a limited number of Hadoop committer actively help review and > merge patches, which causes development progress delays. > > I just want to point this out that this is concerning -- I wanted to help review patches, but it wasn't obvious the patches were raised as PRs in a non-apache git repo. Please be more inclusive.
Re: Thoughts about moving submarine to a separate git repo?
+1 (non-binding) We need a separate repository to keep it quickly development. On Tue, Aug 20, 2019 at 3:00 PM zac yuan wrote: > +1 (non-binding) > Looking forward to having a separate repo and making the development more > conveniently and rapidly. > > Thanks > Zac Zhou > > kevin su 于2019年8月20日周二 下午2:45写道： > > > +1, Agree > > using separate repo can better develop and maintain submarine > > > > Xun Liu 於 2019年8月20日 週二 下午2:05寫道： > > > > > Hi Sunil > > > > > > Thank you for your advice and attention. > > > > > > The Hadoop submarine development team needs a separate repo for several > > > reasons: > > > 1. We have developed a separate submarine website to introduce the > > > functions and features of submarine. The common practice is to put it > in > > > the gh-pages branch of the repo repository, automatically generate HTML > > > code, and update the website. > > > > > > 2. We also want to associate github's repo with apache jira. It can > > > automatically synchronize the status of the issue on github, and the > > > comments will be automatically synchronized to Apache's jira. There are > > > some other open source projects managed like this. Very good > experience. > > > Making these settings requires some administrative privileges with > repo. > > > > > > 3. We also need to create different branches for different versions for > > > version management. > > > > > > So, we need a separate repo repository. > > > > > > thank you for your support. :-) > > > > > > > > > Best Regards > > > Xun Liu > > > > > > > > > > On Aug 16, 2019, at 5:25 PM, dashuiguailu...@gmail.com wrote: > > > > > > > > +1 ，Agree that independent development of submarine can better adapt > to > > > the development of machine learning > > > > > > > > > > > > > > > > dashuiguailu...@gmail.com > > > > > > > > From: Xun Liu > > > > Date: 2019-08-16 12:43 > > > > To: common-dev; yarn-dev; hdfs-dev; submarine-dev > > > > Subject: Thoughts about moving submarine to a separate git repo? > > > > Dear Submarine developers, > > > > > > > > My name is Xun Liu, I am a member of the Hadoop submarine development > > > team. > > > > I'm one of the major contributor of Submarine since June 2018. > > > > > > > > I want to hear your thoughts about creating a separate GitHub repo > > under > > > > Apache to do submarine development. This is an independent effort of > > > > Submarine spin-off from the Hadoop project [ > > > > > > > > > > https://lists.apache.org/thread.html/3fab657f905d081b536d9081dc404f7fd20c80eb824c857bc8e16e3b@ > > > ]. > > > > However, once the spin-off is approved, this effort can benefit the > > > > follow-up processes as well. > > > > > > > > Submarine dev community has a total of 8 developers and submits an > > > average > > > > of 4 to 5 PR per day. > > > > But there are a limited number of Hadoop committer actively help > review > > > and > > > > merge patches, which causes development progress delays. > > > > > > > > So we created an external GitHub repo [ > > > > https://github.com/hadoopsubmarine/submarine] and moved all the code > > for > > > > the Hadoop submarine project into the external Github repo. > > > > In this way, everyone can review the code for each other, and now the > > > > development progress of Hadoop submarine is very fast. > > > > > > > > Also, now Submarine has little dependency on Hadoop, we want to have > a > > > > separate CI/CD pipeline to release and test submarine instead of > every > > > time > > > > build whole Hadoop. Putting Submarine under Hadoop will introduce > > > > unnecessary dependencies to Hadoop's top-level pom.xml. > > > > > > > > Our development process still complies with the development rules of > > the > > > > Hadoop community: first, create a ticket in the submarine JIRA, and > > then > > > > develop, in the external GitHub repo repository, the title of each PR > > > will > > > > be accompanied by the JIRA ID number. > > > > > > > > Once the Apache Github repo is created, we going to move all external > > > > commits to the new Apache Github repo. > > > > > > > > Any suggestions are welcome! > > > > > > > > Best Regards > > > > Xun Liu > > > > > > > > > > > > - > > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > > > > > > > > >
Re: Thoughts about moving submarine to a separate git repo?
+1, Agree using separate repo can better develop and maintain submarine Xun Liu 於 2019年8月20日 週二 下午2:05寫道： > Hi Sunil > > Thank you for your advice and attention. > > The Hadoop submarine development team needs a separate repo for several > reasons: > 1. We have developed a separate submarine website to introduce the > functions and features of submarine. The common practice is to put it in > the gh-pages branch of the repo repository, automatically generate HTML > code, and update the website. > > 2. We also want to associate github's repo with apache jira. It can > automatically synchronize the status of the issue on github, and the > comments will be automatically synchronized to Apache's jira. There are > some other open source projects managed like this. Very good experience. > Making these settings requires some administrative privileges with repo. > > 3. We also need to create different branches for different versions for > version management. > > So, we need a separate repo repository. > > thank you for your support. :-) > > > Best Regards > Xun Liu > > > > On Aug 16, 2019, at 5:25 PM, dashuiguailu...@gmail.com wrote: > > > > +1 ，Agree that independent development of submarine can better adapt to > the development of machine learning > > > > > > > > dashuiguailu...@gmail.com > > > > From: Xun Liu > > Date: 2019-08-16 12:43 > > To: common-dev; yarn-dev; hdfs-dev; submarine-dev > > Subject: Thoughts about moving submarine to a separate git repo? > > Dear Submarine developers, > > > > My name is Xun Liu, I am a member of the Hadoop submarine development > team. > > I'm one of the major contributor of Submarine since June 2018. > > > > I want to hear your thoughts about creating a separate GitHub repo under > > Apache to do submarine development. This is an independent effort of > > Submarine spin-off from the Hadoop project [ > > > https://lists.apache.org/thread.html/3fab657f905d081b536d9081dc404f7fd20c80eb824c857bc8e16e3b@ > ]. > > However, once the spin-off is approved, this effort can benefit the > > follow-up processes as well. > > > > Submarine dev community has a total of 8 developers and submits an > average > > of 4 to 5 PR per day. > > But there are a limited number of Hadoop committer actively help review > and > > merge patches, which causes development progress delays. > > > > So we created an external GitHub repo [ > > https://github.com/hadoopsubmarine/submarine] and moved all the code for > > the Hadoop submarine project into the external Github repo. > > In this way, everyone can review the code for each other, and now the > > development progress of Hadoop submarine is very fast. > > > > Also, now Submarine has little dependency on Hadoop, we want to have a > > separate CI/CD pipeline to release and test submarine instead of every > time > > build whole Hadoop. Putting Submarine under Hadoop will introduce > > unnecessary dependencies to Hadoop's top-level pom.xml. > > > > Our development process still complies with the development rules of the > > Hadoop community: first, create a ticket in the submarine JIRA, and then > > develop, in the external GitHub repo repository, the title of each PR > will > > be accompanied by the JIRA ID number. > > > > Once the Apache Github repo is created, we going to move all external > > commits to the new Apache Github repo. > > > > Any suggestions are welcome! > > > > Best Regards > > Xun Liu > > > > - > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > >
Re: Thoughts about moving submarine to a separate git repo?
+1 ，Agree that independent development of submarine can better adapt to the development of machine learning dashuiguailu...@gmail.com From: Xun Liu Date: 2019-08-16 12:43 To: common-dev; yarn-dev; hdfs-dev; submarine-dev Subject: Thoughts about moving submarine to a separate git repo? Dear Submarine developers, My name is Xun Liu, I am a member of the Hadoop submarine development team. I'm one of the major contributor of Submarine since June 2018. I want to hear your thoughts about creating a separate GitHub repo under Apache to do submarine development. This is an independent effort of Submarine spin-off from the Hadoop project [ https://lists.apache.org/thread.html/3fab657f905d081b536d9081dc404f7fd20c80eb824c857bc8e16e3b@]. However, once the spin-off is approved, this effort can benefit the follow-up processes as well. Submarine dev community has a total of 8 developers and submits an average of 4 to 5 PR per day. But there are a limited number of Hadoop committer actively help review and merge patches, which causes development progress delays. So we created an external GitHub repo [ https://github.com/hadoopsubmarine/submarine] and moved all the code for the Hadoop submarine project into the external Github repo. In this way, everyone can review the code for each other, and now the development progress of Hadoop submarine is very fast. Also, now Submarine has little dependency on Hadoop, we want to have a separate CI/CD pipeline to release and test submarine instead of every time build whole Hadoop. Putting Submarine under Hadoop will introduce unnecessary dependencies to Hadoop's top-level pom.xml. Our development process still complies with the development rules of the Hadoop community: first, create a ticket in the submarine JIRA, and then develop, in the external GitHub repo repository, the title of each PR will be accompanied by the JIRA ID number. Once the Apache Github repo is created, we going to move all external commits to the new Apache Github repo. Any suggestions are welcome! Best Regards Xun Liu
Re: Thoughts about moving submarine to a separate git repo?
Hi Xun,
Thanks for the proposal. Altogether this proposal makes sense to me.
IIUC, an external repo is created and developers found more productive in
that because of faster reviews and commits.
To do more this apache way,
1. I think its better we track all this in apache jira (apparently this is
already happening, thanks for that).
2. In apache, we have a provision called branch committer. This role will
help to drive a feature ('s) in a branch cut from apache trunk,
and selected branch committers can push changes to apache repo. Thus fast
pacing feature development in community.
So nominating couple of active contributors of submarine for branch
committership could address the concerns raised by you above.
For this part, only difference I see is about a new repo instead of branch
from Apache trunk.
AFAIK, This seems not a stopper for the branch committership suggestion
(please correct me if I am wrong).
However we need to ensure that commit rights of that repo is only available
branch committer's and all apache policies are adhered in that.
Overall I feel option #2 mentioned above is a good way to proceed further.
Welcoming thoughts from others who have more contexts into similar past
experiences.
Thanks,
Sunil
On Fri, Aug 16, 2019 at 10:13 AM Xun Liu wrote:
> Dear Submarine developers,
>
> My name is Xun Liu, I am a member of the Hadoop submarine development team.
> I'm one of the major contributor of Submarine since June 2018.
>
> I want to hear your thoughts about creating a separate GitHub repo under
> Apache to do submarine development. This is an independent effort of
> Submarine spin-off from the Hadoop project [
>
> https://lists.apache.org/thread.html/3fab657f905d081b536d9081dc404f7fd20c80eb824c857bc8e16e3b@
> ].
> However, once the spin-off is approved, this effort can benefit the
> follow-up processes as well.
>
> Submarine dev community has a total of 8 developers and submits an average
> of 4 to 5 PR per day.
> But there are a limited number of Hadoop committer actively help review and
> merge patches, which causes development progress delays.
>
> So we created an external GitHub repo [
> https://github.com/hadoopsubmarine/submarine] and moved all the code for
> the Hadoop submarine project into the external Github repo.
> In this way, everyone can review the code for each other, and now the
> development progress of Hadoop submarine is very fast.
>
> Also, now Submarine has little dependency on Hadoop, we want to have a
> separate CI/CD pipeline to release and test submarine instead of every time
> build whole Hadoop. Putting Submarine under Hadoop will introduce
> unnecessary dependencies to Hadoop's top-level pom.xml.
>
> Our development process still complies with the development rules of the
> Hadoop community: first, create a ticket in the submarine JIRA, and then
> develop, in the external GitHub repo repository, the title of each PR will
> be accompanied by the JIRA ID number.
>
> Once the Apache Github repo is created, we going to move all external
> commits to the new Apache Github repo.
>
> Any suggestions are welcome!
>
> Best Regards
> Xun Liu
>
Thoughts about moving submarine to a separate git repo?
Dear Submarine developers, My name is Xun Liu, I am a member of the Hadoop submarine development team. I'm one of the major contributor of Submarine since June 2018. I want to hear your thoughts about creating a separate GitHub repo under Apache to do submarine development. This is an independent effort of Submarine spin-off from the Hadoop project [ https://lists.apache.org/thread.html/3fab657f905d081b536d9081dc404f7fd20c80eb824c857bc8e16e3b@]. However, once the spin-off is approved, this effort can benefit the follow-up processes as well. Submarine dev community has a total of 8 developers and submits an average of 4 to 5 PR per day. But there are a limited number of Hadoop committer actively help review and merge patches, which causes development progress delays. So we created an external GitHub repo [ https://github.com/hadoopsubmarine/submarine] and moved all the code for the Hadoop submarine project into the external Github repo. In this way, everyone can review the code for each other, and now the development progress of Hadoop submarine is very fast. Also, now Submarine has little dependency on Hadoop, we want to have a separate CI/CD pipeline to release and test submarine instead of every time build whole Hadoop. Putting Submarine under Hadoop will introduce unnecessary dependencies to Hadoop's top-level pom.xml. Our development process still complies with the development rules of the Hadoop community: first, create a ticket in the submarine JIRA, and then develop, in the external GitHub repo repository, the title of each PR will be accompanied by the JIRA ID number. Once the Apache Github repo is created, we going to move all external commits to the new Apache Github repo. Any suggestions are welcome! Best Regards Xun Liu
