Radar, it seems that the maven issue could be resolved similarly? 2016-07-01 15:31 GMT+08:00 Guo Gang <[email protected]>:
> Those submodules will be git-cloned and then compiled only with specific > options > are enabled in configure command. If users want to build them without > Internet > connection they will have to download related git repo with specific > commit number in advance and copy them to hawq repo. Or they can set up > internal git repos and modify the git addresses in hawq source. > > > > 2016-07-01 14:07 GMT+08:00 Radar Da lei <[email protected]>: > >> One thought is if they all have released source tarball, we can let user >> download these tarballs to a specified directory. >> >> Then when compiling, we do not do 'git clone' but extract these tarballs >> if >> the specified version tarballs are detected. >> >> But for user who don't have internet access, when compiling, maven >> download >> would be a problem. See: >> >> Downloading: >> >> https://repo.maven.apache.org/maven2/com/pivotal/hawq/hawq-mapreduce-ao/1.1.0/hawq-mapreduce-ao-1.1.0.jar >> Downloading: >> >> https://repo.maven.apache.org/maven2/com/pivotal/hawq/hawq-mapreduce-parquet/1.1.0/hawq-mapreduce-parquet-1.1.0.jar >> [INFO] hawq-hadoop ........................................ FAILURE [ >> 1.757 s] >> >> Thanks. >> >> Regards, >> Radar >> >> On Fri, Jul 1, 2016 at 12:30 AM, Roman Shaposhnik <[email protected]> >> wrote: >> >> > Good discussion so far, but I wanted to make sure that our source >> > tarball has an option of being built in an environment completely >> > disconnected from the internet (you can mandate certain shared >> > librarires to be available in the environment, of course). Will this >> > be possible with the current proposal? >> > >> > Thanks, >> > Roman. >> > >> > On Thu, Jun 30, 2016 at 3:51 AM, Guo Gang <[email protected]> wrote: >> > > Discussed more with Jiali, I realised that to make hawk repo be more >> > > friendly to gporca developers, I could store git address, branch and a >> > > commit number in one file so that gporca developers could easier >> switch >> > > gporca code. >> > > >> > > 2016-06-30 15:35 GMT+08:00 Guo Gang <[email protected]>: >> > > >> > >> 1. HAWQ repo is not a good place for gp-xerces development. My change >> > just >> > >> use a stable and working version of gp-xerces. If gp-xerces >> > developers >> > >> want to >> > >> test it with hawq, they can easily modify code under hawk: >> gp-xerces >> > >> (e.g. pull from >> > >> a branch, etc.), then remove the timestamp file >> > >> depends/thirdpaty/gp-xerces_build_timestamp >> > >> to trigger a rebuild. >> > >> >> > >> 2. Correct. >> > >> >> > >> 2016-06-30 11:31 GMT+08:00 Jiali Yao <[email protected]>: >> > >> >> > >>> Hi Paul >> > >>> >> > >>> Generally agree on the approach while I have some questions: >> > >>> >> > >>> 1. Do we need to add branch handling? For example, I am a developer >> > >>> of gp-xerces, >> > >>> I need to test my branch code. >> > >>> >> > >>> 2. Let me double confirm about the build process. Below is >> gp-xerces >> > repo >> > >>> and now HAWQ will be commit a. So I will set commit file >> > gp-xerces.commit >> > >>> in HAWQ repo which indicate we will use commit a. And then sometime >> > later, >> > >>> we want to use b, then we commit gp-xerces.commit to b right? >> And we >> > >>> will have multiple .commit file for each submodule now. right? >> > >>> gp-xerces: >> > >>> commit c >> > >>> .... >> > >>> commit b >> > >>> .... >> > >>> commit a >> > >>> >> > >>> Thanks >> > >>> Jiali >> > >>> >> > >>> On Thu, Jun 30, 2016 at 11:03 AM, Radar Da lei <[email protected]> >> > wrote: >> > >>> >> > >>> > +1 >> > >>> > Since 'git clone' is called automatically in Makefile, I feel >> this is >> > >>> much >> > >>> > better than use submodule. >> > >>> > >> > >>> > Regards, >> > >>> > Radar >> > >>> > >> > >>> > On Thu, Jun 30, 2016 at 10:07 AM, Lili Ma <[email protected]> wrote: >> > >>> > >> > >>> > > Hi Paul, >> > >>> > > >> > >>> > > I have one question, you mentioned "if I update gp-xerces commit >> > >>> number >> > >>> > in >> > >>> > > the commit file gp-xerces.commit make will trigger an >> auto-build". >> > >>> Could >> > >>> > > we set the $(gp_xerces_commit) to any commit number? What I mean >> > is, >> > >>> > could >> > >>> > > we set it to a commit number already existing in my local code, >> or >> > >>> from >> > >>> > > remote code which I have not fetched? >> > >>> > > >> > >>> > > If the answer is yes, I totally agree this suggestion. >> > >>> > > >> > >>> > > Thanks >> > >>> > > Lili >> > >>> > > >> > >>> > > On Wed, Jun 29, 2016 at 11:52 PM, Kavinder Dhaliwal < >> > >>> > [email protected]> >> > >>> > > wrote: >> > >>> > > >> > >>> > > > +1. I am in favor of this approach, especially if the >> submodules >> > >>> make >> > >>> > the >> > >>> > > > source tarball difficult to build. I also agree it will make a >> > >>> > developers >> > >>> > > > life much easier. >> > >>> > > > >> > >>> > > > On Wed, Jun 29, 2016 at 3:18 AM, Guo Gang <[email protected]> >> > >>> wrote: >> > >>> > > > >> > >>> > > > > I'm proposing this change because: >> > >>> > > > > >> > >>> > > > > 1) We are ready to release the first apache release with >> > "source >> > >>> > > > tarball", >> > >>> > > > > but >> > >>> > > > > submodule is not friendly to source tarball since git >> > >>> submodule >> > >>> > > > > require a git parent. >> > >>> > > > > >> > >>> > > > > 2) With more and more development, I found that submodule >> > >>> mechanism >> > >>> > > > > is not that friendly for development, e.g. >> > >>> > > > > >> > >>> > > > > If the commit number of one submodule is modified, it is >> > hard >> > >>> to >> > >>> > > > detect >> > >>> > > > > this for Makefile. We need to manually update it in an >> old >> > >>> repo. >> > >>> > If >> > >>> > > > > using "git clone", we could easily detect update by set >> a >> > >>> commit >> > >>> > > > number >> > >>> > > > > file >> > >>> > > > > as dependency and thus easily update the code. >> > >>> > > > > >> > >>> > > > > Some developers have complained the annoying output in >> "git >> > >>> > status" >> > >>> > > > > after building submodules. With "git clone" we can easily >> > mask >> > >>> > those >> > >>> > > > > directories via .gitignore. >> > >>> > > > > >> > >>> > > > > It is hard for developers who is not familiar with >> submodule >> > >>> > > (Frankly >> > >>> > > > > speaking >> > >>> > > > > I really do not think submodule is friendly) to >> manipulate >> > >>> related >> > >>> > > > > directories >> > >>> > > > > when necessary. >> > >>> > > > > >> > >>> > > > > While with the "git clone" solution, we save the commit of >> > >>> previous >> > >>> > > > > submodule in a file, >> > >>> > > > > and the related Makefile code change is rather simple, e.g. >> for >> > >>> > > > gp-xerces. >> > >>> > > > > >> > >>> > > > > $(ORCA_BLD_PATH)/gp-xerces_prepare_timestamp: >> > >>> > > > > $(ORCA_SRC_PATH)/gp-xerces.commit >> > >>> > > > > rm -f $(ORCA_BLD_PATH)/gp-xerces_prepare_timestamp >> > >>> > > > > >> > >>> > > > > gp-xerces_prepare: >> $(ORCA_BLD_PATH)/gp-xerces_prepare_timestamp >> > >>> > > > > if [ ! -f $(ORCA_BLD_PATH)/gp-xerces_prepare_timestamp >> ]; >> > >>> then \ >> > >>> > > > > [ "x$(gp_xerces_commit)" != "x" ] || exit 1; \ >> > >>> > > > > cd $(abs_top_srcdir)/$(subdir); mkdir -p gp-xerces; >> cd >> > >>> > > > gp-xerces; \ >> > >>> > > > > [ ! -d .git ] && git clone >> > >>> > > > > https://github.com/greenplum-db/gp-xerces.git .; \ >> > >>> > > > > git reset --hard $(gp_xerces_commit) || exit 2; \ >> > >>> > > > > touch $(ORCA_BLD_PATH)/gp_xerces_prepare_timestamp; >> \ >> > >>> > > > > fi >> > >>> > > > > >> > >>> > > > > With above code change, if I update gp-xerces commit number >> in >> > the >> > >>> > > commit >> > >>> > > > > file >> > >>> > > > > gp-xerces.commit make will trigger an auto-build, If I >> messed >> > up >> > >>> the >> > >>> > > > > gp-xerces directory >> > >>> > > > > I can easily remove the whole gp-xerces, or just remove the >> > >>> timestamp >> > >>> > > > file >> > >>> > > > > gp-xerces_prepare_timestamp to trigger an auto build. >> > >>> > > > > >> > >>> > > > > Any suggestion? Thanks. >> > >>> > > > > >> > >>> > > > >> > >>> > > >> > >>> > >> > >>> >> > >> >> > >> >> > >> > >
