Uploaded these two patches for the fix. Different branches depend on different Impala-lzo versions: https://gerrit.cloudera.org/c/12495/ https://gerrit.cloudera.org/c/12497/
Could anyone help to have a look? Thanks, Quanlong On Thu, Feb 14, 2019 at 12:13 PM Quanlong Huang <[email protected]> wrote: > Thank you for your help, Philip! > > >I added "IMPALA_LZO_BRANCH=2.x" to the properties file generated in > cherrypick-2.x-and-test.... > > This fixes my second failure. For my first failure, it still exists for my > recent run: > https://jenkins.impala.io/job/cherrypick-2.x-and-test/641 -> > https://jenkins.impala.io/job/parallel-all-tests/4992 -> > https://jenkins.impala.io/job/ubuntu-16.04-build-only/5395 > > The ubuntu-16.04-build-only job doesn't check out to the correct branch of > Impala-lzo. I think we need to add these in the Jenkins command before > buildall.sh: > (pushd $IMPALA_LZO && git checkout $IMPALA_LZO_BRANCH) > We also need to pass IMPALA_LZO_BRANCH to this job. Do you think so? Could > you help to have a try? > > Thanks, > Quanlong > > > > > On Thu, Feb 14, 2019 at 6:51 AM Philip Zeyliger <[email protected]> > wrote: > >> Sorry about that. Some Cloudera-internal automation force-pushed that >> branch. It's been fixed now, I hope. >> >> On Wed, Feb 13, 2019 at 2:00 PM Andrew Sherman <[email protected]> >> wrote: >> >> > I am debugging some weird build failures which seem to be confused about >> > branches in https://github.com/cloudera/impala-lzo.git >> > 'master' has the latest changes, but 'origin/master', the default branch >> > when you clone, points to some really old commit. >> > >> > Does anyone understand this? >> > >> > -Andrew >> > >> > asherman@asherman-desktop:~/git/asf/Impala-lzo$ git log -1 >> --pretty=fuller >> > master >> > commit dccb1be88a5e237b06ae69cd99b048a38d9f024b (HEAD -> master) >> > Author: Philip Zeyliger <[email protected]> >> > AuthorDate: Tue Jan 22 14:21:42 2019 -0800 >> > Commit: Philip Zeyliger <[email protected]> >> > CommitDate: Fri Feb 1 11:04:06 2019 -0800 >> > >> > Adapt to new interface for AddDiskIoRange (IMPALA-7980) >> > >> > The change for IMPALA-7980 moves out the "are we done >> > adding disk io ranges" accounting from AddDiskIoRange() >> > to a new UpdateRemainingScanRangeSubmissions call. >> > >> > asherman@asherman-desktop:~/git/asf/Impala-lzo$ git log -1 >> --pretty=fuller >> > origin/master >> > commit 9543908ec82245878c4060dec64a1533d7adbee6 >> > (origin/master-backup-2019-01-23, origin/master) >> > Author: Alex Behm <[email protected]> >> > AuthorDate: Thu Dec 4 16:53:06 2014 -0800 >> > Commit: Alex Behm <[email protected]> >> > CommitDate: Fri Dec 5 14:28:25 2014 -0800 >> > >> > Remove data compaction flag. >> > >> > Change-Id: Ife72c7ee630b6ecd9024a3d33b7673b56fcf1fba >> > >> > I am tracking this at https://issues.apache.org/jira/browse/IMPALA-8200 >> > >> > >> > On Wed, Feb 13, 2019 at 9:27 AM Philip Zeyliger <[email protected]> >> > wrote: >> > >> > > I added "IMPALA_LZO_BRANCH=2.x" to the properties file generated in >> > > cherrypick-2.x-and-test.... >> > > >> > > [image: image.png] >> > > >> > > On Tue, Feb 12, 2019 at 11:36 PM Quanlong Huang < >> [email protected] >> > > >> > > wrote: >> > > >> > >> Thank Tim and Fredy's help! >> > >> >> > >> However, I still find some downstream Jenkins jobs not using the same >> > >> IMPALA_LZO_BRANCH value. There're two failures due to this: >> > >> >> > >> 1) I ran gerrit-verify-dryrun for patch set 12429 >> > >> with IMPALA_LZO_BRANCH=2.x. The job is >> > >> https://jenkins.impala.io/job/gerrit-verify-dryrun/3760 >> > >> This downstream job ubuntu-16.04-build-only still used master branch >> of >> > >> Impala-lzo thus failed: >> > >> - ubuntu-16.04-build-only #5336 >> > >> https://jenkins.impala.io/job/ubuntu-16.04-build-only/5336/ >> > >> >> > >> Looks like this job does not use IMPALA_LZO_BRANCH so not switch >> branch >> > of >> > >> Impala-lzo. I think we should set IMPALA_LZO_BRANCH in >> impala-config.sh >> > >> and >> > >> use it in buildall.sh. >> > >> >> > >> 2) I ran cherrypick-2.x-and-test. The job is >> > >> https://jenkins.impala.io/job/cherrypick-2.x-and-test/638/ >> > >> The problematic downstream jobs: >> > >> - ubuntu-16.04-from-scratch #4470 >> > >> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/4470/ >> > >> - ubuntu-16.04-from-scratch #4471 >> > >> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/4471/ >> > >> >> > >> Looks like the IMPALA_LZO_BRANCH is not set so they still use the >> > default >> > >> value (master). >> > >> I don't have the permission to modify the job configs. Could anyone >> help >> > >> to >> > >> have a look? >> > >> >> > >> Thanks, >> > >> Quanlong >> > >> >> > >> On Mon, Feb 11, 2019 at 1:32 PM Quanlong Huang < >> [email protected] >> > > >> > >> wrote: >> > >> >> > >> > Then branch-2.x needs this again: >> > https://gerrit.cloudera.org/c/12339/. >> > >> > Could anyone have a look at this? I encountered a build error at >> > >> > https://gerrit.cloudera.org/c/12429/ due to this. >> > >> > >> > >> > Thanks, >> > >> > Quanlong >> > >> > >> > >> > On Wed, Feb 6, 2019 at 1:15 AM Philip Zeyliger < >> [email protected]> >> > >> > wrote: >> > >> > >> > >> >> Thanks! (And Jim suggested the same thing.) >> > >> >> >> > >> >> My apologies for the churn. What I didn't realize is that >> > >> >> https://jenkins.impala.io/job/parallel-all-tests/build takes a >> > >> >> "IMPALA_LZO_BRANCH" parameter. I had foolishly assumed that the >> > branch >> > >> >> logic was in >> > >> >> >> > >> >> >> > >> >> > >> https://gitbox.apache.org/repos/asf?p=impala.git;a=blob;f=bin/bootstrap_system.sh;h=cb869ffe79f3c35cf45b2f962a472274319d746c;hb=HEAD#l378 >> > >> >> , >> > >> >> but it's actually passed through in the Jenkins job. Yesterday >> Lars >> > >> and I >> > >> >> piped some of the stuff through to >> > >> >> >> https://jenkins.impala.io/job/gerrit-verify-dryrun/build?delay=0sec >> > as >> > >> >> well. >> > >> >> >> > >> >> Again, apologies for causing the churn. I should have worked it >> out >> > in >> > >> >> more >> > >> >> detail. Please continue to let me know if things are still broken. >> > >> >> >> > >> >> -- Philip >> > >> >> >> > >> >> On Tue, Feb 5, 2019 at 1:00 AM Tim Armstrong < >> > [email protected]> >> > >> >> wrote: >> > >> >> >> > >> >> > I just changed the default to master, because that would make >> > sense. >> > >> >> > >> > >> >> > On Tue, Feb 5, 2019 at 6:56 AM Quanlong Huang < >> > >> [email protected]> >> > >> >> > wrote: >> > >> >> > >> > >> >> > > Oh, I thought the default is master. Just abandoned my patch >> > since >> > >> >> it's >> > >> >> > > redundant. There's a relative patch for impala-3.x to use the >> > >> master >> > >> >> > > branch: >> > >> >> > > >> > >> >> > > >> > >> >> > >> > >> >> >> > >> >> > >> https://github.com/apache/impala/commit/365e35a36f142532044d8a34cbb87646bf9cb240 >> > >> >> > > >> > >> >> > > On Sun, Feb 3, 2019 at 4:40 AM Jim Apple < >> [email protected]> >> > >> >> wrote: >> > >> >> > > >> > >> >> > > > Do you want to change default branch to master, rather than >> > >> >> cdh5-trunk? >> > >> >> > > > >> > >> >> > > > >> https://help.github.com/articles/setting-the-default-branch/ >> > >> >> > > > >> > >> >> > > > On Fri, Feb 1, 2019 at 3:51 PM Philip Zeyliger < >> > >> [email protected] >> > >> >> > >> > >> >> > > > wrote: >> > >> >> > > > >> > >> >> > > > > "cdh5-trunk" was that branch, but I also pushed it to 2.x, >> > for >> > >> a >> > >> >> bit >> > >> >> > > more >> > >> >> > > > > consistency. >> > >> >> > > > > >> > >> >> > > > > -- Philip >> > >> >> > > > > >> > >> >> > > > > On Fri, Feb 1, 2019 at 3:44 PM Quanlong Huang < >> > >> >> > [email protected] >> > >> >> > > > >> > >> >> > > > > wrote: >> > >> >> > > > > >> > >> >> > > > > > Hi Philip, >> > >> >> > > > > > >> > >> >> > > > > > Thanks for your kindly reminder! Could you help to >> create a >> > >> tag >> > >> >> in >> > >> >> > > the >> > >> >> > > > > > cloudera/Impala-lzo project to point to the previous >> > commit? >> > >> >> Then >> > >> >> > > > > > it's convenient for branch-2.x to point to a pined >> version. >> > >> >> > > > > > >> > >> >> > > > > > Thanks, >> > >> >> > > > > > Quanlong >> > >> >> > > > > > >> > >> >> > > > > > On Sat, Feb 2, 2019 at 5:33 AM Philip Zeyliger < >> > >> >> > [email protected]> >> > >> >> > > > > > wrote: >> > >> >> > > > > > >> > >> >> > > > > > > Hi Fredy, >> > >> >> > > > > > > >> > >> >> > > > > > > I think you need to rebase your change onto mine. I've >> > just >> > >> >> done >> > >> >> > so >> > >> >> > > > by >> > >> >> > > > > > > clicking the button, and we'll see. >> > >> >> > > > > > > >> > >> >> > > > > > > -- Philip >> > >> >> > > > > > > >> > >> >> > > > > > > On Fri, Feb 1, 2019 at 12:24 PM Fredy Wijaya < >> > >> >> > [email protected] >> > >> >> > > > >> > >> >> > > > > > wrote: >> > >> >> > > > > > > >> > >> >> > > > > > > > gerrit-code-review-checks seems a bit unhappy with >> the >> > >> >> change, >> > >> >> > > e.g. >> > >> >> > > > > > > > >> > >> >> > > > > > > > >> > >> >> > > > > > > >> > >> >> > > > > > >> > >> >> > > > > >> > >> >> > > > >> > >> >> > > >> > >> >> > >> > >> >> >> > >> >> > >> https://jenkins.impala.io/job/gerrit-code-review-checks/1963/artifact/https%3A%5E%5Ejenkins.impala.io%5Ejob%5Eclang-tidy-ub1604%5E5041%5E/*view*/ >> > >> >> > > > > > > > >> > >> >> > > > > > > > + ./buildall.sh -skiptests -tidy -so -noclean >> > >> >> > > > > > > > + echo 'tidy build failed; See >> /home/ubuntu/tidylog.txt >> > >> for >> > >> >> > full >> > >> >> > > > tidy >> > >> >> > > > > > > > build output. Guess:' >> > >> >> > > > > > > > tidy build failed; See /home/ubuntu/tidylog.txt for >> > full >> > >> >> tidy >> > >> >> > > build >> > >> >> > > > > > > > output. Guess: >> > >> >> > > > > > > > + grep ': error: ' /home/ubuntu/tidylog.txt >> > >> >> > > > > > > > >> > /home/ubuntu/Impala-lzo/hdfs-lzo-text-scanner.cc:140:17: >> > >> >> error: >> > >> >> > > > > > > > ‘class impala::HdfsScanNodeBase’ has no member >> > named >> > >> >> > > > > > > > ‘UpdateRemainingScanRangeSubmissions’ >> > >> >> > > > > > > > >> > /home/ubuntu/Impala-lzo/hdfs-lzo-text-scanner.cc:168:14: >> > >> >> error: >> > >> >> > > > > > > > ‘class impala::HdfsScanNodeBase’ has no member >> > named >> > >> >> > > > > > > > ‘UpdateRemainingScanRangeSubmissions’ >> > >> >> > > > > > > > >> > /home/ubuntu/Impala-lzo/hdfs-lzo-text-scanner.cc:169:59: >> > >> >> error: >> > >> >> > > no >> > >> >> > > > > > > > matching function for call to >> > >> >> > > > > > > > >> > >> >> > > > > > > > >> > >> >> > > > > > > >> > >> >> > > > > > >> > >> >> > > > > >> > >> >> > > > >> > >> >> > > >> > >> >> > >> > >> >> >> > >> >> > >> ‘impala::HdfsScanNodeBase::AddDiskIoRanges(std::vector<impala::io::ScanRange*>&)’ >> > >> >> > > > > > > > >> > /home/ubuntu/Impala-lzo/hdfs-lzo-text-scanner.cc:200:49: >> > >> >> error: >> > >> >> > > no >> > >> >> > > > > > > > matching function for call to >> > >> >> > > > > > > > >> > >> >> > > > > > > > >> > >> >> > > > > > > >> > >> >> > > > > > >> > >> >> > > > > >> > >> >> > > > >> > >> >> > > >> > >> >> > >> > >> >> >> > >> >> > >> ‘impala::HdfsScanNodeBase::AddDiskIoRanges(std::vector<impala::io::ScanRange*>)’ >> > >> >> > > > > > > > >> > >> >> > > > > > > > >> > >> >> > > > > > > > On Fri, Feb 1, 2019 at 1:20 PM Philip Zeyliger < >> > >> >> > > > [email protected]> >> > >> >> > > > > > > > wrote: >> > >> >> > > > > > > > >> > >> >> > > > > > > > > Hi folks, >> > >> >> > > > > > > > > >> > >> >> > > > > > > > > I just pushed >> > >> >> > > > > > > > > >> > >> >> > > > > > > > > >> > >> >> > > > > > > > >> > >> >> > > > > > > >> > >> >> > > > > > >> > >> >> > > > > >> > >> >> > > > >> > >> >> > > >> > >> >> > >> > >> >> >> > >> >> > >> https://gitbox.apache.org/repos/asf?p=impala.git;a=commit;h=a8e30506aafef14646d95a56fb87cf7c28d259d6 >> > >> >> > > > > > > > > and >> > >> >> > > > > > > > > >> > >> >> > > > > > > > > >> > >> >> > > > > > > > >> > >> >> > > > > > > >> > >> >> > > > > > >> > >> >> > > > > >> > >> >> > > > >> > >> >> > > >> > >> >> > >> > >> >> >> > >> >> > >> https://github.com/cloudera/impala-lzo/commit/dccb1be88a5e237b06ae69cd99b048a38d9f024b >> > >> >> > > > > > > > > to tackle IMPALA-7980. If you're working on >> master, >> > and >> > >> >> you >> > >> >> > > have >> > >> >> > > > > > > > > ../Impala-lzo checked out, you'll need to pull >> that >> > >> repo. >> > >> >> > > There's >> > >> >> > > > > > also >> > >> >> > > > > > > a >> > >> >> > > > > > > > > divergence now for Impala-lzo between branches, >> so if >> > >> >> you're >> > >> >> > > > still >> > >> >> > > > > > > > working >> > >> >> > > > > > > > > on 2.x (say), you'll need to figure out how to >> manage >> > >> >> that. >> > >> >> > > > > > > > > >> > >> >> > > > > > > > > Please let me know if you run into issues; I'll be >> > >> happy >> > >> >> to >> > >> >> > > help. >> > >> >> > > > > > > > > >> > >> >> > > > > > > > > -- Philip >> > >> >> > > > > > > > > >> > >> >> > > > > > > > >> > >> >> > > > > > > >> > >> >> > > > > > >> > >> >> > > > > >> > >> >> > > > >> > >> >> > > >> > >> >> > >> > >> >> >> > >> > >> > >> >> > > >> > >> >
