Oh, that's great. Would you like to report a bug to Bigtop JIRA and attach a patch to fix it? A patch can be applied when building. Just need to placing a .diff patch file under bigtop-packages/src/common/hadoop. You can find two examples there: * patch0-HADOOP-12366.-expose-calculated-paths.diff * patch2-HADOOP-14597.diff
Alexander Frolushkin <[email protected]> 於 2020年1月27日 週一 22:54 寫道： > Sorry, if my text somehow made you worry. Of cause this is OSS and I just > want to clearly understand current situation with it. > I already manually fixed hadoop-rpm building task via pom.xml url change to > https. Of cause it is a local workaround. > Anyway big thanks for your help and explanation of some details of how > Bigtop works. > > > On Mon, Jan 27, 2020, 21:46 Evans Ye <[email protected]> wrote: > > > Alex, > > > > Thanks for reporting the issue, but let's not drive the discussion in > this > > way. This is an open source project and we're just trying to make > software > > for public good. We don't have the resource to compete with commercial > > distribution vendor(s). If you have an idea to improve the build quality. > > We're happy to learn from you :) > > > > IMHO either way our latest release can't build. Let's figure out how to > fix > > it first. Then we can come up with a quick bugfix release with just the > > apache tar.gz release. No need to publish the artifacts/repos. > > > > I was trying to fix the problem but I have limited time so it can be > slow. > > Alex if you have some finding please share to us. Technical discussions > are > > always welcome. > > > > Evans > > > > Konstantin Boudnik <[email protected]> 於 2020年1月27日 週一 上午5:55寫道： > > > > > Actually, it isn't about Bigtop build. Bigtop is using whatever > > > source/build > > > materials are particular component is supplying as a part of their > > > release' source > > > tree. Yes, we do an alternation or two when the "official" stuff is > > totally > > > broken but it is rather an exception. > > > > > > In this particular case, Hadoop's builds have fixed on the fly by our > own > > > 'do-component-build' script or to be fixed either in the upstream > > project. > > > > > > -- > > > Cos > > > > > > On Sun, Jan 26, 2020 at 12:47PM, Alexander Frolushkin wrote: > > > > So, someone need to remade src.rpm for Hadoop, replacing url in .pom > > for > > > > this resource to become available? And now Hadoop cannot be build by > > any > > > > version of Bigtop? > > > > > > > > On Fri, Jan 24, 2020, 12:04 Olaf Flebbe <[email protected]> wrote: > > > > > > > > > hi > > > > > > > > > > the failed downloads are http:// downloads > > > > > > > > > > http download from nexus are now turned off for security reasons. > > > > > > > > > > there was another report for oozie with the same problem on this > list > > > iirc. > > > > > > > > > > imo this can only be solved by using additional tooling, smthg > like a > > > > > proxy changing http to https . nexus might have a property for > that. > > > or new > > > > > maven might have smthg in place for working around broken > > > dependencies, but > > > > > i am speculating > > > > > > > > > > best > > > > > olaf > > > > > > > > > > > > > > > > > > > > > Am 24.01.2020 um 05:47 schrieb Konstantin Boudnik < > [email protected] > > >: > > > > > > > > > > > > Looks like some sort of trouble with one of the dependencies of > > > httpfs: > > > > > > > > > > > > > > > > > > Downloading from apache.snapshots.https: > > > > > > > > > > > https://repository.apache.org/content/repositories/snapshots/org/apache/maven/skins/maven-default-skin/1.0/maven-default-skin-1.0.jar > > > > > > Downloading from repository.jboss.org: > > > > > > > > > > > http://repository.jboss.org/nexus/content/groups/public/org/apache/maven/skins/maven-default-skin/1.0/maven-default-skin-1.0.jar > > > > > > [WARNING] Checksum validation failed, expected but is > > > > > da39a3ee5e6b4b0d3255bfef95601890afd80709 from repository.jboss. > > > > > > org for > > > > > > > > > > > http://repository.jboss.org/nexus/content/groups/public/org/apache/maven/skins/maven-default-skin/1.0/maven-default-skin-1.0.jar > > > > > > [WARNING] Could not validate integrity of download from > > > > > > > > > > > http://repository.jboss.org/nexus/content/groups/public/org/apache/maven/skins/maven-default-skin/1.0/maven-default-skin-1.0.jar > > > : > > > > > Checksum validation failed, expected but is > > > > > da39a3ee5e6b4b0d3255bfef95601890afd80709 > > > > > > [WARNING] Checksum validation failed, expected but is > > > > > da39a3ee5e6b4b0d3255bfef95601890afd80709 from repository.jboss.org > > for > > > > > > > > > > > http://repository.jboss.org/nexus/content/groups/public/org/apache/maven/skins/maven-default-skin/1.0/maven-default-skin-1.0.jar > > > > > > Downloaded from repository.jboss.org: > > > > > > > > > > > http://repository.jboss.org/nexus/content/groups/public/org/apache/maven/skins/maven-default-skin/1.0/maven-default-skin-1.0.jar > > > > > (0 B at 0 B/s) > > > > > > > > > > > > And then build fails right at that module. Doesn't look like a > > Bigtop > > > > > issue, methink. I am on the road right now with somewhat limited > > > internet, > > > > > will be back over the weekend and try to look more. Perhaps in the > > > > > meanwhile someone on the list would have any other ideas. > > > > > > > > > > > > -- > > > > > > Cos > > > > > > > > > > > >> On 1/23/20 11:40 PM, Konstantin Boudnik wrote: > > > > > >> Well, I am not familiar with Gradle enterprise - looks like it > has > > > > > everything indeed, thanks! Let me dig and see what I can figure > out. > > > > > >> > > > > > >> -- > > > > > >> Cos > > > > > >> > > > > > >>> On 1/23/20 11:28 PM, Alexander Frolushkin wrote: > > > > > >>> Thank you for responding, Konstantin. > > > > > >>> > > > > > >>> Honestly, I was thinking --scan gradle will publish all > necessary > > > > > >>> information about its execution, which may be required to > > > inverstigate > > > > > the > > > > > >>> problem. > > > > > >>> > > > > > >>> Anyway, I'm ready to provide maximum data for this dicsussion. > > > Firstly, > > > > > >>> here is a complete console output of entire hadoop-pkg-ind task > > > till > > > > > the > > > > > >>> error - > > > > > >>> > > > > > > > > > > > https://drive.google.com/file/d/16zmFTj485RQ44BFMh5zlhB4cyUzmyaNR/view?usp=drivesdk > > > > > >>> > > > > > >>> If I need to collect additional data, please direct me, what > > > exactly I > > > > > need > > > > > >>> to do to get it for processing here. > > > > > >>> > > > > > >>> I really want to build hadoop repo by bigtop-1.4.0. > > > > > >>> > > > > > >>>> On Thu, Jan 23, 2020, 22:07 Konstantin Boudnik < > [email protected]> > > > > > wrote: > > > > > >>> > > > > > >>>> Thanks, Alexander, for bringing this up. > > > > > >>>> > > > > > >>>> I don't see anything wrong with the way you're starting the > > build, > > > > > >>>> except that official CI is now using "hadoop-pkg-ind" target > and > > > build > > > > > >>>> then pulls in a correct container automatically. But I see you > > > already > > > > > >>>> tried that one. > > > > > >>>> > > > > > >>>> As a side note, I am looking at [1] and see the following > error > > > (which > > > > > >>>> seems to be there for some time): > > > > > >>>> > > > > > >>>> Plugin [id: 'de.undercouch.download', version: '3.2.0'] was > not > > > found > > > > > in > > > > > >>>> any of the following sources: > > > > > >>>> > > > > > >>>> - Gradle Core Plugins (plugin is not in 'org.gradle' > namespace) > > > > > >>>> - Plugin Repositories (could not resolve plugin artifact > > > > > >>>> > > > 'de.undercouch.download:de.undercouch.download.gradle.plugin:3.2.0') > > > > > >>>> Searched in the following repositories: > > > > > >>>> Gradle Central Plugin Repository > > > > > >>>> > > > > > >>>> Unfortunately, it is hard to tell what kind of error you're > > > getting in > > > > > >>>> your build. Any chance you can publish a snippet of the actual > > > logs, > > > > > >>>> rather than a screenshot? > > > > > >>>> > > > > > >>>> -- > > > > > >>>> Thanks, > > > > > >>>> Cos > > > > > >>>> > > > > > >>>> [1] > > > > > >>>> > > > > > >>>> > > > > > > > > > > > https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/COMPONENTS=hadoop,OS=centos-7/567/console > > > > > >>>> > > > > > >>>> On 1/23/20 4:50 AM, Alexander Frolushkin wrote: > > > > > >>>>> Hello everyone! > > > > > >>>>> > > > > > >>>>> Sorry for repeating this in dev list, but I really need help > in > > > this > > > > > >>>>> situation. > > > > > >>>>> > > > > > >>>>> On bigtop-1.4.0 build fails on hadoop-hdfs-httpfs with error: > > > > > >>>>> > > > > > >>>>> Failed to execute goal > > > > > >>>>> > > > > > >>>> > > > > > > > > > > > org.apache.maven.plugins:maven-project-info-reports-plugin:2.7:dependencies > > > > > >>>>> (default) on project hadoop-hdfs-httpfs: An error has > occurred > > in > > > > > >>>>> Dependencies report generation.: zip file is empty -> [Help > 1] > > > > > >>>>> > > > > > >>>>> Launched in clean bigtop-1.4.0 dir, with command docker run > -v > > > > > `pwd`:/ws > > > > > >>>>> bigtop/slaves:1.4.0-centos-7 bash -l -c 'cd /ws;gradle > > hadoop-pkg > > > > > >>>>> > > > > > >>>>> I made a --scan for this attempt - > > > > > https://gradle.com/s/7iu4z3fsbrh62 > > > > > >>>>> > > > > > >>>>> Maybe I'm using not a valid way to get it perform correctly? > > > > > >>>>> > > > > > >>>>> Or this can be bug and I need to fill a jira ticket for this? > > > > > >>>>> > > > > > >>>>> Also I tryed to use direct building by ./gradlew hadoop-rpm > and > > > via > > > > > >>>> docker > > > > > >>>>> ./gradlew hadoop-pkg-ind, got the same error. If I use > > > bigtop-trunk, > > > > > this > > > > > >>>>> stage (hadoop-hdfs-httpfs) passed without error, but as I've > > been > > > > > told, > > > > > >>>>> currently hadoop is unbuildable in master branch. > > > > > >>>>> > > > > > >>>>> Plus, the same error I get on bigtop-1.3.0. > > > > > >>>>> > > > > > >>>>> Cleaning the local maven repository does not fix it. > > > > > >>>>> > > > > > >>>>> So, likely, the problem is in building environment. How I can > > > > > configure > > > > > >>>> it > > > > > >>>>> correctly for successful hadoop building in bigtop? > > > > > >>>>> > > > > > >>>>> On Wed, Jan 22, 2020, 16:20 Evans Ye <[email protected]> > > wrote: > > > > > >>>>> > > > > > >>>>>> Currently building Hadoop across all platform on master > branch > > > is > > > > > >>>> failing: > > > > > >>>>>> > > > > > > > https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/ > > > > > >>>>>> > > > > > >>>>>> I'm not sure whether the master code can be built > successfully > > > since > > > > > >>>> we're > > > > > >>>>>> in transition of upgrading zookeeper. > > > > > >>>>>> > > > > > >>>>>> If you just want to use bigtop, please get our latest > release > > > 1.4.0. > > > > > >>>>>> > > > > > >>>>>> If you are willing to contribute, you can report a bug on > our > > > jira > > > > > first > > > > > >>>>>> and we ca work on a fix together. > > > > > >>>>>> > > > > > >>>>>> Let us know if any question. > > > > > >>>>>> > > > > > >>>>>> Evan's > > > > > >>>>>> > > > > > >>>>>> > > > > > >>>>>> > > > > > >>>>>> Yuqi Gu <[email protected]> 於 2020年1月22日 週三 16:31 寫道： > > > > > >>>>>> > > > > > >>>>>>> Hi Alexander, > > > > > >>>>>>> > > > > > >>>>>>> It seems bigtop_toolchain didn't deploy correctly. > > > > > >>>>>>> Please try to build bigtop components in docker with > > > bigtop/slaves > > > > > >>>> docker > > > > > >>>>>>> images < > > > > > https://hub.docker.com/r/bigtop/slaves/tags?page=1&name=centos > > > > > >>>>> . > > > > > >>>>>>> Run: > > > > > >>>>>>> cd BIGTOP_SRC_DIR > > > > > >>>>>>> docker run -v `pwd`:/ws bigtop/slaves:1.4.0-centos-7 bash > -l > > > -c 'cd > > > > > >>>> /ws ; > > > > > >>>>>>> gradle hadoop-pkg' > > > > > >>>>>>> > > > > > >>>>>>> BR, > > > > > >>>>>>> Yuqi > > > > > >>>>>>> > > > > > >>>>>>> On Wed, 22 Jan 2020 at 15:05, Alexander Frolushkin < > > > > > [email protected] > > > > > >>>>>>> wrote: > > > > > >>>>>>> > > > > > >>>>>>>> I have this error on bigtop-trunk. In webtop-1.4.0 it > fails > > > with > > > > > other > > > > > >>>>>>>> error. > > > > > >>>>>>>> > > > > > >>>>>>>> Also, in webtop-trunk this particular error appeard while > > > building > > > > > >>>>>> hadoop > > > > > >>>>>>>> in docker (trunk-centos-7) image (by ./gradlew > > > hadoop-pkg-ind) and > > > > > >>>>>>> without > > > > > >>>>>>>> it (by ./gradlew hadoop-rpm). > > > > > >>>>>>>> > > > > > >>>>>>>> Performed steps: > > > > > >>>>>>>> > > > > > >>>>>>>> I was trying to follow the available guides, do a clean > > > centos 7 > > > > > >>>>>> install, > > > > > >>>>>>>> upgrade to latest official rpms, install openjdk 7 and > unzip > > > from > > > > > >>>> repo. > > > > > >>>>>>>> Then puppetize.sh from bigtop_toolchain/bin and puppet > apply > > > > > >>>>>>>> --modulepath=`pwd`:/etc/puppet/modules -e "include > > > > > >>>>>>>> bigtop_toolchain::installer". > > > > > >>>>>>>> > > > > > >>>>>>>> After this run ./gradlew hadoop-pkg-ind or ./gradlew > > > hadoop-rpm > > > > > and > > > > > >>>> got > > > > > >>>>>>>> this error. > > > > > >>>>>>>> > > > > > >>>>>>>> > > > > > >>>>>>>> > > > > > >>>>>>>> On Wed, Jan 22, 2020, 12:56 Youngwoo Kim (김영우) < > > > [email protected]> > > > > > >>>>>> wrote: > > > > > >>>>>>>>> Hi Alexander, > > > > > >>>>>>>>> > > > > > >>>>>>>>> Which version or branch of Bigtop do you use? and also > can > > > you > > > > > >>>>>> provide > > > > > >>>>>>> us > > > > > >>>>>>>>> the steps you did? > > > > > >>>>>>>>> > > > > > >>>>>>>>> Thanks, > > > > > >>>>>>>>> Youngwoo > > > > > >>>>>>>>> > > > > > >>>>>>>>> > > > > > >>>>>>>>> > > > > > >>>>>>>>> On Wed, Jan 22, 2020 at 2:00 PM Alexander Frolushkin < > > > > > >>>>>>> [email protected] > > > > > >>>>>>>>> wrote: > > > > > >>>>>>>>> > > > > > >>>>>>>>>> Hello everyone. > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> Looks like Bigtop mailing lists is not very active, but > I > > > still > > > > > >>>>>> hope > > > > > >>>>>>>>>> someone could help me. > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> We are trying to employ BigTop to build own hadoop > > > repository > > > > > >>>>>>>> (./gradlew > > > > > >>>>>>>>>> hadoop-rpm), but after all efforts still getting error > on > > > > > zookeeper > > > > > >>>>>>>>> version > > > > > >>>>>>>>>> dependency: > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> > > -org.apache.hadoop:hadoop-yarn-server-resourcemanager:2.8.5 > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> it wants -org.apache.zookeeper:zookeeper:3.4.6 > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> while all other conponents depends on > > > > > >>>>>>>>>> -org.apache.zookeeper:zookeeper:3.4.13 > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> so we got: > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> [ERROR] Failed to execute goal > > > > > >>>>>>>>>> > > org.apache.maven.plugins:maven-enforcer-plugin:1.3.1:enforce > > > > > >>>>>>> (depcheck) > > > > > >>>>>>>>> on > > > > > >>>>>>>>>> project hadoop-yarn-server-tests > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> Please advise me some solution for this show stopper, if > > > > > possible. > > > > > >>>>>>>>> Thanks! > > > > > > > > > > > > > > > >
