[ANNOUNCE] New Bigtop PMC member: Luca Toscano

[Olaf Flebbe]

On behalf of the Apache Bigtop PMC, I am pleased to announce that
Luca Toscano (elukey) has accepted the invitation to join the 
Bigtop Project Management Committee.

Please join me in congratulating Luca!

Re: Preparation status of the 3.0.1 release

[Olaf Flebbe]

Hi Peng,


IIUC there wasn’t any functional Ambari community in place any more, 
so the project went into attic.

As an Apache Software Project we are asked to look into software security
and maintaibility. We cannot use discontinued / obsolete technology for
building infrastructure, since users expect us to be able to react to security
challenges. 

Bigtop does included a lot of 3rd party software, but each of them was only 
chosen 
when the bigtop community was convinced the project will be maintained in 
future and the license
is compatible to the Apache Software License Version 2. Usually we packaged 
Apache Projects.

Patches within Bigtop had been submitted upstream.
If these projects cease to exist or are stalled, Apache Bigtop has to remove 
the dependency sooner or later. 
Ambari is not the first and really not the only component which we have to 
consider removing. 
(Other candidates are the leftovers of the ODPI Initiative, for example )

It is possible to revive Apache Ambari from the Attic and change the project 
goals in almost any direction, 
The process for leaving the Attic is outlined here : https://attic.apache.org 



Best,
Olaf


> Am 07.04.2022 um 10:39 schrieb 李帅 :
> 
> Hi Michiel,
> It will be a web project originated from ambari earlier branch. Its
> back-end, we will refactor, make it more readable and succinct.
> When it finished and tested on schedule, I will try to push it into bigtop
> repo(make it work well with bigtop package system).
> Thanks,
> Peng Lee.
> 
> Michiel Verheul  于2022年4月7日周四 02:01写道：
> 
>> Thank you all for responding on this subject.
>> I agree with Kengo that these mpacks will be horrible to maintain and I can
>> understand that it's (at least) questionable to include that in the bigtop
>> repository.
>> Maybe I will push my current (WIP) mpack to my local GitHub, but it feels
>> like the right way forward would be 李帅's work.
>> @李帅: Do I understand correctly that your work (when complete) will result
>> in a new separate bigtop component, independent from Ambari?
>> 
>> Michiel
>> 
>> Op wo 6 apr. 2022 13:21 schreef 李帅 :
>> 
>>> Now, there are no replacements for Ambari. I am refactoring an earlier
>>> branch ambari to make it work well with
>>> bigtop  components. it will be one web gui tool for bigtopers to deploy
>> and
>>> monitor components (no need
>>> install agents).
>>> 
>>> John Gibson  于2022年4月6日周三 10:34写道：
>>> 
 Hi Michiel,
 
 I really appreciate your offer, that sounds very useful for the HDP
>>> users.
 But to be honest, I'm afraid I'm a bit reluctant to include it into
>>> Bigtop,
 due to the following reasons.
 
 1. Ambari has already been retired this January [1].
   Keeping retired software as a Bigtop component
   brings us security risks and maintenance cost.
   Especially, Ambari depends on several obsolete
   softwares such as Python2 and Bower.
   So I personally think we should drop Ambari
   (and also Mpack) at some point in the future,
   unless it revives.
 
 2. Cloudera has changed the license of their product [2],
   so I'm not sure if we could do that. Doesn't it violate
   their license to install and use HDP without subscription?
   If the new Mpack is derived from their product,
   could we include it in our distributions
   from the viewpoint of license compatibility [3]?
 
 But the above is just my personal opinion, so I'd like to hear from
>>> others.
 
 [1]: https://lists.apache.org/thread/m5jrpn4j28kn3wfn4zzxvy0g450vdlr1
 [2]: https://www.cloudera.com/downloads/paywall-expansion.html
 [3]: https://www.apache.org/legal/resolved.html
 
 Kengo Seki http://apache.org/>>
 
 
 Hi Kengo,
 
 Sorry if you get this twice; I wasn't subscribed when I made my initial
 reply.
 
 In regards to #2, licensing. Even if HDP did change the license when
>> they
 expanded their paywall that could only apply to newly downloaded
>> copies.
 The terms of the Apache, GPL, and most other open source licenses do
>> not
 allow the author to rescind rights to users who previously obtained the
 software. So anyone who created a mirror of the legacy HDP repositories
 prior to the paywall deployment (3.1.4 and earlier, I believe) would
 benefit from Michiel's work. Users without an existing legacy mirror of
>>> the
 HDP repositories could only benefit if they paid to access Cloudera's
 repositories. I'm not sure how many existing HDP users would like to
>> move
 to BigTop, but this would be a nice stepping stone. The lack of a
>> working
 MPack in BigTop is currently limiting our use of BigTop.
 
 Are there any obvious replacements for Ambari going forward? Or would
>> we
 just be left without a stack-management API and GUI? The dependence
>> upon
 Python 2.7 is not good, but also not a showstopper for our deployments
>> on

Re: Followup on yesterday

[Olaf Flebbe]

Hi Luca,

as I was mentioning in the JIRA Ticket. There seems to be an underdocumented 
gradle feature to use a maven mirror.

https://rahulsom.github.io/blog/2016/gradle-equivalent-of-maven-mirror.html 
<https://rahulsom.github.io/blog/2016/gradle-equivalent-of-maven-mirror.html>

Olaf

> Am 20.02.2022 um 11:00 schrieb Luca Toscano :
> 
> Thanks for the explanation! I was able to use nsenter on
> docker-worker-7 to check Nexus logs, and I see artifacts being
> retrieved (mostly Maven related). I also see some logs like the
> following (always for apache.snapshots), repeated every hour:
> 
> 2022-02-20 05:57:06,677+ WARN  [ar-7-thread-4] *TASK
> org.sonatype.nexus.proxy.maven.routing.internal.RemoteContentDiscovererImpl
> - Remote strategy prefix-file error on
> M2Repository(id=apache.snapshots):
> org.apache.http.conn.ConnectTimeoutException: Connect to
> repository.apache.org:443 [repository.apache.org/136.243.146.148]
> failed: connect timed out
> 
> I also filed https://issues.apache.org/jira/browse/BIGTOP-3644 (and
> https://github.com/apache/bigtop/pull/865) as attempt to improve the
> Gradle download (that IIUC doesn't use any proxy but that could use
> some retry logic).
> 
> WIll keep investigating, if anybody has suggestions please let me know :)
> 
> Luca
> 
> On Sat, Feb 19, 2022 at 7:21 PM Olaf Flebbe  wrote:
>> 
>> Hi,
>> 
>> glad you are asking how to make sure that a caching server is used:
>> 
>> https://github.com/apache/bigtop/blob/8c323c4f12534508b6ffb45603db7cbf6e0a145f/build.gradle#L477
>>  
>> <https://github.com/apache/bigtop/blob/8c323c4f12534508b6ffb45603db7cbf6e0a145f/build.gradle#L477>
>> 
>> The gradle task "configure-nexus“
>> is configuring $HOME/.m2/settings.xml file for maven to download for 
>> instance maven central via nexus instead of downloading from maven central 
>> directly.
>> 
>> However this will only work for maven, not handling gradle or ivy builds 
>> AFAIK.
>> 
>> Best
>>Olaf
>> 
>> 
>>> Am 19.02.2022 um 14:23 schrieb Luca Toscano :
>>> 
>>> Hi everybody,
>>> 
>>> I have one doubt related to Nexus, namely when it is used. I tried to
>>> check [1] as random example, and I see that we trigger at some point
>>> the configure-nexus gradle code, but when the debuild script kicks in,
>>> I see stuff like:
>>> 
>>> + mvn clean install -DskipTests -Dhadoop.version=3.2.2
>>> -Dmaven.buildNumber.revisionOnScmFailure=v2.4.1 -Phadoop-3 -Pyarn
>>> -Dmaven.repo.local=/var/lib/jenkins/.m2/repository
>>> 
>>> Is the mvn command launched inside debuild using the nexus cache?
>>> 
>>> Luca
>>> 
>>> 
>>> [1] 
>>> https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/COMPONENTS=alluxio,OS=debian-11/lastBuild/consoleFull
>>> 
>>> On Thu, Feb 17, 2022 at 9:48 PM Olaf Flebbe  wrote:
>>>> 
>>>> Hi everyone,
>>>> 
>>>> Yesterday Luca Toscano and me had a call to look into improving the 
>>>> situation of artifact downloads by caching .
>>>> 
>>>> I was a bit surprised that the „nexus“ code is still in place and still 
>>>> seem to work somehow.
>>>> 
>>>> Since having a repository server (a specialized proxy for maven repos) is 
>>>> technically a much cleaner solution than messing with the .m2/repository 
>>>> maven cache -- since it can be shared across architectures and os and even 
>>>> support more built tools -- I would like to step back from my proposal to 
>>>> use docker volumes to share the raw m2 cache between instances.
>>>> 
>>>> What need to be done is to either update to a current version of nexus or 
>>>> switch to a different maven proxy which can be setup , updated and 
>>>> configured more easily.
>>>> 
>>>> I asked a search machine for alternatives and tripped over this project
>>>> https://github.com/jenkins-x/bucketrepo
>>>> which promises to be a low-footprint minimal replacement for nexus, which 
>>>> could even use S3 as a backing store.
>>>> 
>>>> There was a configuration example for nginx as well :
>>>> https://github.com/lkiesow/weblog.lkiesow.de/blob/master/20170413-nginx-as-fast-maven-repository-proxy.md
>>>> 
>>>> Best
>>>> Olaf
>> 

Re: Followup on yesterday

[Olaf Flebbe]

Hi,

glad you are asking how to make sure that a caching server is used:

https://github.com/apache/bigtop/blob/8c323c4f12534508b6ffb45603db7cbf6e0a145f/build.gradle#L477
 
<https://github.com/apache/bigtop/blob/8c323c4f12534508b6ffb45603db7cbf6e0a145f/build.gradle#L477>

The gradle task "configure-nexus“
is configuring $HOME/.m2/settings.xml file for maven to download for instance 
maven central via nexus instead of downloading from maven central directly.

However this will only work for maven, not handling gradle or ivy builds AFAIK.

Best 
Olaf


> Am 19.02.2022 um 14:23 schrieb Luca Toscano :
> 
> Hi everybody,
> 
> I have one doubt related to Nexus, namely when it is used. I tried to
> check [1] as random example, and I see that we trigger at some point
> the configure-nexus gradle code, but when the debuild script kicks in,
> I see stuff like:
> 
> + mvn clean install -DskipTests -Dhadoop.version=3.2.2
> -Dmaven.buildNumber.revisionOnScmFailure=v2.4.1 -Phadoop-3 -Pyarn
> -Dmaven.repo.local=/var/lib/jenkins/.m2/repository
> 
> Is the mvn command launched inside debuild using the nexus cache?
> 
> Luca
> 
> 
> [1] 
> https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/COMPONENTS=alluxio,OS=debian-11/lastBuild/consoleFull
> 
> On Thu, Feb 17, 2022 at 9:48 PM Olaf Flebbe  wrote:
>> 
>> Hi everyone,
>> 
>> Yesterday Luca Toscano and me had a call to look into improving the 
>> situation of artifact downloads by caching .
>> 
>> I was a bit surprised that the „nexus“ code is still in place and still seem 
>> to work somehow.
>> 
>> Since having a repository server (a specialized proxy for maven repos) is 
>> technically a much cleaner solution than messing with the .m2/repository 
>> maven cache -- since it can be shared across architectures and os and even 
>> support more built tools -- I would like to step back from my proposal to 
>> use docker volumes to share the raw m2 cache between instances.
>> 
>> What need to be done is to either update to a current version of nexus or 
>> switch to a different maven proxy which can be setup , updated and 
>> configured more easily.
>> 
>> I asked a search machine for alternatives and tripped over this project
>> https://github.com/jenkins-x/bucketrepo
>> which promises to be a low-footprint minimal replacement for nexus, which 
>> could even use S3 as a backing store.
>> 
>> There was a configuration example for nginx as well :
>> https://github.com/lkiesow/weblog.lkiesow.de/blob/master/20170413-nginx-as-fast-maven-repository-proxy.md
>> 
>> Best
>>  Olaf

Followup on yesterday

[Olaf Flebbe]

Hi everyone,

Yesterday Luca Toscano and me had a call to look into improving the situation 
of artifact downloads by caching .

I was a bit surprised that the „nexus“ code is still in place and still seem to 
work somehow. 

Since having a repository server (a specialized proxy for maven repos) is 
technically a much cleaner solution than messing with the .m2/repository maven 
cache -- since it can be shared across architectures and os and even support 
more built tools -- I would like to step back from my proposal to use docker 
volumes to share the raw m2 cache between instances.

What need to be done is to either update to a current version of nexus or 
switch to a different maven proxy which can be setup , updated and configured 
more easily.

I asked a search machine for alternatives and tripped over this project 
https://github.com/jenkins-x/bucketrepo 

which promises to be a low-footprint minimal replacement for nexus, which could 
even use S3 as a backing store.

There was a configuration example for nginx as well : 
https://github.com/lkiesow/weblog.lkiesow.de/blob/master/20170413-nginx-as-fast-maven-repository-proxy.md
 


Best 
  Olaf 

Re: Network failures while maven runs

[Olaf Flebbe]

Hi Luca,

I think you are referring to random failures of maven downloads?

We used a couple of countermeasures in the past, all of them proved problematic.

We persisted the maven cache so the next by bind mounting a local directory 
onto the maven cache .m2 . We had nice permission issues and didn‘t triggered 
when a maven repository went out of business: Our build suceeded but everyone 
else wasn‘t able to build any more. Since everything now converges to maven 
central this might not be that big issue any more. 

A different approach was to use a proxy maven repo, i remember we had issues 
because it somehow didn‘t work, and we have the hurdle of yet another tool to 
master. 

Both of these worked as a workaround, somehow to cut releases.

On my daytime job the devs configured their job to prepopulate the maven cache 
, but I am not a fan of this approach.

My next proposal would be to use docker volumes (not bind mounts) for that, so 
we wont have any permission problems any more and we are left with the server 
issues going out if business, eventually. To mitigate that it would be easy to 
simply recreate docker volumes rather to go to all filesystems as root removing 
directories we needed for the bind mount workaround.

We could eventually pair to try to implement that, if you are not too familiar 
with it.
https://docs.docker.com/storage/volumes/

Best
Olaf


> Am 29.01.2022 um 08:31 schrieb Luca Toscano :
> 
> Hi everybody,
> 
> I have been seeing build failures for trunk packages due to maven
> network failures (mostly connection timeouts). Is there a workaround
> for this? For example retrying x times etc.. I am asking mostly
> because of ignorance, I am wondering what we do when cutting a release
> for example (to avoid re-running the main package build job due to
> some package build failures).
> 
> Thanks!
> 
> Luca

New Committer: Luca Toscano

[Olaf Flebbe]

The Project Management Committee (PMC) for Apache Bigtop has asked
Luca Toscano to become a committer and we are pleased to announce 
that he has accepted!

Being a committer enables easier contribution to the project since there is
no need to go via the proxy patch submission process. This should enable better
productivity.

Apache Bigtop practices CTR (commit-then-review) development process which
means that you can checkin the code without a +1 from a committer. This
doesn't mean that you're encouraged to do so. Instead, we strongly recommend 
you to
seek for feedback before you commit. Please read the document before you
act:

* https://github.com/apache/bigtop#ctr-model 


If in doubt, exercise your best judgement and don't hesitate to ask
questions on the dev@bigtop.apache.org  mailing 
list.

We're happy to have you on board and looking for many more contributions to
come. Please join me in congratulating Luca Toscano !

Regards,
  Olaf  (on behalf of the ASF BigTop PMC)

Re: ci.bigtop.apache.org's TLS cert expired

[Olaf Flebbe]

Hi,


> Am 18.12.2021 um 13:30 schrieb Luca Toscano :
> 
> Thanks Kengo! Certs renewed :)
> 
> One nit - the name of the docker container to stop is
> "jenkins-master-8080", not "jenkins-master" (maybe it doesn't even
> need to be stopped? Is stopping httpd sufficient?)

For renewing the CERT restarting of httpd should be sufficient.

Olaf

Re: Some ideas about Jenkins

[Olaf Flebbe]

hi luca

we can do the update together in the next days (evening local time in eu). 

best 
olaf

> Am 15.12.2021 um 09:37 schrieb Luca Toscano :
> 
> Hi everybody,
> 
> Any feedback? I'd like to upgrade, if everybody agrees, Jenkins during
> the next days. If anybody can review the procedure and give me a +1/-1
> I'd be grateful :)
> Moreover, if anybody wants to be online with me when I do the upgrade
> it would be really great, so if anything goes wrong there will be more
> people watching. The upgrade itself shouldn't last long (10/15 mins if
> everything goes fine).
> 
> Thanks in advance!
> 
> Luca
> 
>> On Sat, Dec 11, 2021 at 10:01 AM Luca Toscano  wrote:
>> 
>> Hi everybody,
>> 
>> I opened a couple of Jiras for Jenkins:
>> - https://issues.apache.org/jira/browse/BIGTOP-3611 - Upgrade Jenkins
>> to the latest upstream
>> - https://issues.apache.org/jira/browse/BIGTOP-3612 - Add a backup for
>> Jenkins' /home/jenkins dir
>> 
>> I didn't find anything open for these topics, apologies in advance in
>> case there is known work in progress.
>> 
>> Let me know your thoughts :)
>> 
>> Luca

Re: Power (ppc64le) CI/CD upgrade

[Olaf Flebbe]

Hi,

The node still has the same IP adress, was able to login and set the docker 
credentials.

However: the machine asks for installing security updates and a reboot. 
Amir : Do you have a tooling to install updates and reboot ?

Best
Olaf


> Am 11.12.2021 um 10:06 schrieb Luca Toscano :
> 
> Hi Olaf,
> 
> makes sense! I am unable to ssh to the ppc node though (I have some
> trouble decrypting the jenkins ssh key), if you have time later on
> could you please try to ssh and docker login?
> 
> Thanks!
> 
> Luca
> 
> On Sat, Dec 11, 2021 at 9:43 AM Olaf Flebbe  wrote:
>> 
>> hi
>> 
>> after setup a new node you need to do a „docker login“ on the jenkins slave 
>> account once. IIUC this node has been setup only a few days before
>> 
>> olaf
>> 
>>> Am 11.12.2021 um 09:17 schrieb Luca Toscano :
>>> 
>>> Hi!
>>> 
>>> I am very new to the project so I don't have a lot of context on the
>>> ppc64 node, but today I tried to rebuild the docker images and all the
>>> jobs for ppc64 failed:
>>> https://ci.bigtop.apache.org/job/Docker-Puppet-Trunk/34/
>>> 
>>> It is weird since, from the logs, it seems that the node works but it
>>> fails at the end when trying to push to dockerhub:
>>> 
>>> https://ci.bigtop.apache.org/job/Docker-Puppet-Trunk/DISTRO=debian-10,PLATFORM=ppc64le-slave/lastBuild/console
>>> 
>>> Is it possible that some credentials are not available anymore? Does
>>> anybody else have context on this failure?
>>> 
>>> Thanks!
>>> 
>>> Luca
>>> 
>>>> On Thu, Dec 2, 2021 at 9:15 PM MrAsanjar  wrote:
>>>> 
>>>> done, please verify
>>>> 
>>>>> On Thu, Dec 2, 2021 at 12:33 PM MrAsanjar  wrote:
>>>>> 
>>>>> Hi
>>>>> I have good news and bad news :) The good news is IBM has upgraded the
>>>>> Power CI/CD server to a faster processor with 16 cores and NVMe drives.
>>>>> The bad news is during the process, somehow, cloning had failed :( I'll
>>>>> try to build a new Jenkins slave with the same IP.
>>>>> 
>>>>> 
>>>>>> On Fri, Nov 19, 2021 at 7:13 AM MrAsanjar  wrote:
>>>>> 
>>>>>> Hi team,
>>>>>> IBM is planning to upgrade and relocate the apache bigtop server next
>>>>>> week, again.
>>>>>> There will be a short downtime. Hopefully, the IP addresses will not
>>>>>> change. I'll keep you posted
>>>>>> 
>>>>>> On Sun, Oct 24, 2021 at 11:29 AM Evans Ye  wrote:
>>>>>> 
>>>>>>> Thanks for the notice, Amir.
>>>>>>> 
>>>>>>> MrAsanjar  於 2021年10月22日 週五 下午11:57寫道：
>>>>>>> 
>>>>>>>> Hi team,
>>>>>>>> IBM plans to upgrade the current Jenkins slave to high-end Power9 with
>>>>>>> SSD
>>>>>>>> or nvme drives, perhaps next week. There will be a short interruption
>>>>>>> in
>>>>>>>> the availability.
>>>>>>>> 
>>>>>>> 
>>>>>> 
>> 

Re: Power (ppc64le) CI/CD upgrade

[Olaf Flebbe]

hi

after setup a new node you need to do a „docker login“ on the jenkins slave 
account once. IIUC this node has been setup only a few days before

olaf

> Am 11.12.2021 um 09:17 schrieb Luca Toscano :
> 
> Hi!
> 
> I am very new to the project so I don't have a lot of context on the
> ppc64 node, but today I tried to rebuild the docker images and all the
> jobs for ppc64 failed:
> https://ci.bigtop.apache.org/job/Docker-Puppet-Trunk/34/
> 
> It is weird since, from the logs, it seems that the node works but it
> fails at the end when trying to push to dockerhub:
> 
> https://ci.bigtop.apache.org/job/Docker-Puppet-Trunk/DISTRO=debian-10,PLATFORM=ppc64le-slave/lastBuild/console
> 
> Is it possible that some credentials are not available anymore? Does
> anybody else have context on this failure?
> 
> Thanks!
> 
> Luca
> 
>> On Thu, Dec 2, 2021 at 9:15 PM MrAsanjar  wrote:
>> 
>> done, please verify
>> 
>>> On Thu, Dec 2, 2021 at 12:33 PM MrAsanjar  wrote:
>>> 
>>> Hi
>>> I have good news and bad news :) The good news is IBM has upgraded the
>>> Power CI/CD server to a faster processor with 16 cores and NVMe drives.
>>> The bad news is during the process, somehow, cloning had failed :( I'll
>>> try to build a new Jenkins slave with the same IP.
>>> 
>>> 
 On Fri, Nov 19, 2021 at 7:13 AM MrAsanjar  wrote:
>>> 
 Hi team,
 IBM is planning to upgrade and relocate the apache bigtop server next
 week, again.
 There will be a short downtime. Hopefully, the IP addresses will not
 change. I'll keep you posted
 
 On Sun, Oct 24, 2021 at 11:29 AM Evans Ye  wrote:
 
> Thanks for the notice, Amir.
> 
> MrAsanjar  於 2021年10月22日 週五 下午11:57寫道：
> 
>> Hi team,
>> IBM plans to upgrade the current Jenkins slave to high-end Power9 with
> SSD
>> or nvme drives, perhaps next week. There will be a short interruption
> in
>> the availability.
>> 
> 
 

Re: Set up CI for Debian 11 Bullseye

[Olaf Flebbe]

Hi Luca,

the name changed. There is now some space available.

Check with the EC2 console to see the current public IP adresses .
https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Instances:instanceState=running

Best
  Olaf

> Am 04.12.2021 um 18:23 schrieb Luca Toscano :
> 
> Hi :)
> 
> https://ci.bigtop.apache.org/log/all seems to indicate that we are
> again in trouble, docker-slave-02 seems full. I got the credentials to
> access the node but I can reach it (my ssh session hangs
> indefinitely), is there anybody that can take a look?
> 
> Thanks,
> 
> Luca
> 
> On Sat, Nov 27, 2021 at 8:49 PM Olaf Flebbe  wrote:
>> 
>> Hi,
>> 
>> I had a quick look around. Job History was excessive: availlable back to 
>> 2017, I deleted all until 2019 and set the maximum life time to one year.
>> I see that not any job from bigtopstore have been removed so far. Now having 
>> 52995 jobs on disk.
>> I removed all bug the last ~3000 job dirs.
>> 
>> Can someone please look at the config of „bigpetstore“, please?
>> 
>> At least we have some air to breath now again on jenkins.
>> 
>> Olaf
>> 
>> 
>>> Am 27.11.2021 um 20:04 schrieb Olaf Flebbe :
>>> 
>>> Regarding setup of jenkins, you can find a lot of info here:
>>> 
>>> https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+CI+Setup+Guide 
>>> <https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+CI+Setup+Guide>
>>> 
>>> Olaf
>>> 
>>> 
>>>> Am 27.11.2021 um 19:46 schrieb Olaf Flebbe >>> <mailto:o...@oflebbe.de>>:
>>>> 
>>>> I am working with luca to get him access ..
>>>> 
>>>> Olaf
>>>> 
>>>>> Am 27.11.2021 um 18:29 schrieb Luca Toscano >>>> <mailto:toscano.l...@gmail.com>>:
>>>>> 
>>>>> Keep reporting issues, sorry :)
>>>>> 
>>>>> I see from https://ci.bigtop.apache.org/log/all 
>>>>> <https://ci.bigtop.apache.org/log/all> that we are having
>>>>> disk space issues on the Jenkins node, builds are stopped for the
>>>>> moment. No idea if there is a guide/tutorial for the cleanup, but I
>>>>> guess it means accessing to the VM/hosts that are running Jenkins and
>>>>> I have never done it (not even sure where to put my ssh key etc, what
>>>>> is the hostname, etc..).
>>>>> 
>>>>> Thanks!
>>>>> 
>>>>> Luca
>>>>> 
>>>>> On Fri, Nov 26, 2021 at 7:10 PM Olaf Flebbe >>>> <mailto:o...@oflebbe.de>> wrote:
>>>>>> 
>>>>>> Hi Luca,
>>>>>> 
>>>>>> you are right, 201 should be valid. However there seems to be some 
>>>>>> compatibility problem with at least this repo. So comment it out for now 
>>>>>> if it resolves the isse. Or remove the whole proxy stuff altoghether. We 
>>>>>> had stability issues with the internet connectivity at that time. The 
>>>>>> proxy resolved that mostly.
>>>>>> 
>>>>>> Olaf
>>>>>> 
>>>>>> 
>>>>>>> Am 26.11.2021 um 19:06 schrieb Luca Toscano >>>>>> <mailto:toscano.l...@gmail.com>>:
>>>>>>> 
>>>>>>> Hi Olaf,
>>>>>>> 
>>>>>>> I see that https://repository.jboss.org/nexus/content/groups/public/ 
>>>>>>> <https://repository.jboss.org/nexus/content/groups/public/>
>>>>>>> is valid though (http redirects to https), but I don't get why we
>>>>>>> should delete the line (also the code returns a HTTP 201 afaics so it
>>>>>>> seems not failing to find it).
>>>>>>> 
>>>>>>> Lemme know, I am probably missing something, too many new things :)
>>>>>>> 
>>>>>>> Luca
>>>>>>> 
>>>>>>> On Fri, Nov 26, 2021 at 6:24 PM Olaf Flebbe >>>>>> <mailto:o...@oflebbe.de>> wrote:
>>>>>>>> 
>>>>>>>> Hi Luca,
>>>>>>>> 
>>>>>>>> Seems like jboss.org <http://jboss.org/> <http://jboss.org/ 
>>>>>>>> <http://jboss.org/>> repository does not exist any more.
>>>>>>>> 
>>>>>>>> Try to d

Re: Set up CI for Debian 11 Bullseye

[Olaf Flebbe]

Hi,

I had a quick look around. Job History was excessive: availlable back to 2017, 
I deleted all until 2019 and set the maximum life time to one year.
I see that not any job from bigtopstore have been removed so far. Now having 
52995 jobs on disk. 
I removed all bug the last ~3000 job dirs. 

Can someone please look at the config of „bigpetstore“, please?

At least we have some air to breath now again on jenkins.

Olaf


> Am 27.11.2021 um 20:04 schrieb Olaf Flebbe :
> 
> Regarding setup of jenkins, you can find a lot of info here:
> 
> https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+CI+Setup+Guide 
> <https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+CI+Setup+Guide>
> 
> Olaf
> 
> 
>> Am 27.11.2021 um 19:46 schrieb Olaf Flebbe > <mailto:o...@oflebbe.de>>:
>> 
>> I am working with luca to get him access ..
>> 
>> Olaf
>> 
>>> Am 27.11.2021 um 18:29 schrieb Luca Toscano >> <mailto:toscano.l...@gmail.com>>:
>>> 
>>> Keep reporting issues, sorry :)
>>> 
>>> I see from https://ci.bigtop.apache.org/log/all 
>>> <https://ci.bigtop.apache.org/log/all> that we are having
>>> disk space issues on the Jenkins node, builds are stopped for the
>>> moment. No idea if there is a guide/tutorial for the cleanup, but I
>>> guess it means accessing to the VM/hosts that are running Jenkins and
>>> I have never done it (not even sure where to put my ssh key etc, what
>>> is the hostname, etc..).
>>> 
>>> Thanks!
>>> 
>>> Luca
>>> 
>>> On Fri, Nov 26, 2021 at 7:10 PM Olaf Flebbe >> <mailto:o...@oflebbe.de>> wrote:
>>>> 
>>>> Hi Luca,
>>>> 
>>>> you are right, 201 should be valid. However there seems to be some 
>>>> compatibility problem with at least this repo. So comment it out for now 
>>>> if it resolves the isse. Or remove the whole proxy stuff altoghether. We 
>>>> had stability issues with the internet connectivity at that time. The 
>>>> proxy resolved that mostly.
>>>> 
>>>> Olaf
>>>> 
>>>> 
>>>>> Am 26.11.2021 um 19:06 schrieb Luca Toscano >>>> <mailto:toscano.l...@gmail.com>>:
>>>>> 
>>>>> Hi Olaf,
>>>>> 
>>>>> I see that https://repository.jboss.org/nexus/content/groups/public/ 
>>>>> <https://repository.jboss.org/nexus/content/groups/public/>
>>>>> is valid though (http redirects to https), but I don't get why we
>>>>> should delete the line (also the code returns a HTTP 201 afaics so it
>>>>> seems not failing to find it).
>>>>> 
>>>>> Lemme know, I am probably missing something, too many new things :)
>>>>> 
>>>>> Luca
>>>>> 
>>>>> On Fri, Nov 26, 2021 at 6:24 PM Olaf Flebbe >>>> <mailto:o...@oflebbe.de>> wrote:
>>>>>> 
>>>>>> Hi Luca,
>>>>>> 
>>>>>> Seems like jboss.org <http://jboss.org/> <http://jboss.org/ 
>>>>>> <http://jboss.org/>> repository does not exist any more.
>>>>>> 
>>>>>> Try to delete this line:
>>>>>> https://github.com/apache/bigtop/blob/5f863aae467198070c1230558446a19f308a66ae/build.gradle#L462
>>>>>>  
>>>>>> <https://github.com/apache/bigtop/blob/5f863aae467198070c1230558446a19f308a66ae/build.gradle#L462>
>>>>>>  
>>>>>> <https://github.com/apache/bigtop/blob/5f863aae467198070c1230558446a19f308a66ae/build.gradle#L462
>>>>>>  
>>>>>> <https://github.com/apache/bigtop/blob/5f863aae467198070c1230558446a19f308a66ae/build.gradle#L462>>
>>>>>> 
>>>>>> Olaf
>>>>>> 
>>>>>> 
>>>>>>> Am 26.11.2021 um 14:23 schrieb Luca Toscano >>>>>> <mailto:toscano.l...@gmail.com>>:
>>>>>>> 
>>>>>>> Hi Olaf,
>>>>>>> 
>>>>>>> I restarted the jobs and some packages started to build, thanks! I
>>>>>>> found another weird error though:
>>>>>>> 
>>>>>>> """
>>>>>>> * What went wrong:
>>>>>>> Execution failed for task ':configure-nexus-repository.jboss.org 
>>>>>>> <http://configure-

Re: Set up CI for Debian 11 Bullseye

[Olaf Flebbe]

Regarding setup of jenkins, you can find a lot of info here:

https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+CI+Setup+Guide 
<https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+CI+Setup+Guide>

Olaf


> Am 27.11.2021 um 19:46 schrieb Olaf Flebbe :
> 
> I am working with luca to get him access ..
> 
> Olaf
> 
>> Am 27.11.2021 um 18:29 schrieb Luca Toscano :
>> 
>> Keep reporting issues, sorry :)
>> 
>> I see from https://ci.bigtop.apache.org/log/all that we are having
>> disk space issues on the Jenkins node, builds are stopped for the
>> moment. No idea if there is a guide/tutorial for the cleanup, but I
>> guess it means accessing to the VM/hosts that are running Jenkins and
>> I have never done it (not even sure where to put my ssh key etc, what
>> is the hostname, etc..).
>> 
>> Thanks!
>> 
>> Luca
>> 
>> On Fri, Nov 26, 2021 at 7:10 PM Olaf Flebbe  wrote:
>>> 
>>> Hi Luca,
>>> 
>>> you are right, 201 should be valid. However there seems to be some 
>>> compatibility problem with at least this repo. So comment it out for now if 
>>> it resolves the isse. Or remove the whole proxy stuff altoghether. We had 
>>> stability issues with the internet connectivity at that time. The proxy 
>>> resolved that mostly.
>>> 
>>> Olaf
>>> 
>>> 
>>>> Am 26.11.2021 um 19:06 schrieb Luca Toscano :
>>>> 
>>>> Hi Olaf,
>>>> 
>>>> I see that https://repository.jboss.org/nexus/content/groups/public/
>>>> is valid though (http redirects to https), but I don't get why we
>>>> should delete the line (also the code returns a HTTP 201 afaics so it
>>>> seems not failing to find it).
>>>> 
>>>> Lemme know, I am probably missing something, too many new things :)
>>>> 
>>>> Luca
>>>> 
>>>> On Fri, Nov 26, 2021 at 6:24 PM Olaf Flebbe  wrote:
>>>>> 
>>>>> Hi Luca,
>>>>> 
>>>>> Seems like jboss.org <http://jboss.org/> repository does not exist any 
>>>>> more.
>>>>> 
>>>>> Try to delete this line:
>>>>> https://github.com/apache/bigtop/blob/5f863aae467198070c1230558446a19f308a66ae/build.gradle#L462
>>>>>  
>>>>> <https://github.com/apache/bigtop/blob/5f863aae467198070c1230558446a19f308a66ae/build.gradle#L462>
>>>>> 
>>>>> Olaf
>>>>> 
>>>>> 
>>>>>> Am 26.11.2021 um 14:23 schrieb Luca Toscano :
>>>>>> 
>>>>>> Hi Olaf,
>>>>>> 
>>>>>> I restarted the jobs and some packages started to build, thanks! I
>>>>>> found another weird error though:
>>>>>> 
>>>>>> """
>>>>>> * What went wrong:
>>>>>> Execution failed for task ':configure-nexus-repository.jboss.org'.
>>>>>>> Failed to configure Nexus proxy repository.jboss.org with http code 201 
>>>>>>> returned. Run with --info option to see the executed command
>>>>>> """
>>>>>> More info: 
>>>>>> https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/745/COMPONENTS=flink,OS=debian-9/console
>>>>>> 
>>>>>> I am wondering if https://github.com/apache/bigtop/pull/834 could help :)
>>>>>> 
>>>>>> Luca
>>>>>> 
>>>>>> On Fri, Nov 26, 2021 at 11:44 AM Olaf Flebbe  wrote:
>>>>>>> 
>>>>>>> Hi Luca,
>>>>>>> 
>>>>>>> I once added a nexus container as an download cache for maven 
>>>>>>> dependencies, attached to a seperate docker network. It has to be 
>>>>>>> started manually if it failed.
>>>>>>> https://ci.bigtop.apache.org/job/nexus-restart/ 
>>>>>>> <https://ci.bigtop.apache.org/job/nexus-restart/>
>>>>>>> 
>>>>>>> If this network is not present you might get this kind of error
>>>>>>> So try to restart nexus.
>>>>>>> 
>>>>>>> Best
>>>>>>> Olaf
>>>>>>> 
>>>>>>>> Am 26.11.2021 um 09:04 schrieb Luca Toscano :
>>>>>>>> 
>>>>>>>> Update:
>>>>>&g

Re: Set up CI for Debian 11 Bullseye

[Olaf Flebbe]

I am working with luca to get him access ..

Olaf

> Am 27.11.2021 um 18:29 schrieb Luca Toscano :
> 
> Keep reporting issues, sorry :)
> 
> I see from https://ci.bigtop.apache.org/log/all that we are having
> disk space issues on the Jenkins node, builds are stopped for the
> moment. No idea if there is a guide/tutorial for the cleanup, but I
> guess it means accessing to the VM/hosts that are running Jenkins and
> I have never done it (not even sure where to put my ssh key etc, what
> is the hostname, etc..).
> 
> Thanks!
> 
> Luca
> 
> On Fri, Nov 26, 2021 at 7:10 PM Olaf Flebbe  wrote:
>> 
>> Hi Luca,
>> 
>> you are right, 201 should be valid. However there seems to be some 
>> compatibility problem with at least this repo. So comment it out for now if 
>> it resolves the isse. Or remove the whole proxy stuff altoghether. We had 
>> stability issues with the internet connectivity at that time. The proxy 
>> resolved that mostly.
>> 
>> Olaf
>> 
>> 
>>> Am 26.11.2021 um 19:06 schrieb Luca Toscano :
>>> 
>>> Hi Olaf,
>>> 
>>> I see that https://repository.jboss.org/nexus/content/groups/public/
>>> is valid though (http redirects to https), but I don't get why we
>>> should delete the line (also the code returns a HTTP 201 afaics so it
>>> seems not failing to find it).
>>> 
>>> Lemme know, I am probably missing something, too many new things :)
>>> 
>>> Luca
>>> 
>>> On Fri, Nov 26, 2021 at 6:24 PM Olaf Flebbe  wrote:
>>>> 
>>>> Hi Luca,
>>>> 
>>>> Seems like jboss.org <http://jboss.org/> repository does not exist any 
>>>> more.
>>>> 
>>>> Try to delete this line:
>>>> https://github.com/apache/bigtop/blob/5f863aae467198070c1230558446a19f308a66ae/build.gradle#L462
>>>>  
>>>> <https://github.com/apache/bigtop/blob/5f863aae467198070c1230558446a19f308a66ae/build.gradle#L462>
>>>> 
>>>> Olaf
>>>> 
>>>> 
>>>>> Am 26.11.2021 um 14:23 schrieb Luca Toscano :
>>>>> 
>>>>> Hi Olaf,
>>>>> 
>>>>> I restarted the jobs and some packages started to build, thanks! I
>>>>> found another weird error though:
>>>>> 
>>>>> """
>>>>> * What went wrong:
>>>>> Execution failed for task ':configure-nexus-repository.jboss.org'.
>>>>>> Failed to configure Nexus proxy repository.jboss.org with http code 201 
>>>>>> returned. Run with --info option to see the executed command
>>>>> """
>>>>> More info: 
>>>>> https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/745/COMPONENTS=flink,OS=debian-9/console
>>>>> 
>>>>> I am wondering if https://github.com/apache/bigtop/pull/834 could help :)
>>>>> 
>>>>> Luca
>>>>> 
>>>>> On Fri, Nov 26, 2021 at 11:44 AM Olaf Flebbe  wrote:
>>>>>> 
>>>>>> Hi Luca,
>>>>>> 
>>>>>> I once added a nexus container as an download cache for maven 
>>>>>> dependencies, attached to a seperate docker network. It has to be 
>>>>>> started manually if it failed.
>>>>>> https://ci.bigtop.apache.org/job/nexus-restart/ 
>>>>>> <https://ci.bigtop.apache.org/job/nexus-restart/>
>>>>>> 
>>>>>> If this network is not present you might get this kind of error
>>>>>> So try to restart nexus.
>>>>>> 
>>>>>> Best
>>>>>> Olaf
>>>>>> 
>>>>>>> Am 26.11.2021 um 09:04 schrieb Luca Toscano :
>>>>>>> 
>>>>>>> Update:
>>>>>>> 
>>>>>>> - Found the config diff changes in Jenkins, so I know how to re-enable
>>>>>>> periodic execution.
>>>>>>> 
>>>>>>> - Tried to kick off a build, but I stopped it since all the package
>>>>>>> builds were failing for the same error:
>>>>>>> ++ docker run -d --net=container:nexus bigtop/slaves:trunk-debian-10 
>>>>>>> /sbin/init
>>>>>>> docker: Error response from daemon: cannot join network of a non
>>>>>>> running container
>>>>>>> Example:http

Re: Set up CI for Debian 11 Bullseye

[Olaf Flebbe]

Hi Luca,

you are right, 201 should be valid. However there seems to be some 
compatibility problem with at least this repo. So comment it out for now if it 
resolves the isse. Or remove the whole proxy stuff altoghether. We had 
stability issues with the internet connectivity at that time. The proxy 
resolved that mostly.

Olaf


> Am 26.11.2021 um 19:06 schrieb Luca Toscano :
> 
> Hi Olaf,
> 
> I see that https://repository.jboss.org/nexus/content/groups/public/
> is valid though (http redirects to https), but I don't get why we
> should delete the line (also the code returns a HTTP 201 afaics so it
> seems not failing to find it).
> 
> Lemme know, I am probably missing something, too many new things :)
> 
> Luca
> 
> On Fri, Nov 26, 2021 at 6:24 PM Olaf Flebbe  wrote:
>> 
>> Hi Luca,
>> 
>> Seems like jboss.org <http://jboss.org/> repository does not exist any more.
>> 
>> Try to delete this line:
>> https://github.com/apache/bigtop/blob/5f863aae467198070c1230558446a19f308a66ae/build.gradle#L462
>>  
>> <https://github.com/apache/bigtop/blob/5f863aae467198070c1230558446a19f308a66ae/build.gradle#L462>
>> 
>> Olaf
>> 
>> 
>>> Am 26.11.2021 um 14:23 schrieb Luca Toscano :
>>> 
>>> Hi Olaf,
>>> 
>>> I restarted the jobs and some packages started to build, thanks! I
>>> found another weird error though:
>>> 
>>> """
>>> * What went wrong:
>>> Execution failed for task ':configure-nexus-repository.jboss.org'.
>>>> Failed to configure Nexus proxy repository.jboss.org with http code 201 
>>>> returned. Run with --info option to see the executed command
>>> """
>>> More info: 
>>> https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/745/COMPONENTS=flink,OS=debian-9/console
>>> 
>>> I am wondering if https://github.com/apache/bigtop/pull/834 could help :)
>>> 
>>> Luca
>>> 
>>> On Fri, Nov 26, 2021 at 11:44 AM Olaf Flebbe  wrote:
>>>> 
>>>> Hi Luca,
>>>> 
>>>> I once added a nexus container as an download cache for maven 
>>>> dependencies, attached to a seperate docker network. It has to be started 
>>>> manually if it failed.
>>>> https://ci.bigtop.apache.org/job/nexus-restart/ 
>>>> <https://ci.bigtop.apache.org/job/nexus-restart/>
>>>> 
>>>> If this network is not present you might get this kind of error
>>>> So try to restart nexus.
>>>> 
>>>> Best
>>>> Olaf
>>>> 
>>>>> Am 26.11.2021 um 09:04 schrieb Luca Toscano :
>>>>> 
>>>>> Update:
>>>>> 
>>>>> - Found the config diff changes in Jenkins, so I know how to re-enable
>>>>> periodic execution.
>>>>> 
>>>>> - Tried to kick off a build, but I stopped it since all the package
>>>>> builds were failing for the same error:
>>>>> ++ docker run -d --net=container:nexus bigtop/slaves:trunk-debian-10 
>>>>> /sbin/init
>>>>> docker: Error response from daemon: cannot join network of a non
>>>>> running container
>>>>> Example:https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/COMPONENTS=solr,OS=debian-10/lastBuild/console
>>>>> 
>>>>> I have probably missed something, I don't exactly understand the error
>>>>> so any help would be appreciated :)
>>>>> 
>>>>> Luca
>>>>> 
>>>>> On Thu, Nov 25, 2021 at 12:27 PM Luca Toscano  
>>>>> wrote:
>>>>>> 
>>>>>> Update: I added debian-11 support, if nobody disagrees I'd kick off a
>>>>>> package build. I can also enable periodic execution but I am not sure
>>>>>> what I should put in the related time interval field (I didn't find it
>>>>>> in the docs).
>>>>>> 
>>>>>> Luca
>>>>>> 
>>>>>> On Sun, Nov 21, 2021 at 7:10 PM Luca Toscano  
>>>>>> wrote:
>>>>>>> 
>>>>>>> Hi Olaf! Thanks a lot :)
>>>>>>> 
>>>>>>> I managed to change CI to allow Debian 11 (minus a little error for
>>>>>>> Ubuntu that seems unrelated, see
>>>>>>> https://ci.bigtop.apache.org/view/Docker/job/Docker-Toolchain-Trunk/)
>>>>>>> and I was able to build the hadoop packages locally via the new Debian
>>>>>>> 11 images published to Dockerhub.
>>>>>>> 
>>>>>>> IIUC the last step is to modify
>>>>>>> https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/
>>>>>>> and kick off a build, but I see "2021-10-02: temporarily disable
>>>>>>> periodic execution for preparing 3.0.0 release." so before doing
>>>>>>> anything I'd like some feedback (to avoid causing problems etc..).
>>>>>>> 
>>>>>>> Luca
>>>>>>> 
>>>>>>> On Fri, Nov 19, 2021 at 8:53 PM Olaf Flebbe  wrote:
>>>>>>>> 
>>>>>>>> Hi Luca,
>>>>>>>> 
>>>>>>>> I gave you super powers.
>>>>>>>> 
>>>>>>>> Best
>>>>>>>> Olaf
>>>>>>>> 
>>>>>>>>> Am 19.11.2021 um 08:15 schrieb Luca Toscano :
>>>>>>>>> 
>>>>>>>>> Hi everybody!
>>>>>>>>> 
>>>>>>>>> I am working on https://issues.apache.org/jira/browse/BIGTOP-3600 and
>>>>>>>>> IIUC I'd need to add the CI jobs now, but of course I don't have
>>>>>>>>> permissions. If anybody has some time to help I'd be happy to learn
>>>>>>>>> the process and help out :)
>>>>>>>>> 
>>>>>>>>> Thanks!
>>>>>>>>> 
>>>>>>>>> Luca
>>>>>>>> 
>>>> 
>> 

Re: Set up CI for Debian 11 Bullseye

[Olaf Flebbe]

Hi Luca,

Seems like jboss.org <http://jboss.org/> repository does not exist any more.

Try to delete this line:
https://github.com/apache/bigtop/blob/5f863aae467198070c1230558446a19f308a66ae/build.gradle#L462
 
<https://github.com/apache/bigtop/blob/5f863aae467198070c1230558446a19f308a66ae/build.gradle#L462>

Olaf


> Am 26.11.2021 um 14:23 schrieb Luca Toscano :
> 
> Hi Olaf,
> 
> I restarted the jobs and some packages started to build, thanks! I
> found another weird error though:
> 
> """
> * What went wrong:
> Execution failed for task ':configure-nexus-repository.jboss.org'.
>> Failed to configure Nexus proxy repository.jboss.org with http code 201 
>> returned. Run with --info option to see the executed command
> """
> More info: 
> https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/745/COMPONENTS=flink,OS=debian-9/console
> 
> I am wondering if https://github.com/apache/bigtop/pull/834 could help :)
> 
> Luca
> 
> On Fri, Nov 26, 2021 at 11:44 AM Olaf Flebbe  wrote:
>> 
>> Hi Luca,
>> 
>> I once added a nexus container as an download cache for maven dependencies, 
>> attached to a seperate docker network. It has to be started manually if it 
>> failed.
>> https://ci.bigtop.apache.org/job/nexus-restart/ 
>> <https://ci.bigtop.apache.org/job/nexus-restart/>
>> 
>> If this network is not present you might get this kind of error
>> So try to restart nexus.
>> 
>> Best
>> Olaf
>> 
>>> Am 26.11.2021 um 09:04 schrieb Luca Toscano :
>>> 
>>> Update:
>>> 
>>> - Found the config diff changes in Jenkins, so I know how to re-enable
>>> periodic execution.
>>> 
>>> - Tried to kick off a build, but I stopped it since all the package
>>> builds were failing for the same error:
>>> ++ docker run -d --net=container:nexus bigtop/slaves:trunk-debian-10 
>>> /sbin/init
>>> docker: Error response from daemon: cannot join network of a non
>>> running container
>>> Example:https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/COMPONENTS=solr,OS=debian-10/lastBuild/console
>>> 
>>> I have probably missed something, I don't exactly understand the error
>>> so any help would be appreciated :)
>>> 
>>> Luca
>>> 
>>> On Thu, Nov 25, 2021 at 12:27 PM Luca Toscano  
>>> wrote:
>>>> 
>>>> Update: I added debian-11 support, if nobody disagrees I'd kick off a
>>>> package build. I can also enable periodic execution but I am not sure
>>>> what I should put in the related time interval field (I didn't find it
>>>> in the docs).
>>>> 
>>>> Luca
>>>> 
>>>> On Sun, Nov 21, 2021 at 7:10 PM Luca Toscano  
>>>> wrote:
>>>>> 
>>>>> Hi Olaf! Thanks a lot :)
>>>>> 
>>>>> I managed to change CI to allow Debian 11 (minus a little error for
>>>>> Ubuntu that seems unrelated, see
>>>>> https://ci.bigtop.apache.org/view/Docker/job/Docker-Toolchain-Trunk/)
>>>>> and I was able to build the hadoop packages locally via the new Debian
>>>>> 11 images published to Dockerhub.
>>>>> 
>>>>> IIUC the last step is to modify
>>>>> https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/
>>>>> and kick off a build, but I see "2021-10-02: temporarily disable
>>>>> periodic execution for preparing 3.0.0 release." so before doing
>>>>> anything I'd like some feedback (to avoid causing problems etc..).
>>>>> 
>>>>> Luca
>>>>> 
>>>>> On Fri, Nov 19, 2021 at 8:53 PM Olaf Flebbe  wrote:
>>>>>> 
>>>>>> Hi Luca,
>>>>>> 
>>>>>> I gave you super powers.
>>>>>> 
>>>>>> Best
>>>>>> Olaf
>>>>>> 
>>>>>>> Am 19.11.2021 um 08:15 schrieb Luca Toscano :
>>>>>>> 
>>>>>>> Hi everybody!
>>>>>>> 
>>>>>>> I am working on https://issues.apache.org/jira/browse/BIGTOP-3600 and
>>>>>>> IIUC I'd need to add the CI jobs now, but of course I don't have
>>>>>>> permissions. If anybody has some time to help I'd be happy to learn
>>>>>>> the process and help out :)
>>>>>>> 
>>>>>>> Thanks!
>>>>>>> 
>>>>>>> Luca
>>>>>> 
>> 

Re: Set up CI for Debian 11 Bullseye

[Olaf Flebbe]

Hi Luca,

I once added a nexus container as an download cache for maven dependencies, 
attached to a seperate docker network. It has to be started manually if it 
failed.
https://ci.bigtop.apache.org/job/nexus-restart/ 
<https://ci.bigtop.apache.org/job/nexus-restart/>

If this network is not present you might get this kind of error
So try to restart nexus.

Best
Olaf

> Am 26.11.2021 um 09:04 schrieb Luca Toscano :
> 
> Update:
> 
> - Found the config diff changes in Jenkins, so I know how to re-enable
> periodic execution.
> 
> - Tried to kick off a build, but I stopped it since all the package
> builds were failing for the same error:
> ++ docker run -d --net=container:nexus bigtop/slaves:trunk-debian-10 
> /sbin/init
> docker: Error response from daemon: cannot join network of a non
> running container
> Example:https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/COMPONENTS=solr,OS=debian-10/lastBuild/console
> 
> I have probably missed something, I don't exactly understand the error
> so any help would be appreciated :)
> 
> Luca
> 
> On Thu, Nov 25, 2021 at 12:27 PM Luca Toscano  wrote:
>> 
>> Update: I added debian-11 support, if nobody disagrees I'd kick off a
>> package build. I can also enable periodic execution but I am not sure
>> what I should put in the related time interval field (I didn't find it
>> in the docs).
>> 
>> Luca
>> 
>> On Sun, Nov 21, 2021 at 7:10 PM Luca Toscano  wrote:
>>> 
>>> Hi Olaf! Thanks a lot :)
>>> 
>>> I managed to change CI to allow Debian 11 (minus a little error for
>>> Ubuntu that seems unrelated, see
>>> https://ci.bigtop.apache.org/view/Docker/job/Docker-Toolchain-Trunk/)
>>> and I was able to build the hadoop packages locally via the new Debian
>>> 11 images published to Dockerhub.
>>> 
>>> IIUC the last step is to modify
>>> https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/
>>> and kick off a build, but I see "2021-10-02: temporarily disable
>>> periodic execution for preparing 3.0.0 release." so before doing
>>> anything I'd like some feedback (to avoid causing problems etc..).
>>> 
>>> Luca
>>> 
>>> On Fri, Nov 19, 2021 at 8:53 PM Olaf Flebbe  wrote:
>>>> 
>>>> Hi Luca,
>>>> 
>>>> I gave you super powers.
>>>> 
>>>> Best
>>>> Olaf
>>>> 
>>>>> Am 19.11.2021 um 08:15 schrieb Luca Toscano :
>>>>> 
>>>>> Hi everybody!
>>>>> 
>>>>> I am working on https://issues.apache.org/jira/browse/BIGTOP-3600 and
>>>>> IIUC I'd need to add the CI jobs now, but of course I don't have
>>>>> permissions. If anybody has some time to help I'd be happy to learn
>>>>> the process and help out :)
>>>>> 
>>>>> Thanks!
>>>>> 
>>>>> Luca
>>>> 

Re: Set up CI for Debian 11 Bullseye

[Olaf Flebbe]

Hi Luca,

I gave you super powers.

Best
Olaf

> Am 19.11.2021 um 08:15 schrieb Luca Toscano :
> 
> Hi everybody!
> 
> I am working on https://issues.apache.org/jira/browse/BIGTOP-3600 and
> IIUC I'd need to add the CI jobs now, but of course I don't have
> permissions. If anybody has some time to help I'd be happy to learn
> the process and help out :)
> 
> Thanks!
> 
> Luca

OpenPOWER Cluster

[Olaf Flebbe]

Hi Amir and others involved!

I received an email from the OSL OpenPOWER Computing Cluster Admins concerning 
a survey and possible shutdown of the service of not responding to this survey.

Background:
IBM gave some big OpenPOWER machines to the OSL to be hosted exclusively for 
development of OSS projects.

I am not sure if I got this mail because I opted in for Bigtop for this 
service. 

Other Background:
Amir, at that time in the past with IBM, provided logins to connect our CI to 
some ppc64le machines and was able to resurrect it when it broke lately.

Amir, are the ppc64le accounts from IBM we are using in any way connected to 
this service ?
https://wiki.osuosl.org/openpower/openstack.html# 


If yes I am happy to fill in the survey or forwarding to anyone within this 
community .

Amir, are you able to clarify?

Best,
Olaf

Re: Starting 3.0.0 release process

[Olaf Flebbe]

Hi,

Sounds really great to have a plan for Bigtop 3.0.

Regarding Debian 11: I would not recommend waiting until the final Debian 
release since I would not expect large shifts in focus with respect to current 
„testing“ any more. Usually the fine tuning of all the packages takes long 
time, but the software compatibilty layer we need is being fixed long before.

I am recommending to use the Debian testing images/repository for Bigtop 3 
developing now. That being said, it would be even possible to raise problems 
now, if we hit something serious for some reason.

Does this make sense?

Best 
   Olaf 


> Am 11.05.2021 um 16:32 schrieb Evans Ye :
> 
> I'm +1 to Kengo's plan to release 3.0 w/o waiting.
> We can do minor release with required changes later on. Release early,
> release often is the key here.
> I believe when Debian 11 is out, Luca along with the community is happy to
> work together for a release to support the usage at Wiki.
> Of course, if our release is delayed and Debian 11 is out, we can
> reprioritize then.
> 
> Arnaud Launay  於 2021年5月11日 週二 下午6:14寫道：
> 
>> Hello,
>> 
>> Le Tue, May 11, 2021 at 09:02:57AM +0900, Kengo Seki a écrit:
>>> Is it OK with you, or would you like to include Debian 11 support in
>> this release?
>> 
>> That depends on the timeframe for bigtop 3, I think. I suspect Debian 11
>> will be
>> release in about 2 months. If bigtop3 is to be released this summer or in
>> september,
>> including it would be a good thing. If B3 is to be released next week, no
>> :)
>> 
>>Arnaud.
>> 

Re: PPC CI server failure

[Olaf Flebbe]

I already gave the public key to asanjar. 

Olaf

> Am 16.04.2021 um 10:49 schrieb Evans Ye :
> 
> Let me help. I was busy on a thing.
> 
> 
> MrAsanjar .  於 2021年4月15日 週四 下午10:30寫道：
> 
>> In order to set up the new Jenkins slave for ppc64le (
>> https://issues.apache.org/jira/browse/BIGTOP-3534) we need Jenkins
>> master's
>> public ssh key. Who can help me here?
>> 
>> On Fri, Apr 2, 2021 at 4:00 PM MrAsanjar  wrote:
>> 
>>> I have verified the state of ppc64le VM, it is operational. Could we
>>> enable the ppc64le build before OpenStack flag the VM as ideal again.
>>> 
>>> On Thu, Apr 1, 2021 at 4:08 PM MrAsanjar  wrote:
>>> 
 Hi lads
 I just got an email that IBM has reinstated the ppc64le VM.
 
 
 On Mon, Mar 29, 2021 at 12:05 PM Evans Ye  wrote:
 
> Great news and thanks, Amir!
> 
> Jun HE  於 2021年3月29日 週一 下午1:54寫道：
> 
>> Awesome! Looking forward to its back to CI.
>> Thanks a lot for helping on this, Asanjar!
>> 
>> Regards,
>> 
>> Jun
>> 
>> MrAsanjar  于2021年3月29日周一 上午10:18写道：
>> 
>>> Hi old friends :)
>>> We should have a ppc64le VM back online sometime this week. I'll
> keep you
>>> all posted.
>>> 
>>> On Thu, Nov 19, 2020 at 9:05 PM Evans Ye 
>> wrote:
>>> 
 Hi rbkrishn,
 
 Would you mind to comment whether those PPC servers for Bigtop CI
> can
>> be
 brought up and unlock our release process?
 Thanks!
 
 Best,
 Evans
 
 Kengo Seki  於 2020年11月18日 週三 上午7:26寫道：
 
> Thank you for checking, Evans and Amir!
> 
> Kengo Seki 
> 
> On Wed, Nov 18, 2020 at 2:09 AM Evans Ye 
> wrote:
>> 
>> Thank you, Amir.
>> 
>> MrAsanjar  於 2020年11月18日 週三 00:39 寫道：
>> 
>>> Hi Evans, let me check with IBM again.
>>> 
>>> 
>>> On Mon, Nov 16, 2020 at 9:08 PM Evans Ye <
>> evan...@apache.org
>> 
>>> wrote:
>>> 
 Hi Amir,
 
 We're planning Bigtop 1.5 release and if we don't have
>> the
> CI
>>> nodes
> for
 PPC, we're not able to release 1.5 with PPC supported.
 Could you help to confirm again? Thanks!
 
 Best,
 Evans Ye
 
 
 
 MrAsanjar  於 2020年9月17日 週四 下午8:56寫道：
 
> I have informed IBM management regarding the situation,
>> waiting
> for a
> reply.
> 
> On Thu, Sep 17, 2020 at 3:47 AM Evans Ye <
> evan...@apache.org
>>> 
> wrote:
> 
>> Ok. Thanks for doing this to get the ball rolling.
>> 
>> Kengo Seki  於 2020年9月17日 週四 10:29
> 寫道：
>> 
>>> Thank you for your help, Amir!
>>> It's just a heads-up, I temporarily disabled builds
> for
>> ppc
 in
> the
>>> following Jenkins jobs so that they can finish.
>>> 
>>> * Docker-Puppet-Trunk
>>> * Docker-Puppet-Trunk-pull
>>> * Docker-Toolchain-Trunk
>>> * Docker-Toolchain-Trunk-pull
>>> 
>>> * Bigtop-trunk-packages
>>> * Bigtop-trunk-repos
>>> 
>>> * Remove-All-Docker-Containers-Except-Nexus
>>> * Remove-Dangling-Docker-Images
>>> * Remove-Inactive-Containers
>>> 
>>> Kengo Seki 
>>> 
>>> On Wed, Sep 16, 2020 at 7:35 PM Evans Ye <
>>> evan...@apache.org
> 
>>> wrote:
 
 Awesome! Nice to hear from you, buddy!
 
 MrAsanjar  於 2020年9月16日 週三
>> 上午3:54寫道：
 
> Hi Evans,
> Let me see what I can do. Give me 24 hr :)
> 
> On Tue, Sep 15, 2020 at 10:51 AM Evans Ye <
> evan...@apache.org>
>> wrote:
> 
>> Yes. I think the action is correct. However
>> [2]
>> might
 be
> a
>> different
> thing
>> for PPC integration in Hadoop.
>> 
>> Amir,
>> Could you confirm?
>> 
>> Kengo Seki  於 2020年9月14日
>> 週一
 下午9:56寫道：
>> 
>>> Thank you for the advice, Evans!
>>> Let me confirm about "PPC machine owners".
>> According
 to
>>> Amir's
>> JIRA
>>> issues [1][2] and the powered-by list in the
> OSU
>>> site
> [3],
 we're
>>> using
>>> a VM hosted by OSU OSL, right?
>>> If it's correct, I'm going to a

Re: [External Sender] Failed to build hadoop in our Bamboo agent

[Olaf Flebbe]

> Am 24.03.2021 um 19:42 schrieb Jason Wen  >:
> 
> Thanks Masatake for sharing this.
> Do you know the possible reasons in the case of write failure besides 'No 
> space left on device'?
> 
> Regards,
> Jason

There might be quotas or resouce limits in place.
Olaf

Re: Website update for release 1.5

[Olaf Flebbe]

Hi 

Issue solved:
https://issues.apache.org/jira/browse/INFRA-21413?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel
 
<https://issues.apache.org/jira/browse/INFRA-21413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel>

Best
Olaf

> Am 10.02.2021 um 20:27 schrieb Olaf Flebbe :
> 
> Hi,
> 
> This is very odd, indeed.
> 
> The jenkins job ran on December 15th to publish a new site
> https://builds.apache.org/job/BigTop/job/pipeline-site/71/ 
> <https://builds.apache.org/job/BigTop/job/pipeline-site/71/>
> 
> The log in asf-site branch shows this timing as well
> https://gitbox.apache.org/repos/asf?p=bigtop.git;a=shortlog;h=refs/heads/asf-site
>  
> <https://gitbox.apache.org/repos/asf?p=bigtop.git;a=shortlog;h=refs/heads/asf-site>
> 
> However something has reverted back the site.
> 
> We need to ask INFRA what is going on.
> https://issues.apache.org/jira/browse/INFRA-21413
> 
> Best,
> Olaf
> 
> 
> 
>> Am 10.02.2021 um 10:20 schrieb Konstantin Boudnik :
>> 
>> And there's a ticket for this BIGTOP-3496
>> 
>> --
>> With regards,
>> Cos
>> 
>> On 10.02.2021 12:17, Konstantin Boudnik wrote:
>>> In the old times when the website was living in the SVN, there was an 
>>> additional step to publish the changes to the public stage.
>>> However, I am looking at the site right now and I see this time stamps:
>>> Last Published: 2020-04-18
>>> Which explains 1.4.0
>>> I wonder if this is somehow connected to the recent switch to git-based 
>>> publishing?
>>> -- 
>>> With regards,
>>>  Cos
>>> On 10.02.2021 11:12, Kengo Seki wrote:
>>>> Thank you for the notification, Cos!
>>>> Hmm, that sounds strange. I made sure the website was correctly
>>>> updated when releasing 1.5.0. I took screenshots and pasted them into
>>>> my blog which introduces 1.5.0 in Japanese.
>>>> https://sekikn.github.io/blog/2020/12/19/introducing-bigtop-1.5.0-1.html
>>>> https://sekikn.github.io/blog/2020/12/20/introducing-bigtop-1.5.0-2.html
>>>> 
>>>> I'm going to investigate its cause this weekend at the latest.
>>>> 
>>>> Kengo Seki 
>>>> 
>>>> On Tue, Feb 9, 2021 at 3:11 PM Konstantin Boudnik  wrote:
>>>>> 
>>>>> Fellas,
>>>>> 
>>>>> just noticed that our website wasn't updated as a part of 1.5 release. I
>>>>> presume the release manager did miss a step, perhaps? ;)
>>>>> 
>>>>> Please let me know if any help is needed!
>>>>> 
>>>>> -- 
>>>>> With regards,
>>>>>Cos
> 

Re: Website update for release 1.5

[Olaf Flebbe]

Hi,

This is very odd, indeed.

The jenkins job ran on December 15th to publish a new site
https://builds.apache.org/job/BigTop/job/pipeline-site/71/ 


The log in asf-site branch shows this timing as well
https://gitbox.apache.org/repos/asf?p=bigtop.git;a=shortlog;h=refs/heads/asf-site
 


However something has reverted back the site.

We need to ask INFRA what is going on.
https://issues.apache.org/jira/browse/INFRA-21413

Best,
 Olaf



> Am 10.02.2021 um 10:20 schrieb Konstantin Boudnik :
> 
> And there's a ticket for this BIGTOP-3496
> 
> --
> With regards,
>  Cos
> 
> On 10.02.2021 12:17, Konstantin Boudnik wrote:
>> In the old times when the website was living in the SVN, there was an 
>> additional step to publish the changes to the public stage.
>> However, I am looking at the site right now and I see this time stamps:
>> Last Published: 2020-04-18
>> Which explains 1.4.0
>> I wonder if this is somehow connected to the recent switch to git-based 
>> publishing?
>> -- 
>> With regards,
>>   Cos
>> On 10.02.2021 11:12, Kengo Seki wrote:
>>> Thank you for the notification, Cos!
>>> Hmm, that sounds strange. I made sure the website was correctly
>>> updated when releasing 1.5.0. I took screenshots and pasted them into
>>> my blog which introduces 1.5.0 in Japanese.
>>> https://sekikn.github.io/blog/2020/12/19/introducing-bigtop-1.5.0-1.html
>>> https://sekikn.github.io/blog/2020/12/20/introducing-bigtop-1.5.0-2.html
>>> 
>>> I'm going to investigate its cause this weekend at the latest.
>>> 
>>> Kengo Seki 
>>> 
>>> On Tue, Feb 9, 2021 at 3:11 PM Konstantin Boudnik  wrote:
 
 Fellas,
 
 just noticed that our website wasn't updated as a part of 1.5 release. I
 presume the release manager did miss a step, perhaps? ;)
 
 Please let me know if any help is needed!
 
 -- 
 With regards,
 Cos

FOSDEM

[Olaf Flebbe]

Hi,

See what has been accepted at FOSDEM’s Big data devroom.

https://fosdem.org/2021/schedule/event/big_data_arm/ 



Best 
   Olaf

Re: Bigtop Site: Input needed

[Olaf Flebbe]

 personally prefer the second option (docsy) because of its versioned
> > documentation support, but also think Evans' opinion is reasonable so
> > either is OK to me.
> >
> > > All the 3rdparty services used and advertised (twitter, stackoverflow, 
> > > github, google analytics, google search) are hardcoded in the design. 
> > > Either override almost all these layout templates or actually play on 
> > > these channels as well?
> >
> > Just curiosity, isn't commenting out params.links.* in config.toml
> > enough to hide the social network links?
> > I tried the docsy example following the steps described in [1], and
> > the links which are commented out in config.toml (e.g., Stackoverflow
> > and Slack) seemed to be disabled in the footer.
> >
> > [1]: https://themes.gohugo.io/docsy/#documentation 
> > <https://themes.gohugo.io/docsy/#documentation>
> >
> > Kengo Seki mailto:sek...@apache.org>>
> >
> > On Wed, Dec 30, 2020 at 12:56 PM Evans Ye  > <mailto:evan...@apache.org>> wrote:
> > >
> > > My two cents:
> > >
> > > First of all, thank you for putting in the effort on the bigtop site. The 
> > > example looks super great!
> > >
> > > I'd vote for Ananke for the following reasons:
> > > 1. Seems like this solution strikes the balance between the maintenance 
> > > effort and the result we want.
> > > 2. The docs are all on confluent now so it's as good as it is if we 
> > > decide to stay on confluent.
> > > 3. Even though we just have the site updated, I think it's a huge plus 
> > > for our branding/imaging. And somewhat telling users that we're keep 
> > > evolving and up-to-date.
> > > 4. "Docsy" or "Exploring different options" are too much effort for now 
> > > and future that we can't carry down the road.
> > >
> > > - Evans
> > >
> > >
> > > Olaf Flebbe mailto:o...@oflebbe.de>> 於 2020年12月29日 週二 
> > > 下午8:06寫道：
> > >>
> > >> Hi,
> > >>
> > >> As previously discussed I had a look on making the website authoring 
> > >> more accessible.
> > >>
> > >> I totally underestimated it, my head hurts now (see below) :)
> > >>
> > >> And I am now at crossroads, need your input:
> > >>
> > >> First I decided to look at static site generaters, these are candidates:
> > >> * Hugo : gohugo.io <http://gohugo.io/>
> > >> * Zola: https://www.getzola.org <https://www.getzola.org/>
> > >> (* Ninja)
> > >>
> > >> I looked at various other Project’s design-wise:
> > >>
> > >> I liked arrow.apache.org <http://arrow.apache.org/> most, it’s clean. It 
> > >> looks like built with ninja and a custom layout
> > >> Hadoop.apache.org <http://hadoop.apache.org/> is built with hugo with a 
> > >> custom layout (not adaptable for us), looking not-so-good.
> > >> https://kubernetes.io <https://kubernetes.io/>  and many lot more are 
> > >> built with hugo and https://www.docsy.dev/ <https://www.docsy.dev/>  
> > >> theme looking very professional.
> > >>
> > >> I didn’t want to use ninja for some reason and decided to look into hugo:
> > >>
> > >> * A plus for hugo was support for asciidoctor files out of the box, as 
> > >> Cos was suggested.
> > >>
> > >> * big community repository of ready-made themese. But none if them was a 
> > >> 100% fit, so I did example bigtop landing pages:
> > >>
> > >> Examples
> > >> 
> > >>
> > >> Hugo / Ananke theme:
> > >> =
> > >>
> > >> Example:  http://bigtop-ananke.oflebbe.de 
> > >> <http://bigtop-ananke.oflebbe.de/>
> > >> Source:  github.com/oflebbe/bigtop-site 
> > >> <http://github.com/oflebbe/bigtop-site>
> > >> (Install asciidoctor as an additional prerequisite)
> > >>
> > >> In order to understand Hugo I followed the quickstart and the 
> > >> recommended ananke theme:
> > >> Cons:
> > >> * I had to bang my head for two days against the wall to implement a 
> > >> pulldown navigation menu. (The ASF Menu)
> > >> * Had to fight sizes and image scaling
> > >> * 

Bigtop Site: Input needed

[Olaf Flebbe]

Hi,

As previously discussed I had a look on making the website authoring more 
accessible. 

I totally underestimated it, my head hurts now (see below) :) 

And I am now at crossroads, need your input:

First I decided to look at static site generaters, these are candidates:
* Hugo : gohugo.io 
* Zola: https://www.getzola.org 
(* Ninja)

I looked at various other Project’s design-wise:

I liked arrow.apache.org  most, it’s clean. It looks 
like built with ninja and a custom layout
Hadoop.apache.org  is built with hugo with a custom 
layout (not adaptable for us), looking not-so-good.
https://kubernetes.io   and many lot more are built 
with hugo and https://www.docsy.dev/   theme looking 
very professional.

I didn’t want to use ninja for some reason and decided to look into hugo:

* A plus for hugo was support for asciidoctor files out of the box, as Cos was 
suggested.

* big community repository of ready-made themese. But none if them was a 100% 
fit, so I did example bigtop landing pages:

Examples


Hugo / Ananke theme: 
=

Example:  http://bigtop-ananke.oflebbe.de 
Source:  github.com/oflebbe/bigtop-site 
(Install asciidoctor as an additional prerequisite)

In order to understand Hugo I followed the quickstart and the recommended 
ananke theme:
Cons:
* I had to bang my head for two days against the wall to implement a pulldown 
navigation menu. (The ASF Menu) 
* Had to fight sizes and image scaling 
* No support for docs, if we would move docs from confluence with versioning 
support to the site.
(i.e. seperate bigtop-1.4 from bigtop-1.3 as an example), 
* Allmost no docs.

Pro:
* Looks ok-ish
* Clean and easy to understand.
* Design is Blog-centric

Hugo / docsy theme:
===
Example: http://bigtop-docsy.oflebbe.de 
Source:  github.com/oflebbe/bigtop-site-docsy 


There are a lot of Linux Foundation/Cloud projects built on this theme, but 
that’s a pity as well: All the 3rdparty services used and advertised (twitter, 
stackoverflow, github, google analytics, google search ) are hardcoded in the 
design. Either override almost all these layout templates or actually play on 
these channels as well ?

Cons:
* Harder to use
* Banging my head against the wall to get PostCSS running correctly.
* Banging my head for a day to get the "The ASF“ dropdown menu working (still 
doesn’t look right)
* Fight the social network links and so on in the templates
* Sometimes it seems to trigger bugs in hugo 

Pro:
* Documentation templating seems to work, support for versioned docs
* Mature Design system

 
General Observations
==

It seems like dropdown navigation menus are not a good idea: 
* From accessibilty perspective it is complex to support for vision impaired 
persons
* On mobile on docsy theme they aren’t rendered at all (no idea why)
* Changing that we need to provide more content (hadn’t thought about that 
previously)

We need to decide what additional social media channels we would like to use 
(twitter etc ..)

Need a perspective of how we do documentation: Stay on confluence or move to 
Markdown, into the site.

We offer different Downloads in „Downloads“ and „Releases“. It feels weird. And 
the Instructions how to use are buried deep in confluence and in the Readme.md 
(Readme is not linked from landing page)

=> Your input needed: <===

What do you think about what will fit a future landing page best, what will 
your work/our audience supporting most?

* Everything to complex: 
  We should stay on current tooling and focus on content , because of limited 
resources.
* Docsy:
  Start with a landing page and move over stuff later
* Ananke:
  Only offer a landing page and head over to confluence with everything else
* Exploring different options
  (suggestions, if possible)

Sadly; I am note able to finish this, I had and still a few days off  now the 
clock is ticking. 

What I learned from it (besides tons of CSS/HTML fine points): We should 
improve current content first, work on tooling second. 

What conclusion would you draw ?

Best,
   Olaf 

Re: [ANNOUNCE] Apache Bigtop 1.5.0 released

[Olaf Flebbe]

That’s amazing! Thank you Kengo and all the contributors for all your efforts. 

I am asking myself if we shouldn’t announce it on the @ASFbigtop twitter handle 
? 
We need to have a news page first, IMO . 

Unless someone else is stepping up I can try to create a news section and 
eventually convert the Website from xml to markdown, to make it more 
accessible. Beware: I have no web design skills :) 

Best
   Olaf 


> Am 16.12.2020 um 15:08 schrieb Konstantin Boudnik :
> 
> Great news! And thanks to everyone who contributed to make this release 
> happen!
> I wasn't one of them, but I am getting back to the game as my travels getting 
> more predictable now ;)
> 
> It is a great way to finish this ... a-ah not so easy year: the community and 
> the project are moving forward!
> 
> --
> With regards,
>  Cos
> 
> On 16.12.2020 08:01, Kengo Seki wrote:
>> On behalf of the Apache Bigtop team, I'd love to announce the general
>> availability of the Bigtop 1.5.0 release.
>> The release is available here:
>>   https://bigtop.apache.org/download.html#releases
>> A few highlights of this release include:
>>   * Four different Linux distributions are newly supported: CentOS 8,
>> Debian 10, Fedora 31, and Ubuntu 18.04
>>   * Bigtop Mpack is added as a new feature, that enables users to
>> deploy Bigtop components via Apache Ambari
>>   * Livy and ELK stack (Elasticsearch, Logstash, Kibana) are newly
>> added as components
>>   * Many component upgrades, e.g., Hadoop 2.10.1, Spark 2.4.5, HBase
>> 1.5.0, Hive 2.3.6, Kafka 2.4.0, Zeppelin 0.8.2
>> With Bigtop 1.5.0 the community continues to deliver the most advanced
>> big data stack to date. More details about 1.5.0 release are here:
>>   https://bigtop.apache.org/release-notes.html
>> Deploying Bigtop is easy: grab the repo/list file for your favorite
>> Linux distribution:
>>   https://www.apache.org/dyn/closer.lua/bigtop/bigtop-1.5.0/repos/
>> and you'll be running your very own bigdata cluster in no time!
>> We welcome your help and feedback. For more information on how to
>> report problems, and to get involved, visit the project website at:
>>   https://bigtop.apache.org
>> Lastly, I want to emphasize that this is a collaborative work done by
>> project contributors and other communities,
>> who continue to devote time to make Bigtop a better software. Thank
>> you all for making this release possible!
>> Regards,
>> Kengo Seki, Bigtop 1.5.0 Release Manager

FOSDEM'21

[Olaf Flebbe]

Hi,

your talk submissions are amazing! I love to attend the dev-room and see the 
progress Bigtop made in the past years. Now let’s wait for the dev room 
organizers. They potentially might ask us to squeeze two or three of our 
submissions into one slot.

Best
  Olaf

Re: Strange behaviour of website publishing

[Olaf Flebbe]

Hi,

The mechanism behind the ASF page deployments is git. So you are checking in a 
new revision. If the revision of a file is identical to the previous one (not 
considering the timestamp!) , it will not be included in the patchset deployed 
on server. 

Olaf



> Am 16.12.2020 um 00:29 schrieb Kengo Seki :
> 
> Hi,
> 
> I'm trying to update our website before the release announcement,
> but I encountered a strange behaviour of our site publishing Jenkins job.
> 
> First, after fixing BIGTOP-3465, I manually ran the job [1].
> Timestamps in the website were correctly updated [2].
> 
> Then I merged a commit which updates menus and download links [3] and
> rerun the job manually [4].
> Strangely, that updated some htmls as expected, but also reverted
> timestamps and versions in other htmls [5].
> 
> I tried to rerun the job again without any change [6].
> That recovered the timestamps correctly, but broke a link to the
> latest release in release-notes.html [7].
> 
> I've not figured out the root cause yet, so I'm going to take the
> following actions:
> 
> * Stop the daily build of our website temporarily.
> * Generate the website locally and push it to the asf-site branch
> directly. Then post the 1.5.0 release announcement.
> * Investigate the problem using asf-staging branch instead of
> asf-site. Once it's resolved, resume the daily build.
> 
> [1]: https://ci-builds.apache.org/job/BigTop/job/pipeline-site/69/console
> [2]: 
> https://github.com/apache/bigtop/commit/19f91fde74e7aef445226838b7bb91e6a2f98461
> [3]: 
> https://github.com/apache/bigtop/commit/678375eb75f7367f9114a3ad8087ba40f21444b6
> [4]: https://ci-builds.apache.org/job/BigTop/job/pipeline-site/70/console
> [5]: 
> https://github.com/apache/bigtop/commit/40e0d5946e581d3044c1b0b4067423a9a7c5054f
> [6]: https://ci-builds.apache.org/job/BigTop/job/pipeline-site/71/console
> [7]: 
> https://github.com/apache/bigtop/commit/dbf5d9a0664c35a8b0c28bb0fdd6ebb9d9644615
> 
> Kengo Seki 

Fwd: HPC, Big Data and Data Science devroom at FOSDEM'21

[Olaf Flebbe]

i asked for an extension of the rfp 

olaf


Anfang der weitergeleiteten Nachricht:

> Von: Kenneth Hoste 
> Datum: 15. Dezember 2020 um 08:57:12 MEZ
> An: Olaf Flebbe 
> Betreff: Aw: HPC, Big Data and Data Science devroom at FOSDEM'21
> 
> Hi Olaf,
> 
> We've extended the deadline yesterday evening until Fri Dec 18th, see 
> https://hpc-bigdata-fosdem21.github.io/ .
> 
> We won't do any further extensions.
> 
> 
> regards,
> 
> Kenneth
> 
>> On 15/12/2020 07:35, Olaf Flebbe wrote:
>> Hi Kenneth,
>> will the deadline extended? The Apache Bigtop community is likely to push an 
>> additional proposal otherwise it would like to enhance the already submitted 
>> proposal .
>> best
>> Olaf
>>>> Am 12.12.2020 um 11:43 schrieb Kenneth Hoste :
>>> 
>>> Dear open source enthusiast,
>>> 
>>> This is a one time message to promote the HPC, Big Data, and Data Science 
>>> devroom at FOSDEM'21 (https://fosdem.org).
>>> 
>>> You are receiving this message because you have submitted a talk proposal 
>>> for one of the previous editions of the devroom, or have been involved with 
>>> the organization of it.
>>> 
>>> FOSDEM'21 will be fully virtual, for well known reasons.
>>> This makes the organization of the event and individual devrooms 
>>> particularly challenging.
>>> 
>>> We hope that you are willing to help promote the HPC, Big Data, and Data 
>>> Science devroom at FOSDEM'21, or perhaps submit a talk proposal yourself...
>>> 
>>> The submission deadline is really close (Tue Dec 15th 2020, which may get 
>>> extended by a couple of days), but very light weight: it basically comes 
>>> down to a talk title + short description (no paper, etc.).
>>> 
>>> The actual event is over the weekend of Feb 6-7 2021;
>>> the HPC devroom itself is planned for Sunday Feb 7th 2021.
>>> 
>>> Accepted talks will need to be pre-recorded by mid Jan'21.
>>> Talks will be live streamed during the event with live Q&A & discussions 
>>> after each talk.
>>> 
>>> Please visit https://hpc-bigdata-fosdem21.github.io for more information.
>>> 
>>> Feel free to contact me in case of questions!
>>> 
>>> Please consider forwarding this message to people or projects who may be 
>>> interested in submitting a talk proposal, either to the HPC devroom, or one 
>>> of the many other devrooms at FOSDEM'21 
>>> (https://fosdem.org/2021/schedule/tracks).
>>> 
>>> 
>>> regards,
>>> 
>>> Kenneth

Fwd: HPC, Big Data and Data Science devroom at FOSDEM'21

[Olaf Flebbe]

Hi,

Anyone to promote Bigtop at FOSDEM’21 (virtual) ? 

Would be really great if we could get a Bigtop Version 1.5 Upgrade Talk or even 
a Bigtop@Wikimedia (Luca?) presented.
FOSDEM is the biggest OSS event in Europe, by large. 

Best 
   Olaf





> Anfang der weitergeleiteten Nachricht:
> 
> Von: Kenneth Hoste 
> Betreff: HPC, Big Data and Data Science devroom at FOSDEM'21
> Datum: 12. Dezember 2020 um 11:43:25 MEZ
> An: undisclosed-recipients: ;
> 
> Dear open source enthusiast,
> 
> This is a one time message to promote the HPC, Big Data, and Data Science 
> devroom at FOSDEM'21 (https://fosdem.org ).
> 
> You are receiving this message because you have submitted a talk proposal for 
> one of the previous editions of the devroom, or have been involved with the 
> organization of it.
> 
> FOSDEM'21 will be fully virtual, for well known reasons.
> This makes the organization of the event and individual devrooms particularly 
> challenging.
> 
> We hope that you are willing to help promote the HPC, Big Data, and Data 
> Science devroom at FOSDEM'21, or perhaps submit a talk proposal yourself...
> 
> The submission deadline is really close (Tue Dec 15th 2020, which may get 
> extended by a couple of days), but very light weight: it basically comes down 
> to a talk title + short description (no paper, etc.).
> 
> The actual event is over the weekend of Feb 6-7 2021;
> the HPC devroom itself is planned for Sunday Feb 7th 2021.
> 
> Accepted talks will need to be pre-recorded by mid Jan'21.
> Talks will be live streamed during the event with live Q&A & discussions 
> after each talk.
> 
> Please visit https://hpc-bigdata-fosdem21.github.io 
>  for more information.
> 
> Feel free to contact me in case of questions!
> 
> Please consider forwarding this message to people or projects who may be 
> interested in submitting a talk proposal, either to the HPC devroom, or one 
> of the many other devrooms at FOSDEM'21 
> (https://fosdem.org/2021/schedule/tracks 
> ).
> 
> 
> regards,
> 
> Kenneth

Spam

[Olaf Flebbe]

Hi Luca,

thanks for addressing the spam issue at INFRA.

Best 
  Olaf

Re: Update

[Olaf Flebbe]

hi,

fully supporting evans:
the unconnected disk do not contain anything valuable, please remove. it might 
make sense to even recreate the current disks on ssd, a bit larger as before if 
needed.

olaf

> Am 10.11.2020 um 08:09 schrieb Evans Ye :
> 
> Yes I think overall your plan is good.
> What's the purpose of leveraging EBS snapshot? Is it to backup the things
> we have before migration?
> Except for the master node(have jenkins settings stored on disk), all those
> slaves can be wiped out directly.
> 
> 
> 
> Kengo Seki  於 2020年11月10日 週二 下午2:42寫道：
> 
>> Thanks everyone for the information! Now I understand our circumstances.
>> So we're going to split two 1TB volumes attached to slave06 and 07
>> into four 500GB volumes (and change their type to gp2), reattach them
>> to 02, 03, 06 and 07, and remove currently unused two 1TB volumes,
>> right?
>> 
>>> Kengo would you like to take this, or you need a help?
>> 
>> I think I can do them somehow (maybe using EBS snapshot?), but let me
>> ask your help if I'm stuck. :)
>> 
>> Kengo Seki 
>> 
>> On Tue, Nov 10, 2020 at 1:00 AM Evans Ye  wrote:
>>> 
>>> OK. I got it now.
>>> So the newly created volumes are currently attached to slave06_2 and
>>> slave07_2, respectively.
>>> However, they're standard HDD, not GP2 SSD. I think we can take this
>> chance
>>> to recreate those 2 slaves and do an overhaul of our infrastructure.
>>> 
>>> Kengo would you like to take this, or you need a help?
>>> 
>>> Evans
>>> 
>>> Olaf Flebbe  於 2020年11月6日 週五 上午2:40寫道：
>>> 
>>>> Hi,
>>>> 
>>>> OMG . I think I did it.
>>>> 
>>>> A few years ago two of the instance had a hardware problems and did not
>>>> reboot any more, filesystem was corrupted and so on.  That was at the
>> time
>>>> of the spectre vulnarability discovery. (2018) . At that time AWS had
>> major
>>>> instabilities since updating firmware seem to have failed for some
>> classes
>>>> of hardware.
>>>> 
>>>> I tried to recreate them as close as possible but I may have left
>>>> accidentely the volumes around. Please lets delete them.
>>>> 
>>>> Olaf
>>>> 
>>>>> Am 05.11.2020 um 14:44 schrieb Konstantin Boudnik :
>>>>> 
>>>>> Thanks Evans!
>>>>> 
>>>>> It's great you found the details: they are definitely accurate as I
>> am
>>>>> recalling now. Kengo, do you think splitting the volumes would help
>> us
>>>> for a
>>>>> while? Or perhaps we shall try to expand the resource pool (which
>> might
>>>> take a
>>>>> while)?
>>>>> 
>>>>> Thanks!
>>>>> Cos
>>>>> 
>>>>> On Thu, Nov 05, 2020 at 12:32PM, Evans Ye wrote:
>>>>>> In fact, the original deal of our resource is as follows:
>>>>>> 
>>>>>>> 1 m3.2xlarge for CI
>>>>>>> 4 m3.xlarge for CI and demo
>>>>>>> 3 1TB EBS volumes
>>>>>>> 5 elastic IP addresses
>>>>>> 
>>>>>> So technically we should not use that 2 additional 1T volumes
>> (created
>>>> in
>>>>>> 2018).
>>>>>> Instead, I think what we can do is to split up one of the existing
>> 1TB
>>>>>> volumes(ex: attached to slave07) into smaller volumes for slave02,
>> 03.
>>>>>> 
>>>>>> 
>>>>>> Konstantin Boudnik  於 2020年11月4日 週三 下午2:28寫道：
>>>>>> 
>>>>>>> Kengo,
>>>>>>> 
>>>>>>> We had an agreement with EMR folks that we are using the resources
>>>>>>> available
>>>>>>> to us and it is included into their budget (or something to this
>>>> extent).
>>>>>>> If
>>>>>>> you see some of the resources available under our account - I
>> don't see
>>>>>>> why we
>>>>>>> can't use them.
>>>>>>> 
>>>>>>> If for whatever reason we need to expand the pool, that would
>> require a
>>>>>>> separate conversation with nice folks from that team, I imagine.
>> Please
>>>>>>> let me
>>>>>>> know if I can help

Re: Update

[Olaf Flebbe]

Hi,

OMG . I think I did it.

A few years ago two of the instance had a hardware problems and did not reboot 
any more, filesystem was corrupted and so on.  That was at the time of the 
spectre vulnarability discovery. (2018) . At that time AWS had major 
instabilities since updating firmware seem to have failed for some classes of 
hardware.

I tried to recreate them as close as possible but I may have left accidentely 
the volumes around. Please lets delete them.

Olaf

> Am 05.11.2020 um 14:44 schrieb Konstantin Boudnik :
> 
> Thanks Evans!
> 
> It's great you found the details: they are definitely accurate as I am
> recalling now. Kengo, do you think splitting the volumes would help us for a
> while? Or perhaps we shall try to expand the resource pool (which might take a
> while)? 
> 
> Thanks!
>  Cos
> 
> On Thu, Nov 05, 2020 at 12:32PM, Evans Ye wrote:
>> In fact, the original deal of our resource is as follows:
>> 
>>> 1 m3.2xlarge for CI
>>> 4 m3.xlarge for CI and demo
>>> 3 1TB EBS volumes
>>> 5 elastic IP addresses
>> 
>> So technically we should not use that 2 additional 1T volumes (created in
>> 2018).
>> Instead, I think what we can do is to split up one of the existing 1TB
>> volumes(ex: attached to slave07) into smaller volumes for slave02, 03.
>> 
>> 
>> Konstantin Boudnik  於 2020年11月4日 週三 下午2:28寫道：
>> 
>>> Kengo,
>>> 
>>> We had an agreement with EMR folks that we are using the resources
>>> available
>>> to us and it is included into their budget (or something to this extent).
>>> If
>>> you see some of the resources available under our account - I don't see
>>> why we
>>> can't use them.
>>> 
>>> If for whatever reason we need to expand the pool, that would require a
>>> separate conversation with nice folks from that team, I imagine. Please
>>> let me
>>> know if I can help with this going forward.
>>> 
>>> Thanks!
>>>  Cos
>>> 
>>> On Wed, Nov 04, 2020 at 11:11AM, Kengo Seki wrote:
>>>> Thanks for the comment, Cos! I was able to start docker service on
>>>> docker-slave-02 without replacing and am running some Jenkins jobs on
>>>> it now, so I'll replace it in the short future.
>>>> I have a few things that I'd like to ask additionally:
>>>> 
>>>> * docker-slave-02 and 03 have a gp2 storage as a root volume that has
>>>> only 8GiB capacity, and they sometimes run short and stop the CI.
>>>>  May I increase them to 20 or 30 GiB when I replace those instances?
>>>> (I'm not sure what is our budget)
>>>> 
>>>> * They use an instance store with 30GiB to put docker images into it,
>>>> and they also sometimes run short.
>>>>  It seems there are two unused volumes with 1TiB (vol-ae71114e and
>>>> vol-4efa69ae) on AWS console.
>>>>  May I attach them to 02 and 03 instead of instance stores, or are
>>>> they backups or something?
>>>> 
>>>> Kengo Seki 
>>>> 
>>>> On Mon, Nov 2, 2020 at 6:41 PM Konstantin Boudnik 
>>> wrote:
>>>>> 
>>>>> I'd say let replace the broken one. I don't think there's a sentimental
>>>>> value attached ;)
>>>>> 
>>>>> --
>>>>> With regards,
>>>>>   Cos
>>>>> 
>>>>> On 02.11.2020 08:16, Kengo Seki wrote:
>>>>>> Thanks for updating Olaf! I've just noticed the Jenkins UI became
>>> cool :)
>>>>>> Regarding docker-slave-02, I'll try to replace it after waiting for a
>>>>>> while to make sure there's no objection.
>>>>>> 
>>>>>> Kengo Seki 
>>>>>> 
>>>>>> On Mon, Nov 2, 2020 at 1:39 PM Jun HE  wrote:
>>>>>>> 
>>>>>>> Thanks a lot for the update, Olaf!
>>>>>>> 
>>>>>>> Olaf Flebbe  于2020年10月31日周六 上午3:24写道：
>>>>>>> 
>>>>>>>> Hi,
>>>>>>>> 
>>>>>>>> All machines patched. Jenkins and it plugins are updated:
>>>>>>>> 
>>>>>>>> Things to be noted:
>>>>>>>> 
>>>>>>>> * Slave 2 seems to be in serious problems. The disk image seems to
>>> be
>>>>>>>> corrupt, I would say:
>>>>>>>> One of the problems: docker does not start any more.
>>>>>>>> Is there anything important on it ? If yes please contact me. I
>>> would
>>>>>>>> recommend to set up slave2 from scratch again.
>>>>>>>> 
>>>>>>>> * There was a warning regarding Copy Artifacts Plugin. It now
>>> imposes
>>>>>>>> stricter rules. Not sure if there is a job depending on it.
>>>>>>>> 
>>>>>>>> * I removed the CVS plugin.
>>>>>>>> 
>>>>>>>> Everything else seem to working as usual.
>>>>>>>> 
>>>>>>>> Best,
>>>>>>>> Olaf
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> Am 30.10.2020 um 19:09 schrieb Olaf Flebbe :
>>>>>>>>> 
>>>>>>>>> Hi,
>>>>>>>>> 
>>>>>>>>> I am doing an update of the machines in CI . Seems a couple of
>>> security
>>>>>>>> fixes are to be applied.
>>>>>>>>> 
>>>>>>>>> Olaf
>>>>>>>> 
>>>>>>>> 
>>> 

Re: Update

[Olaf Flebbe]

Hi,

All machines patched. Jenkins and it plugins are updated:

Things to be noted:

* Slave 2 seems to be in serious problems. The disk image seems to be corrupt, 
I would say:
One of the problems: docker does not start any more. 
Is there anything important on it ? If yes please contact me. I would recommend 
to set up slave2 from scratch again.

* There was a warning regarding Copy Artifacts Plugin. It now imposes stricter 
rules. Not sure if there is a job depending on it.

* I removed the CVS plugin.

Everything else seem to working as usual.

Best,
Olaf




> Am 30.10.2020 um 19:09 schrieb Olaf Flebbe :
> 
> Hi, 
> 
> I am doing an update of the machines in CI . Seems a couple of security fixes 
> are to be applied.
> 
> Olaf

Update

[Olaf Flebbe]

Hi, 

I am doing an update of the machines in CI . Seems a couple of security fixes 
are to be applied.

Olaf

Bigtop Website is now automatically built and deployed

[Olaf Flebbe]

Hi,

News:
* Bigtops Website is now automatically built and deployed
* Implemented as a Jenkins pipeline job: see 
https://github.com/apache/bigtop/blob/master/Jenkinsfile.site 

* Updates are running on 
https://ci-builds.apache.org//job/BigTop/job/pipeline-site/ 


Pushes to master will trigger an update of the site once a day.

Best
   Olaf 

Re: Developer documenation

[Olaf Flebbe]

Hi, 

you could either 

add more / updated Markdown files in the repository by an pull request.
Update / Fix the website in src/site in the repository with an pull request.
Move/copy info from the cwiki/confluence to Markdown into the repository.
or if you would like to work on our cwiki please send me your account and the 
pages you like to update.

Olaf

> Am 23.09.2020 um 01:03 schrieb Matt Andruff :
> 
> Olaf, I'm glad we are in agreement.
> 
> I'm not sure what to ask for but I definitely feel we can de-emphasize the
> 0.x stuff and promote the new stuff...  Happy to help, just point me in the
> right direction.  I am excited to contribute and it's really a dream come
> true. (sorry for admitting this out loud)
> 
> On Tue, Sep 22, 2020 at 3:38 PM Olaf Flebbe  wrote:
> 
>> Hi Matt,
>> 
>> I totally agree that documentation needs to be updated.
>> 
>> From my perspective markdown files in the git repository are most suitable
>> for users and accessible to our users. I am very open to accept Pull
>> requests / Patches for docs.
>> 
>> Contributors can ask for confluence (cwiki) privileges as well, if we are
>> aligned on what they want to work on. (At least that’s my opinion, not
>> necessary representative for the project)
>> 
>> Best
>>  Olaf
>> 
>> 
>> 
>>> Am 22.09.2020 um 19:55 schrieb Matt Andruff :
>>> 
>>> This goes back to why we need to update the documentation... I'll update
>>> that soon so it is easier to find.
>>> 
>>> Essentially I suggest using the instructions on git to get going:
>>> 
>>> 
>> https://github.com/apache/bigtop#for-developers-building-a-component-from-git-repository
>>> 
>>> For more info on the commands used (on that git page use this helpful gem
>>> (that is buried a bit)
>>> 
>> https://cwiki.apache.org/confluence/display/BIGTOP/Quickstart+Guide%3A+Bigtop+Integration+Test+Framework+2.0
>>> 
>>> Whatever you do, please use docker to build it It will save you a
>> heap
>>> of dependency issues. Please learn from my pain.
>>> 
>>> Hope this helps
>>> 
>>> Matt
>>> 
>>> 
>>> 
>>> 
>>> On Tue., Sep. 22, 2020, 09:51 Manfred PAUL, 
>> wrote:
>>> 
>>>> Hi,
>>>> 
>>>> I would like to participate in developing bigtop. For that I am
>> searching
>>>> for some material (doc, wiki, video, etc) to get all the information
>> needed
>>>> to build the bigtop distribution.
>>>> 
>>>> All the information I found was for older versions of bigtop (1.1, 1.2
>> and
>>>> 1.3) but might still be up to date, so I am asking.
>>>> 
>>>> Is written that a developer should use the pre-built docker images for
>>>> building the distribution and not set up his or her own toolchain. Or
>>>> should we use only a CI? Is it even possible to build everything on a
>> 16GB
>>>> Macbook Pro? Etc.
>>>> 
>>>> Thanks
>>>> 
>>>> Manfred
>>>> 
>> 
>> 

Re: Current status of the next release

[Olaf Flebbe]

Hi Kengo,

great to see progress here. Thanks for the update

Olaf

> Am 23.09.2020 um 16:48 schrieb Kengo Seki :
> 
> Hi everyone,
> 
> Let me share the current status of the next release.
> We've steadily moved forward and almost reached the goal.
> I really appreciate your cooperation and contribution.
> 
> The attached image is the current packaging matrix [1].
> As you can see, we've succeeded on building most of all combinations
> between components and distros/platforms, though ppc64 is temporarily
> disabled due to the machine problem.
> 
> The only build failure is Spark on CentOS 7 for arm64 (and ppc64),
> but it's tracked as BIGTOP-3397 and is supposed to be resolved
> once the next version of snappy-java is released and we add a small patch
> for Spark that uses it (or, if that release doesn't make it, we can
> build it ourselves).
> 
> So, our remaining task is to complete the deployment and smoke test matrix 
> [2].
> Currently, the following components have not succeeded on any
> distro/platform yet,
> so I'm going to begin with these components.
> 
> - GPDB
> - Livy
> - Oozie (already filed as BIGTOP-3406)
> - QFS
> - Spark
> 
> I'm planning to cut the release branch once all components passed
> their smoke test
> on at least one distro/platform for deb and rpm respectively
> (e.g., it's regarded as OK if the smoke test succeeds on x86_64/CentOS
> 7 and arm64/Debian 9),
> and hopefully, I'd like to release v1.5.0 within this October.
> Let me know if this schedule is not convenient for you :)
> 
> [1]: https://ci.bigtop.apache.org/job/Bigtop-trunk-packages/
> [2]: https://ci.bigtop.apache.org/job/Bigtop-trunk-smoke-tests/
> 
> Kengo Seki 

[jira] [Created] (BIGTOP-3419) Automate Deployment to asf-site

[Olaf Flebbe (Jira)]

Olaf Flebbe created BIGTOP-3419:
---

 Summary: Automate Deployment to asf-site 
 Key: BIGTOP-3419
 URL: https://issues.apache.org/jira/browse/BIGTOP-3419
 Project: Bigtop
  Issue Type: Task
  Components: website
Affects Versions: 1.4.0
Reporter: Olaf Flebbe
Assignee: Olaf Flebbe
 Fix For: 1.5.0


Automate the deployment with a Jenkins pipeline job started by Jenkinsfile.site



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Developer documenation

[Olaf Flebbe]

Hi Matt,

I totally agree that documentation needs to be updated. 

From my perspective markdown files in the git repository are most suitable for 
users and accessible to our users. I am very open to accept Pull requests / 
Patches for docs.

Contributors can ask for confluence (cwiki) privileges as well, if we are 
aligned on what they want to work on. (At least that’s my opinion, not 
necessary representative for the project) 

Best
  Olaf



> Am 22.09.2020 um 19:55 schrieb Matt Andruff :
> 
> This goes back to why we need to update the documentation... I'll update
> that soon so it is easier to find.
> 
> Essentially I suggest using the instructions on git to get going:
> 
> https://github.com/apache/bigtop#for-developers-building-a-component-from-git-repository
> 
> For more info on the commands used (on that git page use this helpful gem
> (that is buried a bit)
> https://cwiki.apache.org/confluence/display/BIGTOP/Quickstart+Guide%3A+Bigtop+Integration+Test+Framework+2.0
> 
> Whatever you do, please use docker to build it It will save you a heap
> of dependency issues. Please learn from my pain.
> 
> Hope this helps
> 
> Matt
> 
> 
> 
> 
> On Tue., Sep. 22, 2020, 09:51 Manfred PAUL,  wrote:
> 
>> Hi,
>> 
>> I would like to participate in developing bigtop. For that I am searching
>> for some material (doc, wiki, video, etc) to get all the information needed
>> to build the bigtop distribution.
>> 
>> All the information I found was for older versions of bigtop (1.1, 1.2 and
>> 1.3) but might still be up to date, so I am asking.
>> 
>> Is written that a developer should use the pre-built docker images for
>> building the distribution and not set up his or her own toolchain. Or
>> should we use only a CI? Is it even possible to build everything on a 16GB
>> Macbook Pro? Etc.
>> 
>> Thanks
>> 
>> Manfred
>> 

Re: Bigtop website

[Olaf Flebbe]

Hey Matt,


> Hey I would totally would love to help fix up the documentation as
> previously mentioned.  So your saying I can search in git to find the site
> make some commits and we should be good to go?

Yes, absolutely. 

Best
Olaf

Re: Developer documenation

[Olaf Flebbe]

Hi Manfred,

Great!

AFAIK the most up to date information on how to (re)compile an package you will 
find here
https://cwiki.apache.org/confluence/display/BIGTOP/How+to+build+Bigtop-trunk 


I am assuming you are targeting linux, not macos, right?

IMO When developing it makes sense to compile „within" the provided docker 
images or improving the docker images.
The CI’s job is to ensure changes do work on all platforms.

I was using a macbook pro 16GB when I developed packages, so it is possible, 
though not always optimal, since docker filesystem (volume mounts from macos to 
docker guest) is buggy and slow.

Running smoke tests on a mac is possible with workarounds.

Needed workarounds depend very much on what you are trying to accomplish:  
VMware Fusion, docker4mac or depending on CI are at least 3 options I would 
recommend depending on usecase. 

Best
Olaf




> Am 22.09.2020 um 15:48 schrieb Manfred PAUL :
> 
> Hi,
> 
> I would like to participate in developing bigtop. For that I am searching
> for some material (doc, wiki, video, etc) to get all the information needed
> to build the bigtop distribution.
> 
> All the information I found was for older versions of bigtop (1.1, 1.2 and
> 1.3) but might still be up to date, so I am asking.
> 
> Is written that a developer should use the pre-built docker images for
> building the distribution and not set up his or her own toolchain. Or
> should we use only a CI? Is it even possible to build everything on a 16GB
> Macbook Pro? Etc.
> 
> Thanks
> 
> Manfred

Re: Bigtop website

[Olaf Flebbe]

Hi Evans,

ok, let me explain it in detail:

* The website was completly static content previously as well. 
* I did not (until now) do any change to the pages.
* The website is build by running „mvn site“ in bigtop root. 
* The target/site directory has to be put onto a webserver.

The only thing what is changing is _how_ to put it on a webserver. Previously 
bigtop used the maven site plugin to stage it to a CMS via svn. See 
 in pom.xml.

The now preferred way is  to check the html pages into the git repository 
(rather svn) on a special branch "asf-site" , together with a file „.asf.yaml“ 
in the root with special content. When we push it a special handler (gitpubsub) 
is triggered to push the static content to the website.

On builds.apache.org <http://builds.apache.org/> system this is done in job 
„Bigtop/site“. Additionally there is a special node "git-websites“ which will 
allow a Jenkins job to push to git, because git credentials are already 
injected into the job by INFRA.

That’s how to push to bigtop.apache.org <http://bigtop.apache.org/>
——

For testing purposes I pushed to bigtop.staged.apache.org 
<http://bigtop.staged.apache.org/> via branch asf-staging and .asf.yaml , but 
the mechanisms are the same. 


Hope that clarifies the situation,
Olaf


> Am 21.09.2020 um 17:34 schrieb Evans Ye :
> 
> Hi Olaf,
> 
> Thanks for the work!
> It seems the site code has changed quite a bit. Could you briefly share
> what changes were made? For example, it seems that the pages are all static
> now. From the CI job it seems that jenkins slave provided by infra has the
> permission to push to any git repository. Have we done something
> differently? Thanks!
> 
> - Evans
> 
> 
> Olaf Flebbe  於 2020年9月20日 週日 下午9:51寫道：
> 
>> Hi *,
>> 
>> Finally found time to look into implementing the workflow for pushing our
>> website automatically bigtop.apache.org <http://bigtop.apache.org/>.
>> 
>> As a test I create a job checkin the git repository once a day for new
>> commits an deploying to bigtop.staged.apache.org <
>> http://bigtop.staged.apache.org/>
>> 
>> Please have a look if you see anything weird / unexpected on it and report.
>> Changing the website is now as easy as having a git commit on src/site .
>> 
>> If I get no or positive (gasp!)  feedback I will change the job in a few
>> days to update the prod website instead
>> 
>> Additional ideas : Using a jenkins pipeline job, refactor pom.xml to
>> remove now obsolete parts.
>> 
>> Best
>>   Olaf

Bigtop website

[Olaf Flebbe]

Hi *,

Finally found time to look into implementing the workflow for pushing our 
website automatically bigtop.apache.org .

As a test I create a job checkin the git repository once a day for new commits 
an deploying to bigtop.staged.apache.org 

Please have a look if you see anything weird / unexpected on it and report.
Changing the website is now as easy as having a git commit on src/site .

If I get no or positive (gasp!)  feedback I will change the job in a few days 
to update the prod website instead

Additional ideas : Using a jenkins pipeline job, refactor pom.xml to remove now 
obsolete parts. 

Best 
   Olaf 

[jira] [Created] (BIGTOP-3409) Move bigtop website off from CMS: staging

[Olaf Flebbe (Jira)]

Olaf Flebbe created BIGTOP-3409:
---

 Summary: Move bigtop website off from CMS: staging
 Key: BIGTOP-3409
 URL: https://issues.apache.org/jira/browse/BIGTOP-3409
 Project: Bigtop
  Issue Type: Task
  Components: website
Affects Versions: 1.4.0
Reporter: Olaf Flebbe
Assignee: Olaf Flebbe
 Fix For: 1.5.0


First step:

Try a staging website 

I created a job for pushing the website to the asf-staging branch at
https://ci-builds.apache.org/job/BigTop/job/site/

According to 
https://cwiki.apache.org/confluence/display/INFRA/git+-+.asf.yaml+features we 
need to create {{.asf.yaml}} file for uploading to final destination.

The content should be
{code}
staging:
  profile: ~
  whoami:  asf-staging
 {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Project website

[Olaf Flebbe]

Hi,

https://issues.apache.org/jira/browse/BIGTOP-3388 
<https://issues.apache.org/jira/browse/BIGTOP-3388>

Olaf

> Am 20.08.2020 um 21:54 schrieb Andrew Wetmore :
> 
> Forgot to copy this to the list.
> 
> -- Forwarded message -
> From: Andrew Wetmore mailto:andr...@apache.org>>
> Date: Thu, Aug 20, 2020 at 4:53 PM
> Subject: Re: Project website
> To: Olaf Flebbe mailto:o...@oflebbe.de>>
> 
> 
> Hi, Olaf:
> 
> Nice to speak with you.
> 
> Could you open a Jira ticket aimed at INFRA to track the project's
> migration off the CMS and ask you question on it? Then one of my smarter
> colleagues will pick it up and run with it.
> 
> Thanks!
> 
> Andrew
> 
> On Thu, Aug 20, 2020 at 4:34 PM Olaf Flebbe  wrote:
> 
>> 
>> Hi Andrew,
>> 
>> thanks for reaching out to Bigtop. I reviewed the information you supplied
>> and I feel a bit lost.
>> 
>> We are already using a static site generate (mvn site plugin) and
>> uploading this to
>> https://svn.apache.org/repos/infra/websites/staging/bigtop/trunk/content 
>> manually.
>> I remember
>> I remember the last one is a bit a pain since one needs proper access
>> rights to do this and this involves to locally overwrite the adress and add
>> credentials. (We do not use subversion for anything but this)
>> 
>> I would very much prefer to have a jenkins job: Checking out git master,
>> running mvn site and uploading the content of the target/site directory to
>> the final destination. There is no need for a staging environment since one
>> can test changes locally (Our website is about 60 files, that’s it). I may
>> have to remove some obsolete info first, but it works for now.
>> 
>> How can we do the „uploading to final destination“ ? I suppose we are not
>> the first one doing this. I suppose this is the pysubpub thing but all I
>> can find is „your" page
>> https://cwiki.apache.org/confluence/display/INFRA/Building+and+deploying+web+sites+with+PyPubSub%2C+Jenkins%2C+and+Buildbot#BuildinganddeployingwebsiteswithPyPubSub,Jenkins,andBuildbot-MavenSitePlugin
>> <https://cwiki.apache.org/confluence/display/INFRA/Building+and+deploying+web+sites+with+PyPubSub,+Jenkins,+and+Buildbot#BuildinganddeployingwebsiteswithPyPubSub,Jenkins,andBuildbot-MavenSitePlugin
>>  
>> <https://cwiki.apache.org/confluence/display/INFRA/Building+and+deploying+web+sites+with+PyPubSub,+Jenkins,+and+Buildbot#BuildinganddeployingwebsiteswithPyPubSub,Jenkins,andBuildbot-MavenSitePlugin>>
>> .
>> 
>> Where can we start from ?
>> 
>> Best,
>> Olaf
>> 
>> 
>> Am 04.08.2020 um 15:36 schrieb Andrew Wetmore :
>> 
>> Hi:
>> 
>> I am part of the Infrastructure team, and am writing to ask whether your
>> project is still using the Apache CMS for your project website. As you
>> know, the CMS is reaching end-of-life, and we need projects to move their
>> websites onto a different option within the next few weeks.
>> 
>> There are several alternatives available, including those listed on this
>> page [1] on managing project websites. Infra is assembling a Wiki page [2]
>> on migrating a website from the CMS, and is looking forward to helping
>> projects with this transition.
>> 
>> Please let me know whether your site is still on the Apache CMS and, if so,
>> who will be the project point-of-contact with Infra for the migration.
>> 
>> Thank you!
>> 
>> 
>> 
>> 
>> [1] https://infra.apache.org/project-site.html
>> 
>> [2]
>> 
>> https://cwiki.apache.org/confluence/display/INFRA/Migrate+your+project+website+from+the+Apache+CMS
>> 
>> 
>> --
>> Andrew Wetmore
>> Technical Writer-Editor
>> Infra
>> *Apache Software Foundation*
>> andr...@apache.org
>> 
>> <
>> https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail
>>> 
>> Virus-free.
>> www.avast.com
>> <
>> https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail
>>> 
>> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>> 
>> 
>> 
> 
> -- 
> Andrew Wetmore
> Technical Writer-Editor
> Infra
> *Apache Software Foundation*
> andr...@apache.org
> 
> 
> -- 
> Andrew Wetmore
> Technical Writer-Editor
> Infra
> *Apache Software Foundation*
> andr...@apache.org <mailto:andr...@apache.org>

[jira] [Created] (BIGTOP-3388) Move Bigtop Site off Apache CMS

[Olaf Flebbe (Jira)]

Olaf Flebbe created BIGTOP-3388:
---

 Summary: Move Bigtop Site off Apache CMS
 Key: BIGTOP-3388
 URL: https://issues.apache.org/jira/browse/BIGTOP-3388
 Project: Bigtop
  Issue Type: Task
 Environment: Apache bigtop 
Reporter: Olaf Flebbe


[~andreww] asked us to look into moving away from CMS.

We are already using a static site generate (mvn site plugin) and uploading 
this right now
to  https://svn.apache.org/repos/infra/websites/staging/bigtop/trunk/content 
manually. 

I would very much prefer to have a jenkins job: Checking out bigtop's git 
master, running mvn site and uploading the content of the target/site directory 
to the final destination. There is no immediate need for a staging environment 
since one can test changes locally (Our website is about 60 files, that’s it). 
I may have to remove some obsolete info first, but it works for now.

How can we do the „uploading to final destination“ ? I suppose we are not the 
first one doing this.  I cannot find any information on how to do this.






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

builds.apache.org: bigtop jobs will get lost

[Olaf Flebbe]

Hi,

honestly I am a bit surprised, looking at builds.apache org:

There are a couple of jobs running there:

One Job:
"Deploys Bigtop test execution into snapshot repository“

mvn clean

export HADOOP_CONF_DIR=
export HADOOP_HOME=
mvn -f bigtop-tests/test-execution/pom.xml -DskipTests -DskipITs deploy 
-DrepositoryId=apache.snapshots.https

Next Job:
Deploys a top level Bigtop pom file into snapshot repository
mvn deploy

And a post job of both of them

Essentially doing 
mvn -f bigtop-tests/test-artifacts/pom.xml -U clean deploy

Question from my side:
Do we need them, since bui...@apache.org  will be 
turned off  in *two* days ?

This seems the result of the job:

https://search.maven.org/search?q=bigtop 


IMO this does not make sense at all any more and can be deleted/lost for good. 
Or do I miss something ?

Best 
   Olaf

Re: Project website

[Olaf Flebbe]

Hi Andrew,

thanks for reaching out to Bigtop. I reviewed the information you supplied and 
I feel a bit lost.

We are already using a static site generate (mvn site plugin) and uploading 
this to 
https://svn.apache.org/repos/infra/websites/staging/bigtop/trunk/content 
 
manually. I remember 
I remember the last one is a bit a pain since one needs proper access rights to 
do this and this involves to locally overwrite the adress and add credentials. 
(We do not use subversion for anything but this)

I would very much prefer to have a jenkins job: Checking out git master, 
running mvn site and uploading the content of the target/site directory to the 
final destination. There is no need for a staging environment since one can 
test changes locally (Our website is about 60 files, that’s it). I may have to 
remove some obsolete info first, but it works for now.

How can we do the „uploading to final destination“ ? I suppose we are not the 
first one doing this. I suppose this is the pysubpub thing but all I can find 
is „your" page 
https://cwiki.apache.org/confluence/display/INFRA/Building+and+deploying+web+sites+with+PyPubSub%2C+Jenkins%2C+and+Buildbot#BuildinganddeployingwebsiteswithPyPubSub,Jenkins,andBuildbot-MavenSitePlugin
 

 . 

Where can we start from ?

Best,
Olaf


> Am 04.08.2020 um 15:36 schrieb Andrew Wetmore :
> 
> Hi:
> 
> I am part of the Infrastructure team, and am writing to ask whether your
> project is still using the Apache CMS for your project website. As you
> know, the CMS is reaching end-of-life, and we need projects to move their
> websites onto a different option within the next few weeks.
> 
> There are several alternatives available, including those listed on this
> page [1] on managing project websites. Infra is assembling a Wiki page [2]
> on migrating a website from the CMS, and is looking forward to helping
> projects with this transition.
> 
> Please let me know whether your site is still on the Apache CMS and, if so,
> who will be the project point-of-contact with Infra for the migration.
> 
> Thank you!
> 
> 
> 
> 
> [1] https://infra.apache.org/project-site.html
> 
> [2]
> https://cwiki.apache.org/confluence/display/INFRA/Migrate+your+project+website+from+the+Apache+CMS
> 
> 
> -- 
> Andrew Wetmore
> Technical Writer-Editor
> Infra
> *Apache Software Foundation*
> andr...@apache.org
> 
> 
> Virus-free.
> www.avast.com
> 
> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

Re: Project website

[Olaf Flebbe]

hi,

thanks for sending out a reminder...

honestly I am not quite sure how our website really should work. it is a mess 
of jenkins, maven, subversion and manual steps involved and at some stage it 
seems to be a CMS in place. 

imo the complete website setup has to be reenginered. 

i can offer to retarget the content with minimal effort away from apache cms, 
but anyone other can step in as well. 

best,
olaf

Von meinem iPad gesendet

> Am 17.08.2020 um 13:54 schrieb Andrew Wetmore :
> 
> Hi:
> 
> I wrote to you two weeks ago about your project website, but have not seen
> a response. Whom should I contact directly? I am trying to see whether your
> project still uses the Apache CMS and, if so, what your plans are to
> migrate off it.
> 
> Thanks in advance for your help.
> 
> Andrew
> 
>>> On Tue, Aug 4, 2020 at 10:36 AM Andrew Wetmore  wrote:
>> Hi:
>> I am part of the Infrastructure team, and am writing to ask whether your
>> project is still using the Apache CMS for your project website. As you
>> know, the CMS is reaching end-of-life, and we need projects to move their
>> websites onto a different option within the next few weeks.
>> There are several alternatives available, including those listed on this
>> page [1] on managing project websites. Infra is assembling a Wiki page [2]
>> on migrating a website from the CMS, and is looking forward to helping
>> projects with this transition.
>> Please let me know whether your site is still on the Apache CMS and, if
>> so, who will be the project point-of-contact with Infra for the migration.
>> Thank you!
>> [1] https://infra.apache.org/project-site.html
>> [2]
>> https://cwiki.apache.org/confluence/display/INFRA/Migrate+your+project+website+from+the+Apache+CMS
>> --
>> Andrew Wetmore
>> Technical Writer-Editor
>> Infra
>> *Apache Software Foundation*
>> andr...@apache.org
>> 
>>  Virus-free.
>> www.avast.com
>> 
>> <#m_-7159533528443685238_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
> 
> 
> -- 
> Andrew Wetmore
> Technical Writer-Editor
> Infra
> *Apache Software Foundation*
> andr...@apache.org

Re: Updated the ci.bigtop.apache.org certificate

[Olaf Flebbe]

Thanks!

> Am 11.06.2020 um 14:24 schrieb Kengo Seki :
> 
> Just for your information, I've just renewed the certificate for
> ci.bigtop.apache.org in accordance with [1].
> Let me know if you find something wrong with it.
> 
> [1]: 
> https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+CI+Setup+Guide#BigtopCISetupGuide-Renewingthecert
> 
> Kengo Seki 

Re: New Committer: Yuqi Gu

[Olaf Flebbe]

Welcome to the Apache Bigtop project, yuqi !

Best 
   Olaf

Von meinem iPad gesendet

> Am 11.06.2020 um 04:13 schrieb Jun HE :
> 
> 
> The Project Management Committee (PMC) for Apache Bigtop has asked Yuqi Gu 
> to become a committer and we are pleased to announce that he has accepted!
> 
> His ASF account has been created as: guy...@apache.org
> 
> Being a committer enables easier contribution to the project since there is no
> need to go via the proxy patch submission process. This should enable better
> productivity.
> 
> Apache Bigtop practices CTR (commit-then-review) development process which 
> means that you can checkin the code without a +1 from a committer. This 
> doesn't 
> mean that you're encouraged to do so. Instead, we strongly recommend you to 
> seek for feedback before you commit. Please read the document before you act: 
> 
> * https://cwiki.apache.org/confluence/display/BIGTOP/CTR+Model  
> 
> If in doubt, exercise your best judgement and don't hesitate to ask questions 
> on 
> this very mailing list.
> 
> We're happy to have you on board and looking for many more contributions to
> come. Please join me in congratulating Yuqi Gu!
> 
> Regards,
> 
> Jun (on behalf of the ASF BigTop PMC)

Re: [DISCUSS] [VOTE] Release Ambari-Mpack as a seperate top component in Bigtop

[Olaf Flebbe]

be the maintainer going forward? :)
>>> 
>>> Regards,
>>> 
>>> Jun
>>> 
>>> 
>>> Matt Andruff  于2020年6月8日周一 下午10:32写道：
>>> 
>>>> What problem/case is this additional component trying to solve?
>>>> 
>>>> My opinion is that it's an enable meant tool to make it easy to use
>>>> BigTop by having it's configuration managed by Ambari.  Now is a great
>>> time
>>>> for making this move as HDP is no longer going to be managed by a
>>> company,
>>>> so BigTop could replace that role in Ambari.
>>>> 
>>>> This mpack allows BigTop to be installed into Ambari, and allows
>>>> Ambari to manage the hadoop packages.  Effectively, it would increase the
>>>> reach of BigTop to be able to be installed into and managed by Ambari.
>>>> 
>>>> I hear that you see a lot of other priorities and they sound like good
>>>> priorities.  I'm not able to contribute to those priorities as I don't
>>> know
>>>> enough in that area/do not understand what's required.  I would
>>> contribute
>>>> right now to Ambari MPack as it's more in my wheelhouse.
>>>> 
>>>> Hope that helps
>>>> 
>>>> Matt
>>>> 
>>>> On Sun, Jun 7, 2020 at 12:07 PM Olaf Flebbe  wrote:
>>>> 
>>>>> Hi,
>>>>> 
>>>>> Sorry, I am super lost about the implications of that vote.
>>>>> 
>>>>> Why do we need a vote in the first place?
>>>>> I am familiar with votes for additional committers, PMC additions and
>>>>> releases. For these votes we have the concept of binding votes by PMC
>>>>> members.
>>>>> 
>>>>> Why should I care about this ?
>>>>> Does it have legal implications?
>>>>> 
>>>>> Who will maintain this ?
>>>>> I am a bit surprised that we do anything with ambari, since we not even
>>>>> have a maintainer for it
>>>>> grep ambari MAINTAINERS.txt -> nothing
>>>>> 
>>>>> What problem/case is this additional component trying to solve?
>>>>> If you are asking for a vote I would like to have a more elaborate
>>>>> explanation. I looked at the linked ticket: Sorry I am totally lost
>>> about
>>>>> what is this about. If you would have simply done it, I wouldn’t have
>>>> cared.
>>>>> 
>>>>> As I understand it should somehow enable more contributions.
>>>>> Really? MPack is a amabari only technology, that seems not to even be
>>>>> defined properly. How could this ever end in more contributions to
>>>> Bigtop ?
>>>>> 
>>>>> What will happen if someone is -1 on this vote ?
>>>>> 
>>>>> If a contributor cares about adopting Bigtop to new business cases,
>>>> that’s
>>>>> very fine to me. Please go ahead. But please don’t ask for a vote. All
>>>>> committers still have the chance to reject patches.
>>>>> 
>>>>> Generally I am concerned that we care about an (from Bigtop side)
>>>>> unmaintained project while neglecting our core infrastructure and
>>>> packages
>>>>> at the same time.
>>>>> 1) The ci.bigtop.apache.org certificate expired again
>>>>> 2) The bigtop-trunk-packages do not reflect the architectures we ought
>>> to
>>>>> support (ubuntu-18.04 for instance)
>>>>> 3) Trunk-packages for amd64 looks like garbage
>>>>> 4) Still no-one working on Hadoop3
>>>>> 5) CI needs to be redone (with pipeline jobs) .
>>>>> IMO we might have wrong priorities.
>>>>> 
>>>>> Best,
>>>>> Olaf
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>>> Am 04.06.2020 um 19:04 schrieb Ganesh Raju :
>>>>>> 
>>>>>> +1
>>>>>> 
>>>>>> On Thu, Jun 4, 2020 at 12:01 PM Matt Andruff <
>>>> m...@andruffsolutions.com>
>>>>>> wrote:
>>>>>> 
>>>>>>> +1
>>>>>>> 
>>>>>>> On Thu., Jun. 4, 2020, 02:35 Yuqi Gu,  wrote:
>>>>>>> 
>>>>>>>> Hi folks,
>>>>>>>> 
>>>>>>>> As the discussion with Evans Ye, Jun He, and Matt Andruff (from
>>>> Ambari)
>>>>>>> on
>>>>>>>> https://github.com/apache/bigtop/pull/555#issuecomment-635228180:
>>>>>>>> 
>>>>>>>> For users / Bigtopers could easily modify Mapck and make
>>>> contributions
>>>>> to
>>>>>>>> it like bugfix and adding more services,
>>>>>>>> How about to decouple Mpack from Ambari and put Mpack as a
>>> standalone
>>>>>>>> component in Bigtop, like
>>> *bigtop-packages/src/common/bigtop-mpack*?
>>>>>>>> If so, it's convenient for users to get and install Mpack rpm/deb
>>>>>>> packages
>>>>>>>> in their own Ambari cluster.
>>>>>>>> 
>>>>>>>> The VOTE:
>>>>>>>> [ ] +1, Agree to put Mpack as a standalone component.
>>>>>>>> [ ] +0, I don't care either way.
>>>>>>>> [ ] -1, do *not * Agree.
>>>>>>>> 
>>>>>>>> Look forward to your comments and feedback, thanks.
>>>>>>>> 
>>>>>>>> BRs,
>>>>>>>> Yuqi
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> IRC: ganeshraju@#linaro on irc.freenode.ne <http://irc.freenode.net/
>>>> t
>>>>> 
>>>>> 
>>>> 
>>> 

Re: [DISCUSS] [VOTE] Release Ambari-Mpack as a seperate top component in Bigtop

[Olaf Flebbe]

Hi,

Sorry, I am super lost about the implications of that vote.

Why do we need a vote in the first place?
I am familiar with votes for additional committers, PMC additions and releases. 
For these votes we have the concept of binding votes by PMC members.

Why should I care about this ?
Does it have legal implications?

Who will maintain this ? 
I am a bit surprised that we do anything with ambari, since we not even have a 
maintainer for it
grep ambari MAINTAINERS.txt -> nothing

What problem/case is this additional component trying to solve?
If you are asking for a vote I would like to have a more elaborate explanation. 
I looked at the linked ticket: Sorry I am totally lost about what is this 
about. If you would have simply done it, I wouldn’t have cared.

As I understand it should somehow enable more contributions. 
Really? MPack is a amabari only technology, that seems not to even be defined 
properly. How could this ever end in more contributions to Bigtop ?

What will happen if someone is -1 on this vote ?

If a contributor cares about adopting Bigtop to new business cases, that’s very 
fine to me. Please go ahead. But please don’t ask for a vote. All committers 
still have the chance to reject patches. 

Generally I am concerned that we care about an (from Bigtop side) unmaintained 
project while neglecting our core infrastructure and packages at the same time.
1) The ci.bigtop.apache.org certificate expired again
2) The bigtop-trunk-packages do not reflect the architectures we ought to 
support (ubuntu-18.04 for instance) 
3) Trunk-packages for amd64 looks like garbage
4) Still no-one working on Hadoop3
5) CI needs to be redone (with pipeline jobs) .
IMO we might have wrong priorities.

Best,
Olaf




> Am 04.06.2020 um 19:04 schrieb Ganesh Raju :
> 
> +1
> 
> On Thu, Jun 4, 2020 at 12:01 PM Matt Andruff 
> wrote:
> 
>> +1
>> 
>> On Thu., Jun. 4, 2020, 02:35 Yuqi Gu,  wrote:
>> 
>>> Hi folks,
>>> 
>>> As the discussion with Evans Ye, Jun He, and Matt Andruff (from Ambari)
>> on
>>> https://github.com/apache/bigtop/pull/555#issuecomment-635228180:
>>> 
>>> For users / Bigtopers could easily modify Mapck and make contributions to
>>> it like bugfix and adding more services,
>>> How about to decouple Mpack from Ambari and put Mpack as a standalone
>>> component in Bigtop, like *bigtop-packages/src/common/bigtop-mpack*?
>>> If so, it's convenient for users to get and install Mpack rpm/deb
>> packages
>>> in their own Ambari cluster.
>>> 
>>> The VOTE:
>>> [ ] +1, Agree to put Mpack as a standalone component.
>>> [ ] +0, I don't care either way.
>>> [ ] -1, do *not * Agree.
>>> 
>>> Look forward to your comments and feedback, thanks.
>>> 
>>> BRs,
>>> Yuqi
>>> 
>> 
> 
> 
> -- 
> IRC: ganeshraju@#linaro on irc.freenode.ne t

Re: Bigtop Hadoop 3.2.1 error

[Olaf Flebbe]

Hi

Thank you for you questions!



> Am 06.06.2020 um 10:41 schrieb Jagat Singh :
> 
> Hi,
> 
> I am trying to use Bigtop to create deb file for Hadoop 3.2.1. I am using 
> Ubuntu 18.04. 
> 
> In the process of doing this, many questions came to my mind which I wanted 
> to learn about.
> 
> 1) I changed the bom file and ran gradle hadoop-deb command. Is this the 
> correct process to create any deb or rpm?

As a Bigtop User: no . You are not expected to roll your own Distribution .
As a Developer: yes somehow, that’s usually the first try to upgrade a package. 
Since you already read the instructions: I like to stresst that you are 
supposed to use the docker containers (for git trunk 
bigtop/slaves:trunk-ubuntu-18.04 and run the gradle command within the 
container.
 

> 2) In what state you will use patches present in folder common/src/hadoop/ , 
> is it only usable when upstream system make additional changes after making a 
> release?

We try to use zero patches but if we find defects which are not fixed in the 
version we want to package, than we do patches. Or the build system is to be 
adapted to our needs… Usually we upstream fixes to the original project unless 
it is very bigtop specific. Most of the time we cherry-pick fixes from 
unreleased versions of the package.

> 3) Many times the build fails in between and I had to clean up build and 
> output folder both and restart. What is the process to pick from where is 
> failed to save time?

No way, unfortunately. Neither deb nor rpm packaging have a way to resume where 
it failed.

When I developed packaging I usually trying to do manually do the build, 
dependencies and try to automate stuff in the build scripts. This is very time 
consuming, indeed. 

Now you are hitting the ugly part. The upstream Hadoop project decided to 
rework all supporting start scripts, introduced different processes and 
frameworks with Hadoop3 , breaking our complete Hadoop setup. 

I did some initial work last year for Hadoop-3.1.2 in the "bigtop-alpha“ git 
branch and dropped out of the project because of lack of time and changed 
priorities.

If you like to invest into hadoop-3.2.1 I would recommend to look into this and 
merging with current master first.


> 4) Follow-up for 3), Building the hadoop and building deb for hadoop are two 
> different things, what command can be used to do one after the another 
> manually to save time in case one fails.
> 

Right, Building Hadoop should be straight forward, but packaging the new Hadoop 
3.2 layout is hard, since Hadoop project decided to change to a monolithic 
approach. Without rewriting nearly all scripts it is not possible anymore to 
start daemons independently any more.

I would recommend to do a manual installation first and start fresh with 
packaging the individual parts.



> > Task :hadoop-deb FAILED
> dpkg-source: warning: extracting unsigned source package (hadoop_3.2.1-1.dsc)
> dpkg-source: error: unpack target exists: hadoop-3.2.1
> dpkg-source: info: extracting hadoop in hadoop-3.2.1
> 
> 4) How does Bigtop ensures compatability between products, example Hive is 
> compiled with Hadoop version 3.2.1. Based on my understanding in the mvn -D 
> commands there is an override and version it takes from the bom file the 
> version information. Is this understanding correct? Do I still need to change 
> any maven pom file for any software to ensure compatibility?

You are right that is the way bigtop currently works. The crucial part is that 
it uses the local maven repository to pick up previous artifacts. (Dependency 
does "mvn install“ and mvn package of an package will pick up the previous 
compiled artifact from ~/.m2/repository ) . 

Additionally, some crucial parts like Hadoop or mapreduce jars are symlinked in 
packages so they are only installed once on the target system.

I was proposing to not do this any more since we might break API’s that way 
(and we had that) . My approach in bigtop-alpha was to use the dependencies the 
devs chose and to accept different versions of a jar in the system, depending 
on the *network* API rather the *Java* API. That would allow us to compile 
packages independently.
 

> 5) Are there any more resources to learn about Bigtop other than 
> https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+Packaging 
> . What 
> should I do next to learn more about Bigtop

Best resources are questions. We will do our best to answer. 
It might help if you look into the jobs behind the scenes in  
ci.bigtop.apache.org  .  Please create an account 
there and I am happy to give you read access to the configuration of 
bigtop-trunk-packages for instance

 

> 6) Beside installing deb on my machine and playing with it, what else tests 
> are needed can be done to ensure it works as intended.

I think Evans can answer this best.  We have docker deployments, 

> 7) For Hadoop 3.1.2

Re: [DISCUSS] Next Generation Bigtop CI Infra

[Olaf Flebbe]

Hi, 

have some limited experience with Jenkins2 aka Pipeline. IMO this resolves 3) 
quite nicely.

Regarding 2) imo our deployment with docker compose is dead-end. Docker is 
simply not designed to support whole machine scenarios white systemd as a init 
process and predictive networking. while lxc seems to be more suitable, I think 
we should not hardcode a toy/test environment with lxc. Would propose Terraform 
here, that ideally should enable our users to use other infra as well.

Regarding 1) If we have Terraform for 2) we can even manage our own build infra 
with that (minus the ppc64le and arm64 slaves).

What do you think?

Olaf


Von meinem iPad gesendet

> Am 26.04.2020 um 13:58 schrieb Evans Ye :
> 
> Hi folks,
> 
> I've been thinking about the revamp of bigtop's CI infra.
> Currently what we have is purely a Jenkins cluster with manual managed:
> 1. hardware/machine resource
> 2. build env: docker, docker compose are all manual installed
> 3. job definition: all jobs are manual created and the definition is not
> tracked w/ version control system.
> 
> I think for 2,3 there're some tools/solutions that can better manage and
> automate the things. So that user can just take the code and build their CI
> directly.
> 
> At Yahoo! there's a tool called screwdriver[1] which is a framework to do
> CICD, however I don't think it fits our scenario well as we're not facing
> production CD. Probably Jenkins 2.0 is more suitable? Anyone has experience
> and suggestions?
> 
> [1] https://screwdriver.cd/
> 
> Evans

Test your account

[Olaf Flebbe]

Hi Masatake,

please test your account by adding yourself to the member page (Like
BIGTOP-2622).

You should be able to update the page (see
https://cwiki.apache.org/confluence/display/BIGTOP/Deploying+Bigtop%27s+Apache+Website)

Best,
Olaf

New Bigtop PMC member: Kengo Seki

[Olaf Flebbe]

On behalf of the Apache Bigtop Project Management Committee, I am pleased
to announce that Kengo Seki has accepted our invitation to join the
Bigtop PMC.

Please join me in congratulating Kengo!

Regards,
  Olaf  (on behalf of the ASF BigTop PMC)

New Committer: Masatake Iwasaki

[Olaf Flebbe]

The Project Management Committee (PMC) for Apache Bigtop has asked
Masatake Iwasaki to become a committer and we are pleased to announce 
that he has accepted!

Being a committer enables easier contribution to the project since there is
no need to go via the proxy patch submission process. This should enable better
productivity.

Apache Bigtop practices CTR (commit-then-review) development process which
means that you can checkin the code without a +1 from a committer. This
doesn't mean that you're encouraged to do so. Instead, we strongly recommend 
you to
seek for feedback before you commit. Please read the document before you
act:

* https://github.com/apache/bigtop#ctr-model

If in doubt, exercise your best judgement and don't hesitate to ask
questions on the dev@bigtop.apache.org mailing list.

We're happy to have you on board and looking for many more contributions to
come. Please join me in congratulating Masatake Iwasaki !

Regards,
  Olaf  (on behalf of the ASF BigTop PMC)

Re: Release-1.4.0

[Olaf Flebbe]

Thanks Luca, Evans.

Release Tags are pushed and two commits to branch-1.4 as well.

Olaf

Am 08.03.20 um 13:52 schrieb Luca Toscano:
> Thanks a lot Olaf!
> 
> Il giorno dom 8 mar 2020 alle ore 10:21 Evans Ye 
> ha scritto:
>>
>> It seems that 1.3.0 does not have the final release tag as well.
>> I think make it consistent by adding 1.4.0 and 1.3.0 release tag would be a
>> great.
>> Thanks for discovering this.
>>
>> Evans
>>
>> Olaf Flebbe  於 2020年3月8日 週日 下午5:08寫道：
>>
>>> Hi,
>>>
>>> since Luca Toscano asked, I am considering to add two bugfixes to the
>>> 1.4 branch with BIGTOP-3308.  I was checking tags first:
>>>
>>> We have
>>> release-1.4.0-RC0
>>> release-1.4.0-RC1
>>> but no
>>> release-1.4.0
>>> tag
>>>
>>> Is it ok to push the "release-1.4.0" tag to latest commit on branch-1.4?
>>>
>>> Additionally I would add the needed commits to it as well ?
>>>
>>> I am not considering a bugfix release, though.
>>>
>>> WDYT ?
>>>
>>> Best
>>>   Olaf
>>>
>>>
>>>

[jira] [Created] (BIGTOP-3323) Fix hadoop pom for 1.4 branch

[Olaf Flebbe (Jira)]

Olaf Flebbe created BIGTOP-3323:
---

 Summary: Fix hadoop pom for 1.4 branch
 Key: BIGTOP-3323
 URL: https://issues.apache.org/jira/browse/BIGTOP-3323
 Project: Bigtop
  Issue Type: Task
Affects Versions: 1.4.0
Reporter: Olaf Flebbe
Assignee: Olaf Flebbe


Hadoop does not compile any more since it is using an insecure repository.
Not necessary on 1.5.0




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (BIGTOP-3322) Lintian Warnings in Hadoop

[Olaf Flebbe (Jira)]

Olaf Flebbe created BIGTOP-3322:
---

 Summary: Lintian Warnings in Hadoop
 Key: BIGTOP-3322
 URL: https://issues.apache.org/jira/browse/BIGTOP-3322
 Project: Bigtop
  Issue Type: Task
Reporter: Olaf Flebbe


Hi [~iwasakims], 
I discovered some lintian warnings while compiling hadoop-kms:

For instance 
{code}
W: hadoop-kms: maintainer-script-needs-depends-on-adduser preinst
{code}

Maybe you want to review these messages.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Release-1.4.0

[Olaf Flebbe]

Hi,

since Luca Toscano asked, I am considering to add two bugfixes to the
1.4 branch with BIGTOP-3308.  I was checking tags first:

We have
release-1.4.0-RC0
release-1.4.0-RC1
but no
release-1.4.0
tag

Is it ok to push the "release-1.4.0" tag to latest commit on branch-1.4?

Additionally I would add the needed commits to it as well ?

I am not considering a bugfix release, though.

WDYT ?

Best
  Olaf

Updated the ci certificate

[Olaf Flebbe]

.. of ci.bigtop.apache.org since it was outdated.

See
https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+CI+Setup+Guide
(last paragraph)

Does anybody know have an idea why we don't receive email notifications
from letsencrypt any more ?

Best
  Olaf

[jira] [Created] (BIGTOP-3318) Use secure repo in bigtop-tests

[Olaf Flebbe (Jira)]

Olaf Flebbe created BIGTOP-3318:
---

 Summary: Use secure repo in bigtop-tests
 Key: BIGTOP-3318
 URL: https://issues.apache.org/jira/browse/BIGTOP-3318
 Project: Bigtop
  Issue Type: Task
Reporter: Olaf Flebbe






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: CI has to be updated

[Olaf Flebbe]

Hi,

Updated jenkins to latest, updated all but one plugin to latest, removed 
the copy to slave plugin (was inactive before) since it is insecure and 
unmaintained.


The slaves need some love too, maybe someday later.

Best,
Olaf

Am 20.02.20 um 07:18 schrieb Evans Ye:

Sorry I'm busy but you can go ahead for sure. I'll check back with you.

Olaf Flebbe  於 2020年2月20日 週四 上午6:13寫道：


Hi Evans,

do you have time to refresh the CI docker image? Seems like jenkins
urgently has to be updated. In case you don't have time I can try to
find half an hour in the next days.

Olaf

CI has to be updated

[Olaf Flebbe]

Hi Evans,

do you have time to refresh the CI docker image? Seems like jenkins 
urgently has to be updated. In case you don't have time I can try to 
find half an hour in the next days.


Olaf

Re: Bigtop error on Centos 7

[Olaf Flebbe]

hi

the failed downloads are http:// downloads

http download from nexus are now turned off for security reasons.

there was another report for oozie with the same problem on this list iirc.

imo this can only be solved by using additional tooling, smthg like a proxy 
changing http to https . nexus might have a property for that. or new maven 
might have smthg in place for working around broken dependencies, but i am 
speculating

best
olaf



> Am 24.01.2020 um 05:47 schrieb Konstantin Boudnik :
> 
> Looks like some sort of trouble with one of the dependencies of httpfs:
> 
> 
> Downloading from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/maven/skins/maven-default-skin/1.0/maven-default-skin-1.0.jar
>  
> Downloading from repository.jboss.org: 
> http://repository.jboss.org/nexus/content/groups/public/org/apache/maven/skins/maven-default-skin/1.0/maven-default-skin-1.0.jar
>  
> [WARNING] Checksum validation failed, expected  but is 
> da39a3ee5e6b4b0d3255bfef95601890afd80709 from repository.jboss.
> org for 
> http://repository.jboss.org/nexus/content/groups/public/org/apache/maven/skins/maven-default-skin/1.0/maven-default-skin-1.0.jar
>  
> [WARNING] Could not validate integrity of download from 
> http://repository.jboss.org/nexus/content/groups/public/org/apache/maven/skins/maven-default-skin/1.0/maven-default-skin-1.0.jar:
>  Checksum validation failed, expected  but is 
> da39a3ee5e6b4b0d3255bfef95601890afd80709
> [WARNING] Checksum validation failed, expected  but is 
> da39a3ee5e6b4b0d3255bfef95601890afd80709 from repository.jboss.org for 
> http://repository.jboss.org/nexus/content/groups/public/org/apache/maven/skins/maven-default-skin/1.0/maven-default-skin-1.0.jar
>  
> Downloaded from repository.jboss.org: 
> http://repository.jboss.org/nexus/content/groups/public/org/apache/maven/skins/maven-default-skin/1.0/maven-default-skin-1.0.jar
>  (0 B at 0 B/s)
> 
> And then build fails right at that module. Doesn't look like a Bigtop issue, 
> methink. I am on the road right now with somewhat limited internet, will be 
> back over the weekend and try to look more. Perhaps in the meanwhile someone 
> on the list would have any other ideas.
> 
> --
>   Cos
> 
>> On 1/23/20 11:40 PM, Konstantin Boudnik wrote:
>> Well, I am not familiar with Gradle enterprise - looks like it has 
>> everything indeed, thanks! Let me dig and see what I can figure out.
>> 
>> -- 
>>   Cos
>> 
>>> On 1/23/20 11:28 PM, Alexander Frolushkin wrote:
>>> Thank you for responding, Konstantin.
>>> 
>>> Honestly, I was thinking --scan gradle will publish all necessary
>>> information about its execution, which may be required to inverstigate the
>>> problem.
>>> 
>>> Anyway, I'm ready to provide maximum data for this dicsussion. Firstly,
>>> here is a complete console output of entire hadoop-pkg-ind task till the
>>> error -
>>> https://drive.google.com/file/d/16zmFTj485RQ44BFMh5zlhB4cyUzmyaNR/view?usp=drivesdk
>>>  
>>> 
>>> If I need to collect additional data, please direct me, what exactly I need
>>> to do to get it for processing here.
>>> 
>>> I really want to build hadoop repo by bigtop-1.4.0.
>>> 
 On Thu, Jan 23, 2020, 22:07 Konstantin Boudnik  wrote:
>>> 
 Thanks, Alexander, for bringing this up.
 
 I don't see anything wrong with the way you're starting the build,
 except that official CI is now using "hadoop-pkg-ind" target and build
 then pulls in a correct container automatically. But I see you already
 tried that one.
 
 As a side note, I am looking at [1] and see the following error (which
 seems to be there for some time):
 
 Plugin [id: 'de.undercouch.download', version: '3.2.0'] was not found in
 any of the following sources:
 
 - Gradle Core Plugins (plugin is not in 'org.gradle' namespace)
 - Plugin Repositories (could not resolve plugin artifact
 'de.undercouch.download:de.undercouch.download.gradle.plugin:3.2.0')
 Searched in the following repositories:
   Gradle Central Plugin Repository
 
 Unfortunately, it is hard to tell what kind of error you're getting in
 your build. Any chance you can publish a snippet of the actual logs,
 rather than a screenshot?
 
 -- 
 Thanks,
 Cos
 
 [1]
 
 https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/COMPONENTS=hadoop,OS=centos-7/567/console
  
 
 On 1/23/20 4:50 AM, Alexander Frolushkin wrote:
> Hello everyone!
> 
> Sorry for repeating this in dev list, but I really need help in this
> situation.
> 
> On bigtop-1.4.0 build fails on hadoop-hdfs-httpfs with error:
> 
> Failed to execute goal
> 
 org.apache.maven.plugins:maven-project-info-reports-plugin:2.7:dependencies
  
> (default) on project hadoop-hdfs-httpfs: An error has occurred in
> Dependencies report generation.: zip file is empty -> [Help 1]

Re: Happy new year!

[Olaf Flebbe]

Hi ,

happy to see some progress!
Wish you all great twenties!

olaf

> Am 31.12.2019 um 21:35 schrieb Evans Ye :
> 
> Hi folks,
> 
> Another great year at Apache Bigtop. I'm glad for being one of the
> community member.
> 
> This year we have a big initiative: cloud native bigtop. I'm very excited
> to see the upcoming first release of it. In the meantime, we also have
> other folks keep advancing the core bigtop distribution. Thank you all &
> happy new year!
> 
> Evans

Re: [DISCUSS] Target distros on the 1.5.0 release

[Olaf Flebbe]

hi

one source of problems was using puppet-forge for instance for puppet-stdlib 
and puppet-apt, since they require rather new versions. look out for 'puppet 
module install '.  While all distros using apt do have matching prepackaged 
versions in their repository. 

other was different search paths of all these versions. we never fixed that 
consistently.

olaf

Von meinem iPad gesendet

> Am 21.11.2019 um 03:43 schrieb Jun HE :
> 
> I'm fine with the new distros list. Just one concern about the puppet
> recipes compatibility across multiple puppet versions (3.8.5 for
> Ubuntu-16.04, 4.8.2 for Debian-9, and 5.x for other new distros). I didn't
> do any investigation yet. If such issues arise, I'll vote for drop distros
> with older puppet.
> 
> Evans Ye  于2019年11月21日周四 上午1:51写道：
> 
>> Fine by me for the OS side.
>> What do you think about the components? Is there a list of components you'd
>> like to upgrade?
>> We can target a subset of current supported matrix as we previously
>> discussed about this and the community was lean to the direction of having
>> important component better supported instead of spending resources for
>> 20~30 components.
>> 
>> Youngwoo Kim (김영우)  於 2019年11月20日 週三 上午9:41寫道：
>> 
>>> Kengo,
>>> 
>>> Looks good to me. I think puppet on CentOS 8 would be fine.
>>> 
>>> On Cloud Native Bigtop, I believe we should consider that components as a
>>> 'contrib' at this point.
>>> I'm considering about Jay's idea, making 'CNB' on master as a contrib
>>> module. A development branch is good but on our "two-tracks" development,
>>> 'contrib' module will be easier for us to maintain traditional distros
>> and
>>> cnb.
>>> 
>>> Thanks,
>>> Youngwoo
>>> 
 On Wed, Nov 20, 2019 at 9:28 AM Kengo Seki  wrote:
>>> 
 Hi folks,
 
 I'd like to discuss the target distros for the next 1.5.0 release [1],
 because over 1.5 years have passed since Ubuntu 18.04 was released
 and the next LTS will be released within half a year. In addition,
 Fedora 26 and openSUSE 42.3 have already been EOL'd.
 
 (I understand the "Cloud Native Bigtop" project is going on
 and am really looking forward to it, but my customers still requires
 the traditional software stack :)
 
 Based on the past discussion [2], here's my proposal:
 
 - Add Debian 10, Fedora 31 and Ubuntu 18.04 as the target distros
  and use the puppet package provided by each distro, so that
  we can support all CPU architectures (x86_64, aarch64, and ppc64le).
  Their puppet versions are 5.4.0 (ubuntu) and 5.5.10 (debian and
>>> fedora).
 
  Keep Debian 9 and Ubuntu 16.04 since they are still in the support
 period.
 
  Drop Fedora 26 since it has reached to the EOL on 2018-05-29.
 
 - Add CentOS 8. Unfortunately, that version doesn't seem to
  provide the distro's puppet package, even including EPEL.
  Even though, I'd like to support it since that distro
  (and RHEL8) are widely used especially in enterprise systems.
  So, as the next best option, how about using Puppet 5.5 provided by
  Puppetlabs and only supporting the x86_64 architecture on this
>> version?
 
  Keep CentOS 7 since it's still in the support period.
 
 - Drop openSUSE 42.3 since it has reached to the EOL on 2019-07-01
  and don't add a new version of that distro, as discussed in [2].
 
 To summarize the above, the supported distros and their versions
 in the 1.5.0 release are as follows:
 
 - CentOS 7, 8 (8 is only supported on x86_64)
 - Debian 9, 10
 - Fedora 31
 - Ubuntu 16.04, 18.04
 
 Does this sound reasonable? I'd appreciate any comments or suggestions.
 
 (Honestly, I'd actually like to drop CentOS 7, Debian 9, and Ubuntu
>>> 16.04,
 so that we can consolidate the Puppet version to 5.x.
 But it may be too aggressive for users.)
 
 [1]: https://issues.apache.org/jira/browse/BIGTOP-3123
 [2]:
 
>>> 
>> https://lists.apache.org/thread.html/26e14cf36e9cfd61e0de581ed83bf305565c2e65234f1ce3bfb97628@%3Cdev.bigtop.apache.org%3E
 
 Kengo Seki 
 
>>> 
>> 

Re: During debuild, 'dh_strip_nondeterminism' can hang for about 6 hours?

[Olaf Flebbe]

Let me tell you I had similar issues when I worked on hadoop3 on amd64.

As a workaround i disabled this step. 

olaf

> Am 18.09.2019 um 09:19 schrieb Guodong Xu :
> 
>> On Mon, Sep 16, 2019 at 4:15 PM Youngwoo Kim (김영우)  wrote:
>> 
>> Hi Guodong,
>> 
>> It looks strange... I've never seen this before on x86.
>> Did you look into our CI?
>> https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/
> 
> 
> There is no timestamp in logs in CI. So I cannot tell how much time cost
> between each line of log output.
> 
> I tried to run bigtop build on an x86 server machine, and so far I see no
> issues. I will further investigate.
> 
> -Guodong
> 
> 
> 
>> 
>> 
>> Thanks,
>> Youngwoo
>> 
>> 
>>> On Wed, Sep 11, 2019 at 3:45 PM Guodong Xu  wrote:
>>> 
>>> Hi, all
>>> 
>>> Did anybody noticed this? When I do ./gradlew deb inside of docker
>>> container, as part of 'debuild', 'dh_strip_nondeterminism' can take as
>> long
>>> as 6 hours to finish.
>>> 
>>> Here is an example, the timestamp is my local system time, inside a
>> docker
>>> bigtop/slave container, run './gradlew phoenix-deb',
>>> 'dh_strip_nondeterminism' started at 08:36 in the morning, however, it
>>> hangs there until 14:00 of the afternoon. Here is log:
>>> 
>>> 08:36:13dh_strip_nondeterminism
>>> 14:00:47 Using 1568161110 as canonical time
>>> 14:00:47 Normalizing
>>> debian/phoenix/usr/lib/phoenix/phoenix-4.13.1-HBase-1.3-queryserver.jar
>>> 14:00:47 Normalizing
>>> debian/phoenix/usr/lib/phoenix/phoenix-core-4.13.1-HBase-1.3.jar
>>> 14:00:47 Normalizing
>>> debian/phoenix/usr/lib/phoenix/phoenix-core-4.13.1-HBase-1.3-tests.jar
>>> 
>>> I am building on an Arm machine. Does anybody know whether there is the
>>> same problem in x86 servers?
>>> 
>>> -Guodong Xu
>>> 
>> 

Re: RFC: Building .tar.gz binary packaging for all components in Bigtop

[Olaf Flebbe]

Hi,

Thanks.

Olaf



Am 05.08.19 um 04:07 schrieb Roman Shaposhnik:

On Sun, Aug 4, 2019 at 9:42 AM Evans Ye  wrote:


It is because your input is valuable so we want to take that into
consideration seriously. I personally like constructive discussions even if
it doesn't look good. It is actually the beauty of the community. And I'd
like to emphasize that all the existing members have gone through a history
of contribution which made Bigtop what it is now. At Apache merit does not
expire. This is important. So welcome back anytime.


+1! (this is the reason I still hang around although my day job is as
far from bigdata and Hadoop as it can possibly be)

Thanks,
Roman.

Re: RFC: Building .tar.gz binary packaging for all components in Bigtop

[Olaf Flebbe]

Hi,

out of Bigtop. At least until further notice.

My move is not to stop anyone to do smthg. I cannot and I will not stop 
it. I am missing contributions. There is no need to raise opinions over 
and over again. Just Do it! If someone would have contributed it 
instead, it would have been in for ages.


Olaf



Am 29.07.19 um 23:54 schrieb Konstantin Boudnik:

Hey Olaf.

Sorry, not sure how to read this? Out of the Bigtop or this silly commotion
with tar.gz? Hopefully, just the latter one ;)

Cos

On Mon, Jul 29, 2019 at 10:21PM, Olaf Flebbe wrote:


I am out of this project now. Do whatever you like.

Olaf

Re: RFC: Building .tar.gz binary packaging for all components in Bigtop

[Olaf Flebbe]

Hi,


1. People are using docker for packaging and deployment instead of rpm/deb.


You can package everything as a container, but it may or may not make sense.


2. Hadoop 3's messy layout don't fit into rpm/deb conventions. Maybe it's
because they think it's the right design for the container era.


There is nothing prepared to run hadoop in a container. BTW it would be 
messy since yarn is supporting compute loads to run in a container. That 
would mean a nested container runtime. Feasible but unnecessary mind 
twisting.


Hadoop3 does fit in rpm/deb, one only would have to change the packaging 
layout to a monolith approach. Can be done, not nice, but feasible.


I am exhausted in explaining the points over and over again. Go and try, 
I tried it. For a secure hadoop service you need a known IP address, and 
a FQDN hostname resolution. You will find ton's of docker bug reports 
like https://github.com/moby/moby/issues/14282 or 
https://github.com/moby/moby/issues/29100 only to pick up two. I used 
lxc to demonstrate setting up a secure hadoop environment at Apache 
Conference in Budapest for a reason.


I am out of this project now. Do whatever you like.

Olaf

Re: [DISCUSS] Roadmap for the next release.1.5? 2.0?

[Olaf Flebbe]

Hi

hadoop changed their scripts to break when not absolutely everything is in 
place.

The start scripts for yarn for example expect to all jars for hdfs, yarn and 
mapreduce to be present. They renamed (to be precise deprecated the old name) 
of environment variables in a way it looks more convenient, but it will mix up 
hadoop subprojects. we cannot use independent /run and /var dirs as the debian 
build rules and some other unwritten rules of linux packaging demand in order 
to be installed independently.

So hadoop has tied everything together. without rewriting their build scripts 
it is not possible to have yarn without hdfs or hdfs without yarn to be 
installed independently. 

Olaf

ps: The way pid files are written confuses systemd (or only me). effectively 
systemd somehow doesnt properly detect the state of the daemons.

Von meinem iPad gesendet

> Am 25.07.2019 um 02:11 schrieb Konstantin Boudnik :
> 
> Olaf, thanks for trying... this could be exhausting, for sure ;(
> 
> Just to clarify: when you say "monolithic chunk" doesn't it mean that it
> relies heavily on relative paths or something of the sort and couldn't be
> broken into pieces because... well, it is so broken?
> 
> Thanks,
>  Cos
> 
>> On Mon, Jul 22, 2019 at 10:34PM, Olaf Flebbe wrote:
>> Hi,
>> 
>> If I would have only known that google document before ... I am
>> asking myself if we should link it from confluence.
>> 
>> + hadoop 3.
>> As you may know, I worked on it (bigtop-alpha branch), got it
>> compiled and packaged. However, while testing, I discovered that
>> Hadoop project built on top of some assumptions which do not hold
>> true for current Bigtop. One of them is that Hadoop 3 is to be
>> installed as a monolithic chunk on a filesystem. This is not what I
>> understand as integrated into a Linux distribution. Other point is
>> that hadoop's installation proposal does not follow the LSB:
>> platform specfic libs are in "share" directories. I am exhausted,
>> this is so broken.
>> 
>> I will stop here ... for a reason.
>> 
>> Olaf
>> 
>> 
>> 
>> 
>> 
>>> Am 17.07.19 um 09:09 schrieb Youngwoo Kim (김영우):
>>> Hi folks,
>>> 
>>> After 1.4.0 release, there is no discussion for the next release yet. so I
>>> believe we need to share the ideas and prioritize the items for
>>> development.
>>> 
>>> And also https://issues.apache.org/jira/browse/BIGTOP-3123 and
>>> https://docs.google.com/document/d/1F2Gxu8GARQDZXgqHn12LKkQ5wCV_AF4b_tVmjYB6YfA/edit
>>> are good starting point for this discussion.
>>> 
>>> My personal preferences are:
>>> - Distribution based on Hadoop 3
>>> - Up-to-date BigPetStore
>>> - Software stacks and framework for streaming data & Machine Learning
>>> - Containers and Cloud Native: What? How?
>>> 
>>> It would be great to hear your thoughts.
>>> 
>>> Thanks,
>>> Youngwoo
>>> 

Re: [DISCUSS] Roadmap for the next release.1.5? 2.0?

[Olaf Flebbe]

Hi,

If I would have only known that google document before ... I am asking 
myself if we should link it from confluence.


+ hadoop 3.
As you may know, I worked on it (bigtop-alpha branch), got it compiled 
and packaged. However, while testing, I discovered that Hadoop project 
built on top of some assumptions which do not hold true for current 
Bigtop. One of them is that Hadoop 3 is to be installed as a monolithic 
chunk on a filesystem. This is not what I understand as integrated into 
a Linux distribution. Other point is that hadoop's installation proposal 
does not follow the LSB: platform specfic libs are in "share" 
directories. I am exhausted, this is so broken.


I will stop here ... for a reason.

Olaf





Am 17.07.19 um 09:09 schrieb Youngwoo Kim (김영우):

Hi folks,

After 1.4.0 release, there is no discussion for the next release yet. so I
believe we need to share the ideas and prioritize the items for
development.

And also https://issues.apache.org/jira/browse/BIGTOP-3123 and
https://docs.google.com/document/d/1F2Gxu8GARQDZXgqHn12LKkQ5wCV_AF4b_tVmjYB6YfA/edit
are good starting point for this discussion.

My personal preferences are:
  - Distribution based on Hadoop 3
  - Up-to-date BigPetStore
  - Software stacks and framework for streaming data & Machine Learning
  - Containers and Cloud Native: What? How?

It would be great to hear your thoughts.

Thanks,
Youngwoo

Re: RFC: Building .tar.gz binary packaging for all components in Bigtop

[Olaf Flebbe]

Hi

Maybe we have different target groups: For educational users, trying to figure 
out how everything works together, the single computer cluster is great.

This Tar.gz artefact were never ever been sufficient to run in production. They 
did not contain 64bit libs for instance. They do not provide start scripts, 
error prone when you accidently using the wrong java, do not place logfiles in 
suitable places and so on. 

This tar artefacts and instructions are POC quality, that's it.  

Please be aware that Bigtop changes directory layout, directory permissions, 
configuration and startup of the original code in order to support large scale 
installations and automation.

We have a much better alternative: Distribution integrated repositories. Youll 
only have to apt/yum install hadoop-hdfs-datanode and everything is already 
setup, including java, runscripts, directory layout and users. Then you can 
concentrate on configuring that beast. You do not even need no special 
instruction, since it is distribution native -- well, almost. If you feel that 
bigtops runscripts / layout is not suitable , you are very welcome to 
contribute. 

If i look at the instructions [1] . they contain tons of settings you should 
do. With our packages you can actually do exactly this, without bothering to 
install and configure all the dependencies. 

Olaf





Von meinem iPad gesendet

> Am 18.07.2019 um 08:57 schrieb Guodong Xu :
> 
> Hi, Evans
> 
> Comments in below.
> 
>>  On Tue, Jul 16, 2019 at 10:46 PM Evans Ye  wrote:
>> 
>> I'm not objecting this, but I'd love to have more discussion to figure out
>> whether this is the right thing to do. What I get from your proposal is
>> users want to do things which RPM/DEB don't do well while tar.gz is good at
>> it. However is it able to do it in another way which is far more beneficial
>> in more scenarios? In general, it's like when users are asking for a faster
>> horses, can we come up with cars?
>> 
>> How about we start with the first step which is to elaborate why users
>> choose to go for tar.gz?
> 
> 
> Right, agree. Here is what I learned from users of Arm servers:
> 
> One background for this is, currently, CDH and Hortonworks both have no
> official release for Arm server yet. So, Bigtop is the only available and
> verified distribution to them. As you know, with effort from the community
> and Arm Inc., Linaro, Bigtop now has officially supported Arm64.
> 
> To these users, before they start to use Bigtop on Arm, they are already
> familiar with each component's individual installation and usage on x86.
> Most of them are released in .tar.gz format. (i.e. Most apache big data
> component doesn't release in deb/rpm. So, if we tell users that the only
> available format in Bigtop is deb/rpm, this just hesitates them).
> 
> Eg. For Hadoop, the official site for installation is here [1] and here
> [2], their release format is .tar.gz.
> 
> So,
> 1. using .tar.gz is the minimum effort route for them to start their touch
> with Bigtop (if we can support .tar.gz).
> 2. Bigtop has all components tested. That provides very big confidence to
> the users. So they like to use the binary built from Bigtop on Arm64
> servers.
> 
> [1]
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html
> [2] https://www.apache.org/dyn/closer.cgi/hadoop/common/
> 
> -Guodong
> 
> 
>> 
>> Guodong Xu  於 2019年7月16日 週二 下午12:24寫道：
>> 
>>> Hi, all
>>> 
>>> This is a request for comments for a potential new feature. Please have a
>>> look and let me know whether this adds value to the community. Appreciate
>>> your opinions on this.
>>> 
>>> As most of you know already, so far, Bigtop provides two packaging
>> format:
>>> deb and rpm. Users are recommended to install through apt and yum for
>> their
>>> respective linux distributions.
>>> 
>>> Yet, there are still users coming to me and asking for the more
>> traditional
>>> .tar.gz packages. I know that shouldn't be a problem for x86 users, since
>>> they can always download .tar.gz bin releases from each component's
>>> official release website. But for users from other architectures, like
>>> Arm64 and powerpc, story is different. .tar.gz are not readily available
>> to
>>> these architectures.
>>> 
>>> In Bigtop, we are doing the job of building/packaging/testing big data
>>> components in one single place. So, it makes sense that we step in, and
>>> during our process of building each component, we keep a binary .tar.gz
>>> copy for each component we supported.
>>> 
>>> If you are supportive to the above idea, then next step will be how to
>>> achieve that. I did some research actually. So, my method is to do two
>>> things:
>>> 0. this can be a subsequent task of either $target-deb or $target-rpm. It
>>> depends on one of them two.
>>> 1. add tar and cp into each component's 'do-component-build' file.
>>> 2. add a new set of packaging tasks for each component. This means to
>> add a
>>> new task in genT

Hadoop-3

[Olaf Flebbe]

Hi,

in case anyone interested: I got Hadoop-3.1.2 somehow packaged for 
debian-9. Still a ton of configuration files and shortcuts to be adapted.


It sits in Branch bigtop-alpha.

Trying to reproduce the build with
https://ci.bigtop.apache.org/job/Bigtop-alpha/

I am not sure for how long I can continue to work on this side project.

Best,
Olaf

Re: [Bigtop] Some question about build Hadoop on AArch64

[Olaf Flebbe]

Hi Jay,

no not a stretch. We already have it.

yes we have containerized builds for long on all platforms (ppc64le, 
aarch64 and amd64). Local builders are deprecated since bigtop-1.2.


Documented in confluence
https://cwiki.apache.org/confluence/display/BIGTOP/How+to+build+Bigtop-trunk

What specifically are you missing?

Olaf



On 08.07.19 21:27, Jay Vyas wrote:

Hi olaf- we do have containeirzed builders now, right? I might be wrong about 
this, as I haven’t built full packages for bigtop in the last 12 months or so, 
but I used to use docker... Am thinking we should update docs to reflect that 
to build bigtop it’s easier to use our docker containers (Evans gave a good 
presentation on this 
https://events.static.linuxfound.org/sites/events/files/slides/How%20Bigtop%20Leveraged%20Docker%20for%20Build%20Automation%20and%20%20One-click%20Hadoop%20Provisioning%20ApacheBigData2015EU%20pub.pdf).
 ... and really just move away from using local builders of any other sort 
is that too much of a stretch?


On Jul 8, 2019, at 3:11 PM, Olaf Flebbe  wrote:

Hi

Right. This is automated compilation of protobuf is done while preparing the 
docker images.

bigtop_toolchain/manifests/protobuf.pp

I like to note we use this approach for all platforms for consistency.

Best,
Olaf



On 08.07.19 10:18, Zhenyu Zheng wrote:
Hi Yuqi,
Thanks for the reply, so can I understand that what bigtop did is similar
to what was suggested in:
https://groups.google.com/forum/#!topic/protobuf/fwLF5_t3q3U  ? I mean,
backport some patches
to 2.5.0 and build a protobuf that can work on ARM64?
BR,
Kevin Zheng

On Mon, Jul 8, 2019 at 4:14 PM Yuqi Gu  wrote:
Hi Zhenyu Zheng,

Pls try to build Hadoop in docker:
docker run -v `pwd`:/ws bigtop/slaves:1.4.0-ubuntu-16.04-aarch64 bash -l -c
'cd /ws ; hadoop-pkg'

The protobuf patches for Hadoop are list: Here
<
https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/do-component-build#L45-L105




BRs,
Yuqi

On Mon, 8 Jul 2019 at 15:42, Zhenyu Zheng 
wrote:


Hi Bigtop:

I'm trying to build Hadoop on ARM64 platform and I've meet some problem
related to protobuf 2.5.0 seems it does not build on ARM64, and there is

an

issue in Hadoop about this:
https://issues.apache.org/jira/browse/HADOOP-13363 which has not yet

been

solved. But I noticed that Bigtop has successfuly built Hadoop 2.8.5 on
ARM64, eg.



https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/COMPONENTS=hadoop,OS=ubuntu-16.04-aarch64/

so
I'm quite curious how bigtop solved this problem?

Thanks in advance

Kevin Zheng

Re: [Bigtop] Some question about build Hadoop on AArch64

[Olaf Flebbe]

Hi

Right. This is automated compilation of protobuf is done while preparing 
the docker images.


bigtop_toolchain/manifests/protobuf.pp

I like to note we use this approach for all platforms for consistency.

Best,
Olaf


On 08.07.19 10:18, Zhenyu Zheng wrote:

Hi Yuqi,

Thanks for the reply, so can I understand that what bigtop did is similar
to what was suggested in:
https://groups.google.com/forum/#!topic/protobuf/fwLF5_t3q3U  ? I mean,
backport some patches
to 2.5.0 and build a protobuf that can work on ARM64?

BR,

Kevin Zheng

On Mon, Jul 8, 2019 at 4:14 PM Yuqi Gu  wrote:


Hi Zhenyu Zheng,

Pls try to build Hadoop in docker:
docker run -v `pwd`:/ws bigtop/slaves:1.4.0-ubuntu-16.04-aarch64 bash -l -c
'cd /ws ; hadoop-pkg'

The protobuf patches for Hadoop are list: Here
<
https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/do-component-build#L45-L105




BRs,
Yuqi

On Mon, 8 Jul 2019 at 15:42, Zhenyu Zheng 
wrote:


Hi Bigtop:

I'm trying to build Hadoop on ARM64 platform and I've meet some problem
related to protobuf 2.5.0 seems it does not build on ARM64, and there is

an

issue in Hadoop about this:
https://issues.apache.org/jira/browse/HADOOP-13363 which has not yet

been

solved. But I noticed that Bigtop has successfuly built Hadoop 2.8.5 on
ARM64, eg.



https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/COMPONENTS=hadoop,OS=ubuntu-16.04-aarch64/

so
I'm quite curious how bigtop solved this problem?

Thanks in advance

Kevin Zheng

Re: systemd reported fatal error: New main PID [xxx] does not belong to service, and PID file is not owned by root. Refusing.

[Olaf Flebbe]

Hi,

Systemd version 241 will be in the next stable version of debian, called 
"buster"

Debian Buster is not supported today, anything might happen. Interesting to 
note that systemd seems to break. Has to be investigated. Guadong Xu, please 
file a JIRA ticket .

I recommend to stick with debian 9 (called stretch) for now.

Olaf



> Am 24.06.2019 um 19:32 schrieb Evans Ye :
> 
> First thing we can do is to cross validate this with our CI:
> 
> https://ci.bigtop.apache.org/view/Test/job/Bigtop-trunk-smoke-tests-1.4.0/
> 
> From your log it seems that failure happens at deployment stage, while our
> CI are good on Debian 9 with ARM and X86. I checked that both arm and x86
> images are using following systemd version:
> 
> root@04812187f0f2:~# systemd --version
> systemd 232
> +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP
> +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN
> 
> The version diff is probably the reason as what you investigated. I don't
> have any idea yet. Maybe other folks at Bigtop can chime in?
> 
> 
> Guodong Xu  於 2019年6月24日 週一 下午4:18寫道：
> 
>> Hi, all
>> 
>> I found a crucial failure related to 'systemd' when installing Bigtop 1.4
>> onto Debian 9. (Problem may exist on other versions of bigtop releases as
>> well, but I didn't test).
>> 
>> The issue is when running '# systemctl  status flink-taskmanager', it
>> always fail with such message:
>> Jun 24 07:28:59 j12-d05-09 systemd[1]: flink-taskmanager.service: New main
>> PID 26238 does not belong to service, and PID file is not owned by root.
>> Refusing.
>> 
>> A further investigation leads me to this similar bug report in VNC: a bug
>> report from VNC users: https://bugzilla.redhat.com/show_bug.cgi?id=1583159
>> 
>> And to this change commit in systemd: commit link:
>> 
>> https://github.com/systemd/systemd/commit/db256aab13d8a89d583ecd2bacf0aca87c66effc
>> 
>> "core: be stricter when handling PID files and MAINPID sd_notify()
>> messages"
>> 
>> Did anybody see this issue before? Should I log a bug for it? Solutions
>> maybe? Thanks a lot.
>> 
>> PS, my 'systemd' version is:
>> # systemd --version
>> systemd 241 (241)
>> +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP
>> +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN
>> -PCRE2 default-hierarchy=hybrid
>> 
>> Best regards
>> Guodong Xu
>> 

Re: [ANNOUNCE] Apache Bigtop 1.4.0 released

[Olaf Flebbe]

Great work, indeed! Thank you very much Evans!

Olaf

> Am 18.06.2019 um 21:48 schrieb Jay Vyas :
> 
> Great work Evans .!.!.!
> 
>> On Jun 18, 2019, at 9:28 PM, Jun HE  wrote:
>> 
>> Great! Thanks Evans for your hard work.
>> 
>> Evans Ye  于2019年6月19日周三 上午12:07写道：
>>> On behalf of the Apache Bigtop team, I'd love to announce the general
>>> availability of the Bigtop 1.4.0 release.
>>> 
>>> The release is available here:
>>>https://bigtop.apache.org/download.html#releases
>>> 
>>> A few highlights of this release include:
>>> - Integration Test Framework 2.0: one-stop integrated build and test
>>>  framework at a single entry: ./gradlew [1]
>>> - Newly developed Smoke Test CI Matrix to guard the quality of releases [2]
>>> - Hadoop 2.8.5, Spark 2.2.3, Flink 1.6.0, Alluxio 1.8.1 and more [3]
>>> - 100+ JIRAs are resolved in this release
>>> 
>>> With Bigtop 1.4.0 the community continues to deliver the most advanced big
>>> data stack to date. More details about 1.4.0 release are here:
>>>https://bigtop.apache.org/release-notes.html
>>> 
>>> Deploying Bigtop is easy: grab the repo/list file for your favorite Linux
>>> distribution:
>>>  https://www.apache.org/dyn/closer.lua/bigtop/bigtop-1.4.0/repos/
>>> and you'll be running your very own big data cluster in no time!
>>> 
>>> We welcome your help and feedback. For more information on how to report
>>> problems, and to get involved, visit the project website at:
>>>  https://bigtop.apache.org
>>> 
>>> Lastly, I want to emphasize that this is a collaborative work done by
>>> project
>>> contributors and other communities, who continue to devote time to make
>>> Bigtop a better software. Thank you all for making this release possible!
>>> 
>>> Thanks,
>>> Evans Ye, Release Manager
>>> 
>>> [1]
>>> https://cwiki.apache.org/confluence/display/BIGTOP/Quickstart+Guide%3A+Bigtop+Integration+Test+Framework+2.0
>>> [2]
>>> https://ci.bigtop.apache.org/view/Test/job/Bigtop-trunk-smoke-tests-1.4.0/
>>> [3] https://issues.apache.org/jira/browse/BIGTOP-3162

Re: [VOTE] Release Bigtop version 1.4.0 (Release Candidate 1)

[Olaf Flebbe]

Hi,

only found time to check the signatures and checksum. Played for the first time 
with fedora deployments, seems to have some rough edges, but basically works.

+1 

Olaf

> Am 05.06.2019 um 00:29 schrieb Evans Ye :
> 
> Let's extend the vote to *Sunday, June 9th, 2019 at noon PDT*.
> Let me know if it's still not enough time for you to evaluate the release :)
> 
> Youngwoo Kim (김영우)  於 2019年6月3日 週一 上午3:23寫道：
> 
>> Sorry for the late, Evans.
>> 
>> We need to extend time window for this vote. just a couple of days?
>> I would like to test the release candidate when I come back to my office.
>> 
>> Thanks,
>> Youngwoo
>> 
>> On Mon, Jun 3, 2019 at 8:02 AM Evans Ye  wrote:
>> 
>>> Hi folks,
>>> 
>>> Any info I can provide and help you to evaluate the release candidate?
>>> 
>>> Evans Ye  於 2019年5月28日 週二 下午8:37寫道：
>>> 
 Hi folks,
 
 If you have voted RC0, this is pretty much the same except for the fix
>> to
 change docker image name from trunk to 1.4.0 :)
 
 Evans Ye 於 2019年5月23日 週四，下午4:12寫道：
 
> BTW, binary artifacts for centos7, debian9, opensuse42.3 on PPC are
>> all
> uploaded to S#. However only debian 9 seems good. Others encountered
> dependency issues when installing packages. We shall only mark those
> working Distros as supported. I'll make this clear in the release
>> note.
> 
> Evans Ye  於 2019年5月23日 週四 下午8:59寫道：
> 
>> Hi folks,
>> 
>> This is the vote to release 1.4.0 of Apache Bigtop with Release
>> Candidate 1.
>> 
>> It fixes the following issues:
>> *
>> 
>>> 
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12344113&projectId=12311420
>> 
>> The vote will be going for at least 72 hours and will be closed on
>> Sunday,
>> *May 26th, 2019 at noon PDT*.  Please download, test and vote with
>> 
>> [ ] +1, accept RC1 as the official 1.4.0 release of Apache Bigtop
>> [ ] +0, I don't care either way,
>> [ ] -1, do not accept RC1 as the official 1.4.0 release of Apache
>> Bigtop, because...
>> 
>> Change(s) compared to RC0:
>> * Switch to 1.4.0 docker images for provisioner:
>> 
>>> 
>> https://github.com/apache/bigtop/commit/4921ebd737bacb48ad22c7e8b572185b572ad7b4
>> 
>> Source and binary files:
>> * https://dist.apache.org/repos/dist/dev/bigtop/bigtop-1.4.0-RC1
>> 
>> Maven staging repo:
>> *
>> 
>>> https://repository.apache.org/content/repositories/orgapachebigtop-1023
>> 
>> The git tag to be voted upon is release-1.4.0-RC1
>> 
>> Bigtop's KEYS file containing PGP keys we use to sign the release:
>> * https://dist.apache.org/repos/dist/release/bigtop/KEYS
>> 
>> Bigtop 1.4.0 CI result:
>> * Packaging:
>>> https://ci.bigtop.apache.org/view/Releases/job/Bigtop-1.4.0
>> * Deploy & Test:
>> 
>>> 
>> https://ci.bigtop.apache.org/view/Test/job/Bigtop-trunk-smoke-tests-1.4.0
>> 
>> 
>> Evans Ye, RM
>> 
> 
>>> 
>> 

Re: [VOTE] Release Bigtop version 1.4.0

[Olaf Flebbe]

Hi Evans

i am running out of time:tested amd64 dockerhadoop on linux and it run very 
smooth! It failed on docker4mac, as we discussed it before: it is an issue with 
docker4mac filesystem implementation

+1 from my side

we should update the KEYS file since it contains some invalid keys.  

Olaf


> Am 14.05.2019 um 14:37 schrieb Evans Ye :
> 
> Hi Guodong,
> 
> Thanks for the feedback. You're right the images should be 1.4.0 instead of
> trunk. Let me fix this and have RC1 out.
> I'll add notes in our release guide before we implemented something to
> better handle this automatically.
> 
> For missing trunk images, I've uploaded.
> 
> Evans
> 
> Guodong Xu  於 2019年5月14日 週二 上午11:29寫道：
> 
>> Hi, Evans
>> 
>> I checked out tag release-1.4.0-RC0.
>> 
>> Two issues:
>> 1. for 1.4.0 release, should we change config yaml files to fetch from
>> 'trunk' to '1.4.0'? Like this:
>> $ git diff
>> diff --git a/provisioner/docker/config_centos-7.yaml
>> b/provisioner/docker/config_centos-7.yaml
>> index ab0e9d8..916bf0e 100644
>> --- a/provisioner/docker/config_centos-7.yaml
>> +++ b/provisioner/docker/config_centos-7.yaml
>> @@ -15,7 +15,7 @@
>> 
>> docker:
>> memory_limit: "4g"
>> -image: "bigtop/puppet:trunk-centos-7"
>> +image: "bigtop/puppet:1.4.0-centos-7"
>> 
>> repo: "http://repos.bigtop.apache.org/releases/1.4.0/centos/7/$basearch";
>> distro: centos
>> 
>> 
>> 2. for bigtop/puppet images in docker.hub, it misses
>> 'bigtop/puppet:trunk-centos-7-aarch64'. See the error message below when
>> you run ./docker-hadoop.sh for arm. Can you or somebody upload
>> 'bigtop/puppet:trunk-centos-7-aarch64' to docker hub?
>> 
>> $ ./docker-hadoop.sh -C config_centos-7.yaml -c 3
>> Environment check...
>> Check docker:
>> Docker version 18.09.2, build 6247962
>> Check docker-compose:
>> docker-compose version 1.18.0, build 8dd22a9
>> Check ruby:
>> ruby 2.0.0p648 (2015-12-16) [aarch64-linux]
>> Pulling bigtop (bigtop/puppet:trunk-centos-7-aarch64)...
>> ERROR: manifest for bigtop/puppet:trunk-centos-7-aarch64 not found
>> 
>> [LOG] Docker container(s) startup failed!
>> 
>> -Guodong Xu
>> 
>>> On Fri, May 10, 2019 at 6:49 PM Evans Ye  wrote:
>>> 
>>> Let me try to build up those artifacts and see how things go ;)
>>> 
>>> Olaf Flebbe  於 2019年5月10日 週五 下午1:01寫道：
>>> 
>>>> Hi Evans,
>>>> 
>>>> debian 9 should be supported on ppc64le
>>>> 
>>>> olaf
>>>> 
>>>> Von meinem iPad gesendet
>>>> 
>>>>> Am 10.05.2019 um 03:27 schrieb Evans Ye :
>>>>> 
>>>>> Yes it's expected. The failure was cause by not having corresponding
>>>>> images. And all the failed builds are not supported in bigtop:
>>>>> ARM: opensuse 42.3
>>>>> PPC: centos 7, debian 9,  opensuse 42.3
>>>>> 
>>>>> Youngwoo Kim (김영우)  於 2019年5月10日 週五 上午8:40寫道：
>>>>> 
>>>>>> There are build failures on our CI, aarch64 and ppc64le. Is this
>>>> expected
>>>>>> results?
>>>>>> 
>>>>>> - Youngwoo
>>>>>> 
>>>>>>> On Mon, May 6, 2019 at 1:33 AM Evans Ye 
>> wrote:
>>>>>>> 
>>>>>>> Hi folks,
>>>>>>> 
>>>>>>> This is the vote for release 1.4.0 of Apache Bigtop.
>>>>>>> 
>>>>>>> It fixes the following issues:
>>>>>>> *
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>>> 
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12344113&projectId=12311420
>>>>>>> 
>>>>>>> The vote will be going for at least 168 hours and will be closed on
>>>>>> Sunday,
>>>>>>> *May 12th, 2019 at noon PDT*.  Please download, test and vote with
>>>>>>> 
>>>>>>> [ ] +1, accept RC0 as the official 1.4.0 release of Apache Bigtop
>>>>>>> [ ] +0, I don't care either way,
>>>>>>> [ ] -1, do not accept RC0 as the official 1.4.0 release of Apache
>>>> Bigtop,
>>>>>>> because...
>>>>>>> 
>>>>>>> Source and binary files:
>>>>>>> * https://dist.apache.org/repos/dist/dev/bigtop/bigtop-1.4.0-RC0
>>>>>>> 
>>>>>>> Maven staging repo:
>>>>>>> *
>>>>>> 
>>> https://repository.apache.org/content/repositories/orgapachebigtop-1022
>>>>>>> 
>>>>>>> The git tag to be voted upon is release-1.4.0-RC0
>>>>>>> 
>>>>>>> Bigtop's KEYS file containing PGP keys we use to sign the release:
>>>>>>> * https://dist.apache.org/repos/dist/release/bigtop/KEYS
>>>>>>> 
>>>>>>> Bigtop 1.4.0 CI result:
>>>>>>> * Packaging:
>>>> https://ci.bigtop.apache.org/view/Releases/job/Bigtop-1.4.0
>>>>>>> * Deploy & Test:
>>>>>>> 
>>>>>> 
>>>> 
>>> 
>> https://ci.bigtop.apache.org/view/Test/job/Bigtop-trunk-smoke-tests-1.4.0
>>>>>>> 
>>>>>> 
>>>> 
>>> 
>> 

Re: [VOTE] Release Bigtop version 1.4.0

[Olaf Flebbe]

Hi Evans,

debian 9 should be supported on ppc64le

olaf

Von meinem iPad gesendet

> Am 10.05.2019 um 03:27 schrieb Evans Ye :
> 
> Yes it's expected. The failure was cause by not having corresponding
> images. And all the failed builds are not supported in bigtop:
> ARM: opensuse 42.3
> PPC: centos 7, debian 9,  opensuse 42.3
> 
> Youngwoo Kim (김영우)  於 2019年5月10日 週五 上午8:40寫道：
> 
>> There are build failures on our CI, aarch64 and ppc64le. Is this expected
>> results?
>> 
>> - Youngwoo
>> 
>>> On Mon, May 6, 2019 at 1:33 AM Evans Ye  wrote:
>>> 
>>> Hi folks,
>>> 
>>> This is the vote for release 1.4.0 of Apache Bigtop.
>>> 
>>> It fixes the following issues:
>>> *
>>> 
>>> 
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12344113&projectId=12311420
>>> 
>>> The vote will be going for at least 168 hours and will be closed on
>> Sunday,
>>> *May 12th, 2019 at noon PDT*.  Please download, test and vote with
>>> 
>>> [ ] +1, accept RC0 as the official 1.4.0 release of Apache Bigtop
>>> [ ] +0, I don't care either way,
>>> [ ] -1, do not accept RC0 as the official 1.4.0 release of Apache Bigtop,
>>> because...
>>> 
>>> Source and binary files:
>>> * https://dist.apache.org/repos/dist/dev/bigtop/bigtop-1.4.0-RC0
>>> 
>>> Maven staging repo:
>>> *
>> https://repository.apache.org/content/repositories/orgapachebigtop-1022
>>> 
>>> The git tag to be voted upon is release-1.4.0-RC0
>>> 
>>> Bigtop's KEYS file containing PGP keys we use to sign the release:
>>> * https://dist.apache.org/repos/dist/release/bigtop/KEYS
>>> 
>>> Bigtop 1.4.0 CI result:
>>> * Packaging: https://ci.bigtop.apache.org/view/Releases/job/Bigtop-1.4.0
>>> * Deploy & Test:
>>> 
>> https://ci.bigtop.apache.org/view/Test/job/Bigtop-trunk-smoke-tests-1.4.0
>>> 
>> 

Re: Question about Hadoop ARM CI

[Olaf Flebbe]

Hi Zhenyu,

Apache Hadoop (and most of the other Big Data projects) do not care about cross 
platform, deployment, distribution, configuration, this is were Apache Bigtop 
jumps in.

Cross platform builds (amd64, aarch64, ppc64le) are one of the main features of 
Apache Bigtop.

Right now the jobs https://ci.bigtop.apache.org/job/Bigtop-trunk-packages/535/ 
are stopped in order to prepare for the next release, but you may see the 
breadth of the effort.
Please feel invited to contribute !

The arm CI infrastructure is donated and maintained by the Linaro folks. 

Regards,
Olaf


> Am 04.05.2019 um 08:01 schrieb Roman Shaposhnik :
> 
> Hi Zhenyu!
> 
> you may want to ask around on Apache Bigtop mailing list (CCed). We used to 
> have
> Linaro folks maintain CI pipelines for Hadoop there.
> 
> Thanks,
> Roman.
> 
> On Sun, Apr 28, 2019 at 8:44 PM Zhenyu Zheng  
> wrote:
>> 
>> Hi,
>> 
>> I'm from Huawei and currently our team is trying to enable ARM CI for 
>> popular softwares and Hadoop is one of them, when I search related topics on 
>> the web, I found this ticket: 
>> https://issues.apache.org/jira/browse/INFRA-12439 it was way back to Aug 
>> 2016, and was closed with some comment, and saying that if anyone has any 
>> questions, can send email to this address, so that's what I'm doing here.
>> 
>> So my question is, what is the status or plan? Because I can see that Apache 
>> CI now has an ARM server, but seems only a few jobs uses it 
>> https://builds.apache.org/computer/arm1/builds and Hadoop is not one of them.
>> 
>> BR,
>> 
>> Zhenyu Zheng

Re: R.I.P. mrdocs (1963–2019)

[Olaf Flebbe]

Ruhe in Frieden, Peter

Re: [DISCUSS] Jump to 1.4.0 release directly instead of 1.3.1

[Olaf Flebbe]

Hi Cos,

if you are answering to
my question regarding removal of spark1: yes please.

Olaf

> Am 17.03.2019 um 13:10 schrieb Konstantin Boudnik :
> 
> Honestly, I think it is long overdue... I don't think we should waste
> any resources on it. I can do a quick patch to get rid of it in
> master, if there's no objections.
> --
>  With regards,
> Konstantin (Cos) Boudnik
> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
> 
> Disclaimer: Opinions expressed in this email are those of the author,
> and do not necessarily represent the views of any company the author
> might be affiliated with at the moment of writing.
> 
>> On Wed, Mar 13, 2019 at 1:48 AM Olaf Flebbe  wrote:
>> 
>> Hi,
>> 
>> We learned the hard way spark1 vs spark2 . Can we drop spark1 now? What do 
>> you think?
>> 
>> olaf
>> 
>>> Am 13.03.2019 um 06:16 schrieb Konstantin Boudnik :
>>> 
>>> While I am all for the latest and greatest, I learned to be cautious
>>> when it gets to the software. Especially, when we look at something
>>> considering itself on the "top of the stack". Hence I welcome your the
>>> conservative approach!
>>> --
>>> With regards,
>>> Konstantin (Cos) Boudnik
>>> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>>> 
>>> Disclaimer: Opinions expressed in this email are those of the author,
>>> and do not necessarily represent the views of any company the author
>>> might be affiliated with at the moment of writing.
>>> 
>>>> On Wed, Mar 13, 2019 at 5:58 AM Evans Ye  wrote:
>>>> 
>>>> Thank you all for the support!
>>>> 
>>>> Until now, I was dealing with various compatibility issue for Spark.
>>>> However there're too many incompatible components with spark 2.4.0:
>>>> 
>>>> https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/
>>>> 
>>>> Therefore I'd like to revoke the 2.4.0 upgrade and just slightly bump it up
>>>> to 2.2.3.
>>>> The new release date is targeted at the end of March.
>>>> 
>>>> 
>>>> 
>>>> Jun HE  於 2019年3月6日 週三 下午9:28寫道：
>>>> 
>>>>> Awesome, and huge +1 to this!
>>>>> 
>>>>> Konstantin Boudnik  于2019年3月6日周三 下午9:11写道：
>>>>> 
>>>>>> Oh my, that's amazing! Thank you man!
>>>>>> +1 on moving directly to 1.4
>>>>>> --
>>>>>> With regards,
>>>>>> Konstantin
>>>>>> 
>>>>>> On March 5, 2019 12:58:44 AM GMT+03:00, Olaf Flebbe 
>>>>> wrote:
>>>>>>> Hi Evans,
>>>>>>> 
>>>>>>> You found and fixed a lot of bugs and made the smoke tests work, so a
>>>>>>> big +1 .
>>>>>>> 
>>>>>>> Olaf
>>>>>>> 
>>>>>>>> Am 04.03.2019 um 08:12 schrieb Evans Ye :
>>>>>>>> 
>>>>>>>> -1 welcomed as well !!!
>>>>>>>> 
>>>>>>>> Evans Ye  於 2019年3月4日 週一 下午3:11寫道：
>>>>>>>> 
>>>>>>>>> Hi all,
>>>>>>>>> 
>>>>>>>>> The overhaul as well as development is almost completed. With
>>>>>>> several new
>>>>>>>>> features added, I'd like to propose to release Bigtop 1.4.0 directly
>>>>>>>>> instead of doing 1.3.1 release(https://s.apache.org/2a8K). This
>>>>>>> helps to
>>>>>>>>> promote these new features:
>>>>>>>>> 
>>>>>>>>> 1. Full support to build and test inside docker with one stop
>>>>>>> seamlessly
>>>>>>>>> integration.
>>>>>>>>> Now just one command to build and test components:
>>>>>>>>> Example:
>>>>>>>>> $ ./gradlew spark-ind repo-ind docker-provisioner -POS=ubuntu-16.04
>>>>>>>>> -Pnexus -Penable_local_repo -Pstack=spark-standalone
>>>>>>> -Psmoke-tests=spark
>>>>>>>>> 
>>>>>>>>> 2. Support to build from git commit hash
>>>>>>>>> Example:
>>>>>>>>> $ ./gradlew kafka-pkg-ind
>>>>>>> -Pgit_repo=https://github.com/apache/kafka.git
>>>>>>>>> -Pgit_ref=1.1 -Pgit_sha1=4dae083af486eaedd27c69c973c74605bffd416b
>>>>>>>>> -Pbase_version=1.1.1
>>>>>>>>> 
>>>>>>>>> 3. Version bumps and tests
>>>>>>>>> Hadoop 2.8.5, Kafka 1.1.1, Spark 2.4.0, Alluxio 1.8.1
>>>>>>>>> New smoke tests: Flink, Giraph (Crunch)
>>>>>>>>> 
>>>>>>>>> 4. Lts of bug fixes!
>>>>>>>>> 
>>>>>>>>> If agreed, please +1 and I'll still serve as the RM for 1.4 release,
>>>>>>> if no
>>>>>>>>> preemption ;)
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Best,
>>>>>>>>> Evans
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>>>>> 

Re: [DISCUSS] Jump to 1.4.0 release directly instead of 1.3.1

[Olaf Flebbe]

Hi,

We learned the hard way spark1 vs spark2 . Can we drop spark1 now? What do you 
think?

olaf

> Am 13.03.2019 um 06:16 schrieb Konstantin Boudnik :
> 
> While I am all for the latest and greatest, I learned to be cautious
> when it gets to the software. Especially, when we look at something
> considering itself on the "top of the stack". Hence I welcome your the
> conservative approach!
> --
>  With regards,
> Konstantin (Cos) Boudnik
> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
> 
> Disclaimer: Opinions expressed in this email are those of the author,
> and do not necessarily represent the views of any company the author
> might be affiliated with at the moment of writing.
> 
>> On Wed, Mar 13, 2019 at 5:58 AM Evans Ye  wrote:
>> 
>> Thank you all for the support!
>> 
>> Until now, I was dealing with various compatibility issue for Spark.
>> However there're too many incompatible components with spark 2.4.0:
>> 
>> https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/
>> 
>> Therefore I'd like to revoke the 2.4.0 upgrade and just slightly bump it up
>> to 2.2.3.
>> The new release date is targeted at the end of March.
>> 
>> 
>> 
>> Jun HE  於 2019年3月6日 週三 下午9:28寫道：
>> 
>>> Awesome, and huge +1 to this!
>>> 
>>> Konstantin Boudnik  于2019年3月6日周三 下午9:11写道：
>>> 
>>>> Oh my, that's amazing! Thank you man!
>>>> +1 on moving directly to 1.4
>>>> --
>>>> With regards,
>>>>  Konstantin
>>>> 
>>>> On March 5, 2019 12:58:44 AM GMT+03:00, Olaf Flebbe 
>>> wrote:
>>>>> Hi Evans,
>>>>> 
>>>>> You found and fixed a lot of bugs and made the smoke tests work, so a
>>>>> big +1 .
>>>>> 
>>>>> Olaf
>>>>> 
>>>>>> Am 04.03.2019 um 08:12 schrieb Evans Ye :
>>>>>> 
>>>>>> -1 welcomed as well !!!
>>>>>> 
>>>>>> Evans Ye  於 2019年3月4日 週一 下午3:11寫道：
>>>>>> 
>>>>>>> Hi all,
>>>>>>> 
>>>>>>> The overhaul as well as development is almost completed. With
>>>>> several new
>>>>>>> features added, I'd like to propose to release Bigtop 1.4.0 directly
>>>>>>> instead of doing 1.3.1 release(https://s.apache.org/2a8K). This
>>>>> helps to
>>>>>>> promote these new features:
>>>>>>> 
>>>>>>> 1. Full support to build and test inside docker with one stop
>>>>> seamlessly
>>>>>>> integration.
>>>>>>> Now just one command to build and test components:
>>>>>>> Example:
>>>>>>> $ ./gradlew spark-ind repo-ind docker-provisioner -POS=ubuntu-16.04
>>>>>>> -Pnexus -Penable_local_repo -Pstack=spark-standalone
>>>>> -Psmoke-tests=spark
>>>>>>> 
>>>>>>> 2. Support to build from git commit hash
>>>>>>> Example:
>>>>>>> $ ./gradlew kafka-pkg-ind
>>>>> -Pgit_repo=https://github.com/apache/kafka.git
>>>>>>> -Pgit_ref=1.1 -Pgit_sha1=4dae083af486eaedd27c69c973c74605bffd416b
>>>>>>> -Pbase_version=1.1.1
>>>>>>> 
>>>>>>> 3. Version bumps and tests
>>>>>>> Hadoop 2.8.5, Kafka 1.1.1, Spark 2.4.0, Alluxio 1.8.1
>>>>>>> New smoke tests: Flink, Giraph (Crunch)
>>>>>>> 
>>>>>>> 4. Lts of bug fixes!
>>>>>>> 
>>>>>>> If agreed, please +1 and I'll still serve as the RM for 1.4 release,
>>>>> if no
>>>>>>> preemption ;)
>>>>>>> 
>>>>>>> 
>>>>>>> Best,
>>>>>>> Evans
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>> 
>>> 

Re: [DISCUSS] Jump to 1.4.0 release directly instead of 1.3.1

[Olaf Flebbe]

Hi Evans,

You found and fixed a lot of bugs and made the smoke tests work, so a big +1 . 

Olaf

> Am 04.03.2019 um 08:12 schrieb Evans Ye :
> 
> -1 welcomed as well !!!
> 
> Evans Ye  於 2019年3月4日 週一 下午3:11寫道：
> 
>> Hi all,
>> 
>> The overhaul as well as development is almost completed. With several new
>> features added, I'd like to propose to release Bigtop 1.4.0 directly
>> instead of doing 1.3.1 release(https://s.apache.org/2a8K). This helps to
>> promote these new features:
>> 
>> 1. Full support to build and test inside docker with one stop seamlessly
>> integration.
>> Now just one command to build and test components:
>> Example:
>> $ ./gradlew spark-ind repo-ind docker-provisioner -POS=ubuntu-16.04
>> -Pnexus -Penable_local_repo -Pstack=spark-standalone -Psmoke-tests=spark
>> 
>> 2. Support to build from git commit hash
>> Example:
>> $ ./gradlew kafka-pkg-ind -Pgit_repo=https://github.com/apache/kafka.git
>> -Pgit_ref=1.1 -Pgit_sha1=4dae083af486eaedd27c69c973c74605bffd416b
>> -Pbase_version=1.1.1
>> 
>> 3. Version bumps and tests
>> Hadoop 2.8.5, Kafka 1.1.1, Spark 2.4.0, Alluxio 1.8.1
>> New smoke tests: Flink, Giraph (Crunch)
>> 
>> 4. Lts of bug fixes!
>> 
>> If agreed, please +1 and I'll still serve as the RM for 1.4 release, if no
>> preemption ;)
>> 
>> 
>> Best,
>> Evans
>> 
>> 
>> 
>> 

[jira] [Created] (BIGTOP-3178) Fix two insecure maven repositories

[Olaf Flebbe (JIRA)]

Olaf Flebbe created BIGTOP-3178:
---

 Summary: Fix two insecure maven repositories
 Key: BIGTOP-3178
 URL: https://issues.apache.org/jira/browse/BIGTOP-3178
 Project: Bigtop
  Issue Type: Task
Reporter: Olaf Flebbe
Assignee: Olaf Flebbe
 Fix For: 1.3.1


FIx two insecure maven repositories which slipped previous code audits



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Support LZO codec

[Olaf Flebbe]

Hi 

lzo is distributed under GPL. This is incompatible to ASLv2 . 
See category x 
https://apache.org/legal/resolved.html#what-can-we-not-include-in-an-asf-project-category-x

olaf

> Am 25.02.2019 um 17:50 schrieb Mikhail Epikhin :
> 
> Hello.
> 
> Why did you drop lzo codec support?
> 
> I found a task https://issues.apache.org/jira/browse/BIGTOP-164 with 
> deletion, but did not found a reason.
> Why?
> 
> Mikhail Epikhin

Re: [DISCUSS] Upgrade to Puppet 5.X

[Olaf Flebbe]

Hi all,

some may recall we had this discussion before:

I voted to stay at the distro provided packages and modules, since puppetlabs 
distro does not support our core target of being arch-independent.

However, the downside of puppet in general is that we lose recent opensuse leap 
( no support from puppetlabs  nor distro)

There are other / additional options, though:

3) remove distros without proper puppet support. If someone complains, ask 
him/her to go for options 4

My vote was to remove opensuse from our build matrix.

4a) Building ruby and puppet ourselfs and providing packages for all platforms.

4b) Work with distro upstream to support a decent version (see opensuse build 
service as an example)

5) switch from puppet to a different configmanagment/automation tool: This has 
to be carefully planned, since we will likely have the same version/arch/distro 
provided problem anyway.


my 2cents

olaf


> Am 14.02.2019 um 09:15 schrieb Evans Ye :
> 
> Hi folks,
> 
> Recently I spend some effort trying to upgrade to Puppet 5.X[1]. However as
> what Olaf pointed out: Puppet 5 is not available for AARCH64 or PPC64LE
> except for EL7.  And it seems that Puppet have not intention to add cross
> platform support.
> *
> https://puppet.com/docs/puppet/5.5/system_requirements.html#platforms-without-packages
> * https://tickets.puppetlabs.com/browse/PA-1330
> 
> I think here we have two options:
> 1.  Upgrade to Puppet to 5.X as much as we can, as what we did for others.
> For example, we don't have full set of Distros supported on AARCH64. The
> same idea goes here. We don't have Puppet 5 on all the supported archs. But
> we'll ensure what we provide still works well. For example Puppet 3.X or
> 4.X still works fine with Bigtop on AARCH64.
> 2. Discard this upgrade and relies on Distro's version of puppet. Not a
> good sign in software though. When the gap become too big. We'll spend a
> lot of effort to catch up...
> 
> I'd like to take option 2. What do you think?
> 
> - Evans

Re: [DISCUSS] Next minor Release of Apache Bigtop

[Olaf Flebbe]

Hi Evans,

+1 to your proposal! 


Happy new year
olaf

> Am 06.02.2019 um 14:10 schrieb Evans Ye :
> 
> Thank you all for supporting. I'll take the RM role and start driving the
> release process.
> 
> Due to limited development capacity of our community, my intention for this
> release as well as the entire overhauling,
> is to focus on automation/CI/Documentation. The goal is to spend less
> amount of time to sustain the project and yield
> more time for having fun with new stuffs :)
> 
> BTW, right now we're celebrating Chinese New Year, so happy new year to the
> community.
> 
> - Evans in Taiwan
> 
> 
> Konstantin Boudnik  於 2019年2月6日 週三 上午4:51寫道：
> 
>> Great idea, I'll try to help as much as my time permits. Thanks for
>> volunteering!
>> --
>> Regards,
>>  Cos
>> 
>> On February 3, 2019 2:04:34 PM GMT+03:00, Evans Ye 
>> wrote:
>>> Hi all,
>>> 
>>> I'd like to propose a next minor release of Apache Bigtop, which is
>>> 1.3.1.
>>> 
>>> Although Bigtop 1.3.0 was released back in Nov. 2018. It's a good sign
>>> for
>>> a project to release early, release often.
>>> 
>>> The release will contain the following features:
>>> 1. Overhaul the deployment and testing modules
>>> 2. Project Frontier: Bigtop Integration Test Framework 2.0
>>> 3. Minor upgrade to components such as: Hadoop, Spark, Kafka, etc
>>> 4. Other bug fixes
>>> 
>>> This minor release is more focus on the quality end instead of catching
>>> up
>>> to latest version of components.
>>> The plan is to have the RC out around mid February, 2019.
>>> 
>>> If the community is up to the release, I volunteer to be the RM.
>>> However I'll yield if someone would like to take the role :)
>>> 
>>> 
>>> - Evans
>> 

Re: [DISCUSS] Next minor Release of Apache Bigtop

[Olaf Flebbe]

Overhauling testing is great +1.


> Am 03.02.2019 um 12:04 schrieb Evans Ye :
> 
> Hi all,
> 
> I'd like to propose a next minor release of Apache Bigtop, which is 1.3.1.
> 
> Although Bigtop 1.3.0 was released back in Nov. 2018. It's a good sign for
> a project to release early, release often.
> 
> The release will contain the following features:
> 1. Overhaul the deployment and testing modules
> 2. Project Frontier: Bigtop Integration Test Framework 2.0
> 3. Minor upgrade to components such as: Hadoop, Spark, Kafka, etc
> 4. Other bug fixes
> 
> This minor release is more focus on the quality end instead of catching up
> to latest version of components.
> The plan is to have the RC out around mid February, 2019.
> 
> If the community is up to the release, I volunteer to be the RM.
> However I'll yield if someone would like to take the role :)
> 
> 
> - Evans

Re: Define the future of Apache Bigtop

[Olaf Flebbe]

Hi,

Thanks all for sharing your views.

Since I am not able to invest in Bigtop any more, I will not give 
recommendations.

However I like to share to results of my previous work on topics (1) and (3)

(1a)  If you see that a particular software does not fit your needs, it is best
to work with the respective project. At an downstream user of a particular 
project
Bigtop cannot maintain forks. 

(1b)  In my previous live I was with many HPC and Big Data sysadmins. 
The all loved to automate things. There are a of tools fitting the needs, but 
surely 
one does not need an UI to bootstrap/maintain a Linux/Hadoop/HPC/Big Data 
Cluster.
(It might be helpful for beginners and for software concepts running on 
powerpoint)

All the tools (chef/saltstack/puppet/ansible/whatever) are already able to do 
the job, not necessary to 
reinventing the wheel and creating a UI to bootstrap a hadoop cluster.
Since part of the job is to install kerberos, configure name and directory 
services, you name it ... there 
are ready and battletested recipies already available for each of the named 
tools.
At Bigtop there are puppet recipies which can be tuned to really relevant 
environments. 
(Some of my collegues rewrote the stuff with the tool the are used at their 
site for their needs)

I am particulary refering to my talk at 2015 Big Data conference 
http://oflebbe.de/presentations/2015/kerberos_ldap_puppet_bigtop.pdf

There is no easy way to run a production ready Big Data Cluster. It is an 
complex environment 
where you need to know how to plan your HW, networking, enterprise integration 
etc ...
A beginner will fail. Better let them fail fast.

This comment does not necessary extend to the aspects of monitoring, reporting, 
proxying and failure detection.

(3)   No standard configuration of kerberos servers can be handled by k18s (as 
of 2017)

Without kerberos you will not get secure hadoop. 

Without security .. it is not production ready.

Argument is as follows: Standard Kerberos needs to have the reverse ip address 
lookup of a server to match the FQDN of the server. 

In k18s you do not have control over the reverse name lookup of a  service. 
(This argument is valid for docker as well. That is the reason I chose to 
provision to lxc in my 2015 talk)

(3a) There might be other very elaborate ways by not using reverse name lookups 
in kerberos, 
but it is way beyond my insight into kerberos.

(3b) This is based on k18s as of ~2017. Iff k18s is now able to support 
canonical hostnames, this argument is void

(3c) Hadoop3 may support other ways to create a secure environment

Best Regards,
Olaf



> Am 08.01.2019 um 06:43 schrieb Jun HE :
> ance
> Thanks for the summary, Evans.
> 
> From Linaro's perspective, it would be great to see:
> 1. Simple and unified deployment/management for Bigtop SW stack across
> various distros and architectures.
>We've seen people like Bigtop stack (open and easy to customize) but
> failed to deploy and manage in an easy way. While Ambari provides such
> capabilities it combines tight with HDP and x86. And this prevents the
> wider aoption for Bigtop. As some academies and industry players are
> leveraging certain Bigdata workloads to non-x86 platforms it would give
> them great help if we can enable Bigtop SW stack support in Ambari. Also I
> think community can work with ODPi (https://www.odpi.org/) on this to move
> things forward more effectively.
> 2. Add and improve tests (smoke tests/package tests) to improve Bigtop
> stack quality.
>There are quite a few components enabled in the stack but lack of
> tests. And some existing smoke tests need improvement to align with new
> versions/settings. This could be improved to achieve good tests quality and
> results to give users pretty confident to use Bigtop in their daily work.
> 3. Native K8s support as Jay has mentioned
> 4. Hadoop 3.x supports.
>This could take times and effort (I think most come from ecosystem
> dependecies), but it could be our primary but not prioritized work. :)
> 
> That's things came up to me so far. Pls feel free to comment.
> 
> Thanks.
> 
> Jun
> 
> Evans Ye  于2019年1月4日周五 上午12:10写道：
> 
>> To sum up the discussion of this thread, there're two ideas proposed:
>> * Native K8S packages - by Jay
>> * Deploy Bigtop via Ambari - by Jun
>> 
>> I think both are very good ideas. However I'd like to gather more info from
>> the requirement point of view. For example, Jun what do you see from the
>> Linaro perspective for user requirements on Big Data? I think that's a
>> valuable input to the community since you guys are also becoming a
>> significant contributors at the community.
>> 
>> - Evans
>> 
>> 
>> Jun HE  於 2018年12月17日 週一 上午9:17寫道：
>> 
>>> +1 for this!
>>> 
>>> And another thing is I'm not clear about the status of ambari mpack in
>>> Bigtop, so can we deploy Bigtop SW stack using ambari now? If the answer
>> is
>>> no, maybe this is what end user would like to see.
>>> 
>>> Jay Vyas  于2018年12月15日

Gitbox (please review)

[Olaf Flebbe]

Hi,

* CI has been bulk edited.

* I updated all (hopefully) pages in confluence (not touching historic pages)

* BIGTOP-3128
https://github.com/apache/bigtop/pull/438

has the changes in our repository in place, please review. (This is a monkey 
replace job)

Cheers,
Olaf