Re: Re: [Discuss] Initial Draft for Roadmap

[Jun HE]

I agree with "1. Fix some security problems by upgrading Spring Framework
and others".Security is important and we can use this as a ramp-up as the
community is gradually bringing everything in Ambari back on track, like PR
review, CI jobs,...

For other things like "frontend refactor", it would be good to have more
inputs on existing issues/limitations before we decide to "refactor".
For the mpack part, my understanding is that "management packs" is a
framework that allows different stack vendors to provide a compatible stack
that can be maintained and operated by Ambari. There could be minor
differences between different vendors. So the definition like
path/repo/name etc should be the vendor's decision instead of being defined
by Ambari. I would suggest leaving the mpack related work to a certain
community, for example, Bigtop, and Ambari just review and accept their PR
just for the user's convenience.

Just my 2 cents. And happy to learn any comments.

吴治国  于2022年8月16日周二 17:50写道：

> Hi community,
>
> In my original idea, I was planning to provide Bigtop3.1.0 + Ambari2.7.5
> for users at first, and keep all big changes in Ambari2.8.0, e.g.
> 1、Fix some security problems by upgrading Spring Framework and others
> 2、Frontend refactor
> 3、Upgrade python version
> etc.
>
> All these changes will take long time, which may be bad for the release
> cycle of Ambari itself.
> So I'm considering we can move Mpack development to Ambari side, and 2.8.0
> is all about Bigtop stack, move other big changes to next major
> version(except the security problem, which absolutely need to be fixed in
> next release).
>
> But we need to make final decision for the things we discussed before,
> like,
> 1、Add prefix to installation path(/usr/bigtop)
> 2、Which repo to store the packages(with prefix in installation path)?
> 3、Related name(Stack name: BIGTOP, scripts: conf-select, distro-select,
> bigtop-select are all ok for me)?
> Any other things I forgot?
>
> What’s your opinion?
>
> Best Regards,
> Zhiguo Wu
>
> On 2022/06/23 06:26:44 "Battula, Brahma Reddy" wrote:
> >
> > IMO, First way is not heavier when we consider all the efforts for mpack
> with upgrades. And first one is easiest to adopt now.
> >
> > We'll start discussing the bigtop mailing list further.
> >
> >
> > On 21/06/22, 1:11 PM, "吴治国"  wrote:
> >
> > Thanks for the details, Brahma and Kengo.
> >
> > If I understand it correctly, there are two approaches under
> discussion:
> >
> > 1、Prepare to release 2.7.7, in this version, we won't use HDP
> components anymore, we'll replace it with the community version provided by
> Bigtop, and update RPM and DEB build scripts provided by Bigtop, change the
> installation path so hdp-select can be reused. (Due to component version
> changes, I don't know if the code under /stacks needs to be updated)
> > 2、Develop Bigtop mpack for Ambari 2.7.5, we don't need to change
> Ambari's code (if the code under /stacks can be reused in the first way,
> maybe it also works in this way?)
> >
> > Isn't the workload of the first way heavier?
> >
> >
> > About the meeting
> >
> > I thought maybe email discussion would be better?
> > The discussion can be open for a few days and all interested people
> in the community can participate, not just a few of us.
> > And due to time zone and language issues, many people are hard to
> attending the meetings, email discussions may be more suitable for these
> people.
> >
> > What do you think?
> >
> > Best Regards,
> > Zhiguo Wu
> >
> > > On Jun 20, 2022, at 23:15, Battula, Brahma Reddy
>  wrote:
> > >
> > >>> Assuming the understanding above is correct, my new concerns are:
> > >
> > > Yes, Hope we'll in same page now. Still we can discuss in weekly
> call.
> > >
> > >>> * Can we still call it as "HDP"? Doesn't it infringe Cloudera's
> trademark?
> > >>> If so, should we give a new stack name to it, as you mentioned as
> > >>> planA in [3]?
> > >>> (or it makes upgrading HDP cluster difficult? I don't understand
> it so much)
> > >
> > > Yes, it may be. Hence I introduced the planA and preferred it. one
> more idea(bad) came in my mind that can we treat "HDP: Hadoop Data
> Platform", not sure whether it should be ok or not(and we might not
> delivering as distro's).
> > >
> > >>> * It may be a bit tough work to fix Bigtop's build scripts so
> that its
> > >>> artifacts match Ambari's requirements.
> > >>> (package versioning, file layouts, included and not included
> files,
> > >>> file permissions, user/group creation, etc.)
> > >
> > > Only paths matter right everything else can be re-used from the
> hdp-select.
> > >
> > >
> > > On 20/06/22, 8:13 PM, "Kengo Seki"  ma...@apache.org>> wrote:
> > >
> > > Thanks for the elaboration Brahma, but I'm a bit confused.
> > > At this point, my understanding is as follows. Are these correct?
> > >
> > 

Re: Propose to drop Docker Sandbox and Vagrant Provisioner

[Jun HE]

+1.

Yuqi Gu  于2022年8月5日周五 11:32写道：

> +1,
> Thanks, Evans.
>
> BRs,
> Yuqi
>
> Matt Andruff  于2022年8月4日周四 23:44写道：
>
> > +1
> >
> > On Wed, Aug 3, 2022 at 11:49 PM Evans Ye  wrote:
> >
> > > Hi folks,
> > >
> > > Both Docker Sandbox[1] and Vagrant Provisioner[2] are not
> > > updated/maintained for a long time. I propose to remove those code to
> > > improve our codebase readability and eliminate potential bugs yield by
> > > non-active maintained code.
> > > I'll wait for 3 days. If there are no objections I'll proceed with a
> PR.
> > >
> > > [1] https://github.com/apache/bigtop/tree/master/docker/sandbox
> > > [2] https://github.com/apache/bigtop/tree/master/provisioner/vagrant
> > >
> >
>

Re: [Discussion] support OpenEuler in Bigtop?

[Jun HE]

Fully agree with Cos's comments. Yes, we need to ensure there are enough
resources, not only HW but also people's bandwidth, to contribute or a new
distro support will become an extra burden besides the existing list. I
think the Linaro folks can discuss with the OpenEuler community the details
of how they could support.

I will do some early tests to understand how much effort it might be.

Konstantin Boudnik  于2022年6月7日周二 04:15写道：

> Whatever we are going to decide, we need to keep in mind a couple of
> things:
>  - CI hardware resources (where are they coming from and how much we need)
>  - on-going support for this new OS: most of us here are well versed in
>deb/centos family of OSes. However, new one might prove to be a
> challenge
>even if it seems to be a descendant of Fedora (is it?).
>
> At any rate, if there's someone from that community willing to designate
> their
> effort on making it work (as usual, on a branch first and then rolling
> back to
> master if we can make it relatively non-disruptive) - I don't have much
> objections.
>
> --
>   Cos
>
>
> On Mon, Jun 06, 2022 at 08:25PM, Jun HE wrote:
> > OpenEuler is a distro initiated by Huawei and it is in rapid evolvement.
> > From its stat [1] OpenEuler has 300+ organization members and received
> > pretty many deployments in enterprise users, especially in China. I'm
> > wondering about your idea about OpenEuler support in Bigtop. The packages
> > management in OpenEuler is dnf based, which means it should be easy to
> port
> > our existing stack to OpenEuler. If the community is interested in this I
> > can do some exploration. :)
> >
> > 1. https://datastat.openeuler.org/en/overview
>

[Discussion] support OpenEuler in Bigtop?

[Jun HE]

OpenEuler is a distro initiated by Huawei and it is in rapid evolvement.
>From its stat [1] OpenEuler has 300+ organization members and received
pretty many deployments in enterprise users, especially in China. I'm
wondering about your idea about OpenEuler support in Bigtop. The packages
management in OpenEuler is dnf based, which means it should be easy to port
our existing stack to OpenEuler. If the community is interested in this I
can do some exploration. :)

1. https://datastat.openeuler.org/en/overview

Re: Docker arm-5 node not reachable by Jenkins

[Jun HE]

Hi Luca,

Sorry for the late response. Yuqi and I are responsible for these Arm CI
nodes. I'll check with this and update you later.

Regards,

Jun

Luca Toscano  于2021年12月28日周二 16:48写道：

> Hi everybody,
>
> I've set https://ci.bigtop.apache.org/computer/docker-slave-arm-5/
> temporarily offline in the Jenkins UI, it seems not reachable via ssh
> from the Jenkin's perspective.
>
> What is the procedure to follow for these nodes?
>
> Luca
>

[jira] [Created] (BIGTOP-3557) Java on Fedora-33 is set to JDK11 as default

[Jun He (Jira)]

Jun He created BIGTOP-3557:
--

 Summary: Java on Fedora-33 is set to JDK11 as default
 Key: BIGTOP-3557
 URL: https://issues.apache.org/jira/browse/BIGTOP-3557
 Project: Bigtop
  Issue Type: Bug
  Components: toolchain
Reporter: Jun He
Assignee: Jun He


On Fedora-33, JDK-11 is set as default JDK when installing pandoc for R. This 
causes hive build failure on fedora-33 as following:
{code:java}
[INFO] Hive Kryo Registrator .. SKIPPED
[INFO] Hive TestUtils . SKIPPED
[INFO] Hive Packaging . SKIPPED
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time:  04:11 min
[INFO] Finished at: 2021-06-11T01:00:36Z
[INFO] 
[ERROR] Failed to execute goal on project hive-upgrade-acid: Could not resolve 
dependencies for project org.apache.hive:hive-upgrade-acid:jar:3.1.2: Could not 
find artifact jdk.tools:jdk.tools:jar:1.7 at specified path 
/usr/lib/jvm/java-11-openjdk-11.0.11.0.9-2.fc33.x86_64/../lib/tools.jar -> 
[Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
{code}





--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (BIGTOP-3554) Phoenix build failure caused by protoc-2.5.0

[Jun He (Jira)]

Jun He created BIGTOP-3554:
--

 Summary: Phoenix build failure caused by protoc-2.5.0
 Key: BIGTOP-3554
 URL: https://issues.apache.org/jira/browse/BIGTOP-3554
 Project: Bigtop
  Issue Type: Bug
  Components: build
Reporter: Jun He
Assignee: Jun He


Similar to Hbase/hive, com.google.protobuf:protoc:exe:2.5.0 doesn't provide 
non-x86 binary
{code:java}
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time:  01:03 min
[INFO] Finished at: 2021-06-09T09:04:44Z
[INFO] 
[ERROR] Failed to execute goal 
org.xolstice.maven.plugins:protobuf-maven-plugin:0.6.1:compile (compile-protoc) 
on project phoenix-core: Unable to resolve artifact: Missing:
[ERROR] --
[ERROR] 1) com.google.protobuf:protoc:exe:linux-aarch_64:2.5.0
[ERROR]
[ERROR]   Try downloading the file manually from the project website.
[ERROR]
[ERROR]   Then, install it using the command:
[ERROR]   mvn install:install-file -DgroupId=com.google.protobuf 
-DartifactId=protoc -Dversion=2.5.0 -Dclassifier=linux-aarch_64 -Dpackaging=exe 
-Dfile=/path/to/file
[ERROR]
[ERROR]   Alternatively, if you host your own repository you can deploy the 
file there:
[ERROR]   mvn deploy:deploy-file -DgroupId=com.google.protobuf 
-DartifactId=protoc -Dversion=2.5.0 -Dclassifier=linux-aarch_64 -Dpackaging=exe 
-Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
[ERROR]
[ERROR]   Path to dependency:
[ERROR] 1) org.apache.phoenix:phoenix-core:jar:5.1.0
[ERROR] 2) com.google.protobuf:protoc:exe:linux-aarch_64:2.5.0
[ERROR]
[ERROR] --
[ERROR] 1 required artifact is missing.
[ERROR]
[ERROR] for artifact:
[ERROR]   org.apache.phoenix:phoenix-core:jar:5.1.0
[ERROR]
[ERROR] from the specified remote repositories:
[ERROR]   apache release 
(https://repository.apache.org/content/repositories/releases/, releases=true, 
snapshots=true),
[ERROR]   apache.snapshots (https://repository.apache.org/snapshots, 
releases=false, snapshots=true),
[ERROR]   central (https://repo.maven.apache.org/maven2, releases=true, 
snapshots=false) {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (BIGTOP-3551) Phantomjs caused Ambari build failure on non-x86

[Jun He (Jira)]

Jun He created BIGTOP-3551:
--

 Summary: Phantomjs caused Ambari build failure on non-x86
 Key: BIGTOP-3551
 URL: https://issues.apache.org/jira/browse/BIGTOP-3551
 Project: Bigtop
  Issue Type: Bug
  Components: build
Reporter: Jun He
Assignee: Jun He


After ambari is bumped to 2.7.5, it cannot build on non-x86 platforms as 
phantomjs prebuilt is required.
{code:java}
[^[[1;34mINFO^[[m] Running 'yarn install --ignore-engines --pure-lockfile' in 
/var/lib/jenkins/bigtop/output/ambari/ambari-2.7.5.0/ambari-web
[^[[1;34mINFO^[[m] yarn install v0.23.2
[^[[1;31mERROR^[[m] warning Ambari@2.4.0: No license field
[^[[1;34mINFO^[[m] [1/4] Resolving packages...
[^[[1;34mINFO^[[m] [2/4] Fetching packages...
[^[[1;31mERROR^[[m] warning fsevents@0.3.8: The platform "linux" is 
incompatible with this module.
[^[[1;34mINFO^[[m] info "fsevents@0.3.8" is an optional dependency and failed 
compatibility check. Excluding it from installation.
[^[[1;31mERROR^[[m] warning fsevents@1.1.1: The platform "linux" is 
incompatible with this module.
[^[[1;34mINFO^[[m] info "fsevents@1.1.1" is an optional dependency and failed 
compatibility check. Excluding it from installation.
[^[[1;34mINFO^[[m] [3/4] Linking dependencies...
[^[[1;34mINFO^[[m] [4/4] Building fresh packages...
[^[[1;31mERROR^[[m] error 
/var/lib/jenkins/bigtop/output/ambari/ambari-2.7.5.0/ambari-web/node_modules/phantomjs-prebuilt:
 Command failed.
[^[[1;31mERROR^[[m] Exit code: 1
[^[[1;31mERROR^[[m] Command: sh
[^[[1;34mINFO^[[m] info Visit https://yarnpkg.com/en/docs/cli/install for 
documentation about this command.
[^[[1;31mERROR^[[m] Arguments: -c node install.js
[^[[1;31mERROR^[[m] Directory: 
/var/lib/jenkins/bigtop/output/ambari/ambari-2.7.5.0/ambari-web/node_modules/phantomjs-prebuilt
[^[[1;31mERROR^[[m] Output:
[^[[1;31mERROR^[[m] PhantomJS not found on PATH
[^[[1;31mERROR^[[m] Unexpected platform or architecture: linux/arm64
[^[[1;31mERROR^[[m] It seems there is no binary available for your 
platform/architecture
[^[[1;31mERROR^[[m] Try to install PhantomJS globally {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Turn off patch

[Jun HE]

I'm a little confused. So the idea is to build source code from HDP using
Bigtop? If that is true, then as Masatake mentioned removing a patch is not
enough. You may need to change do-component-build, install_, and
even those svc and configuration files, to make it work with existing HDP
components.

Masatake Iwasaki  于2021年6月3日周四 下午2:36写道：

> > I want to build an old version of HDP.
>
> I think you need some considerations.
>
> * Products of HDP x.y.z is expected to be built against products of HDP
> x.y.z.
>The versions of products of HDP and Bigtop do not match.
>You will need to fix scripts of Bigtop for resolving mismatch.
>(do-component-build and install_*.sh tend to be relevant.)
>
> * Even if you succeeded to build Bigtop package from HDP source code,
>the contents should be quite different from HDP one.
>(You should just use .srpm of HDP for building .rpm of HDP if
> available.)
>
> * Artifacts of HDP usually depend on HDP specific artifacts
>published in Hortonworks Maven repositories.
>(This could be resolved by just tweaking ~/.m2/settings.xml or pom.xml
> of products.)
>
>
> > Practical example:  *In master:* I want to build spark 2.3.0.  If I use
> > bigtop.bom or git to build both fail when they try to apply
> > patch2-snappy-java-1.1.8.diff.
>
> The master branch of Bigtop is expected to build Spark 3 against Hadoop 3.
>
> https://github.com/apache/bigtop/commit/f47d4bcb16e67f50b2403f4fc5071b60ec88c081
>
> Just removing patches should not be enough.
>
>
> On 2021/06/03 10:26, Jun HE wrote:
> > Hi Matt,
> >
> > Are you just trying to disable a certain patch temporarily? If yes, my
> > understanding is that just delete that patch
> > in bigtop-packages/src/common// would work. Pls have a try.
> >
> > Regards,
> >
> > Jun
> >
> > Matt Andruff  于2021年6月3日周四 上午4:16写道：
> >
> >> Good Day Team,
> >>
> >> I want to build an old version of HDP.  Theoretically I should be able
> to
> >> do that with BigTop, but I'm having issues because patches are running
> and
> >> not finding what they're looking for and failing my build.
> >>
> >> How do I turn off a patch intended for a different version (say spark
> >> 2.3.0)?
> >>
> >> Practical example:  *In master:* I want to build spark 2.3.0.  If I use
> >> bigtop.bom or git to build both fail when they try to apply
> >> patch2-snappy-java-1.1.8.diff.
> >> Which is totally needed for a fix in some later version of spark, but
> not
> >> the version I'm using. As far as I can tell, there isn't a way to turn
> >> on/off specific version patches, or perhaps this patch isn't following
> the
> >> paradigm.
> >>
> >> Should I open a ticket to create a new functionality that only applies
> >> patches to specific versions? Or is this already baked in and I have
> hit a
> >> subtle bug?
> >> Thoughts?
> >>
> >
>

Re: Turn off patch

[Jun HE]

Hi Matt,

Are you just trying to disable a certain patch temporarily? If yes, my
understanding is that just delete that patch
in bigtop-packages/src/common// would work. Pls have a try.

Regards,

Jun

Matt Andruff  于2021年6月3日周四 上午4:16写道：

> Good Day Team,
>
> I want to build an old version of HDP.  Theoretically I should be able to
> do that with BigTop, but I'm having issues because patches are running and
> not finding what they're looking for and failing my build.
>
> How do I turn off a patch intended for a different version (say spark
> 2.3.0)?
>
> Practical example:  *In master:* I want to build spark 2.3.0.  If I use
> bigtop.bom or git to build both fail when they try to apply
> patch2-snappy-java-1.1.8.diff.
> Which is totally needed for a fix in some later version of spark, but not
> the version I'm using. As far as I can tell, there isn't a way to turn
> on/off specific version patches, or perhaps this patch isn't following the
> paradigm.
>
> Should I open a ticket to create a new functionality that only applies
> patches to specific versions? Or is this already baked in and I have hit a
> subtle bug?
> Thoughts?
>

[jira] [Created] (BIGTOP-3549) HBase build failure caused by protobuf in xolstice

[Jun He (Jira)]

Jun He created BIGTOP-3549:
--

 Summary: HBase build failure caused by protobuf in xolstice
 Key: BIGTOP-3549
 URL: https://issues.apache.org/jira/browse/BIGTOP-3549
 Project: Bigtop
  Issue Type: Bug
  Components: hbase
Reporter: Jun He
Assignee: Jun He


HBase build on Arm is failed with following output:
{code:java}
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time:  01:35 min
[INFO] Finished at: 2021-06-01T07:40:05Z
[INFO] 
[ERROR] Failed to execute goal 
org.xolstice.maven.plugins:protobuf-maven-plugin:0.5.0:compile (compile-protoc) 
on project hbase-protocol: Missing:
[ERROR] --
[ERROR] 1) com.google.protobuf:protoc:exe:linux-aarch_64:2.5.0
[ERROR]
[ERROR]   Try downloading the file manually from the project website.
[ERROR]
[ERROR]   Then, install it using the command:
[ERROR]   mvn install:install-file -DgroupId=com.google.protobuf 
-DartifactId=protoc -Dversion=2.5.0 -Dclassifier=linux-aarch_64 -Dpackaging=exe 
-Dfile=/path/to/file
[ERROR]
[ERROR]   Alternatively, if you host your own repository you can deploy the 
file there:
[ERROR]   mvn deploy:deploy-file -DgroupId=com.google.protobuf 
-DartifactId=protoc -Dversion=2.5.0 -Dclassifier=linux-aarch_64 -Dpackaging=exe 
-Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
[ERROR]
[ERROR]   Path to dependency:
[ERROR]1) org.apache.hbase:hbase-protocol:jar:2.2.6
[ERROR]2) com.google.protobuf:protoc:exe:linux-aarch_64:2.5.0
[ERROR]
[ERROR] --
[ERROR] 1 required artifact is missing.
[ERROR]
[ERROR] for artifact:
[ERROR]   org.apache.hbase:hbase-protocol:jar:2.2.6
[ERROR]
[ERROR] from the specified remote repositories:
[ERROR]   apache.snapshots (https://repository.apache.org/snapshots, 
releases=false, snapshots=true),
[ERROR]   central (https://repo.maven.apache.org/maven2, releases=true, 
snapshots=false) {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (BIGTOP-3548) Hive build failure caused by protoc-jar-maven-plugin missing aarch64 support

[Jun He (Jira)]

Jun He created BIGTOP-3548:
--

 Summary: Hive build failure caused by protoc-jar-maven-plugin 
missing aarch64 support
 Key: BIGTOP-3548
 URL: https://issues.apache.org/jira/browse/BIGTOP-3548
 Project: Bigtop
  Issue Type: Bug
  Components: hive
Reporter: Jun He
Assignee: Jun He


Failed to build Standalone Metastore and output following log on reason:
{code:java}
[ERROR] Failed to execute goal 
^[com.github.os72:protoc-jar-maven-plugin:3.5.1.1:run^[[m ^[[1m(default)^[[m on 
project ^[[36mhive-standalone-metastore^[[m: ^[[1;31mError resolving artifact: 
com.google.protobuf:protoc:2.5.0^[[m: Failure to find 
com.google.protobuf:protoc:exe:linux-aarch_64:2.5.0 in 
https://repo.maven.apache.org/maven2 was cached in the local repository, 
resolution will not be reattempted until the update interval of central has 
elapsed or updates are forced
[ERROR]
[ERROR] Try downloading the file manually from the project website.
[ERROR]
[ERROR] Then, install it using the command:
[ERROR] mvn install:install-file -DgroupId=com.google.protobuf 
-DartifactId=protoc -Dversion=2.5.0 -Dclassifier=linux-aarch_64 -Dpackaging=exe 
-Dfile=/path/to/file
[ERROR]
[ERROR] Alternatively, if you host your own repository you can deploy the file 
there:
[ERROR] mvn deploy:deploy-file -DgroupId=com.google.protobuf 
-DartifactId=protoc -Dversion=2.5.0 -Dclassifier=linux-aarch_64 -Dpackaging=exe 
-Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
[ERROR]
[ERROR]
[ERROR]   com.google.protobuf:protoc:exe:2.5.0
[ERROR]
[ERROR] from the specified remote repositories:
[ERROR]   apache.snapshots (https://repository.apache.org/snapshots, 
releases=false, snapshots=true),
[ERROR]   central (https://repo.maven.apache.org/maven2, releases=true, 
snapshots=false)
[ERROR] -> ^[[1m[Help 1]^[[m
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the 
^[[1m-e^[[m switch.
[ERROR] Re-run Maven using the ^[[1m-X^[[m switch to enable full debug logging. 
{code}
 

The root cause is protoc-jar-maven-plugin is used to reference protoc 
executable of certain os/arch to process metastore.proto



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] AWS Graviton

[Jun HE]

Yes, as Ganesh mentioned, Bigtop releases after v1.3 can run on Graviton
without any issues.
And AWS released Graviton2 based on Arm's N1 core last year. It is very
powerful and also works with Bigtop release.

Ganesh Raju  于2021年5月22日周六 上午3:50写道：

> Hi Evans
> We are very well aware of it and also do quite a lot of testing on AWS
> Graviton. Our distribution will not have any issues running on it.
>
> Regards
> Ganesh
>
> On Fri, May 21, 2021 at 2:17 PM Evans Ye  wrote:
>
> > Hi folks,
> >
> > I recently came across AWS Graviton and it seems to be an Arm based
> chip. I
> > wonder do our Arm folks know more about this? Shall our arm based
> > distribution able to run on that chip directly? OR, whether we can
> > support Graviton to further expand our impact to big data world?
> >
>
>
> --
> IRC: ganeshraju@#linaro on irc.freenode.ne t
>

[ANNOUNCE] Kengo Seki is now officially the chair of Apache Bigtop

[Jun HE]

Hi all,

I'm very pleased to announce that, with ASF board's approval at April's
meeting, Kengo Seki is now officially appointed as the chair of Apache
Bigtop. Please join me in congratulating Kengo Seki!

Regards,

Jun

Re: PPC CI server failure

[Jun HE]

Awesome! Looking forward to its back to CI.
Thanks a lot for helping on this, Asanjar!

Regards,

Jun

MrAsanjar  于2021年3月29日周一 上午10:18写道：

> Hi old friends :)
> We should have a ppc64le VM back online sometime this week. I'll keep you
> all posted.
>
> On Thu, Nov 19, 2020 at 9:05 PM Evans Ye  wrote:
>
> > Hi rbkrishn,
> >
> > Would you mind to comment whether those PPC servers for Bigtop CI can be
> > brought up and unlock our release process?
> > Thanks!
> >
> > Best,
> > Evans
> >
> > Kengo Seki  於 2020年11月18日 週三 上午7:26寫道：
> >
> > > Thank you for checking, Evans and Amir!
> > >
> > > Kengo Seki 
> > >
> > > On Wed, Nov 18, 2020 at 2:09 AM Evans Ye  wrote:
> > > >
> > > > Thank you, Amir.
> > > >
> > > > MrAsanjar  於 2020年11月18日 週三 00:39 寫道：
> > > >
> > > > > Hi Evans, let me check with IBM again.
> > > > >
> > > > >
> > > > > On Mon, Nov 16, 2020 at 9:08 PM Evans Ye 
> wrote:
> > > > >
> > > > > > Hi Amir,
> > > > > >
> > > > > > We're planning Bigtop 1.5 release and if we don't have the CI
> nodes
> > > for
> > > > > > PPC, we're not able to release 1.5 with PPC supported.
> > > > > > Could you help to confirm again? Thanks!
> > > > > >
> > > > > > Best,
> > > > > > Evans Ye
> > > > > >
> > > > > >
> > > > > >
> > > > > > MrAsanjar  於 2020年9月17日 週四 下午8:56寫道：
> > > > > >
> > > > > > > I have informed IBM management regarding the situation, waiting
> > > for a
> > > > > > > reply.
> > > > > > >
> > > > > > > On Thu, Sep 17, 2020 at 3:47 AM Evans Ye 
> > > wrote:
> > > > > > >
> > > > > > > > Ok. Thanks for doing this to get the ball rolling.
> > > > > > > >
> > > > > > > > Kengo Seki  於 2020年9月17日 週四 10:29 寫道：
> > > > > > > >
> > > > > > > > > Thank you for your help, Amir!
> > > > > > > > > It's just a heads-up, I temporarily disabled builds for ppc
> > in
> > > the
> > > > > > > > > following Jenkins jobs so that they can finish.
> > > > > > > > >
> > > > > > > > > * Docker-Puppet-Trunk
> > > > > > > > > * Docker-Puppet-Trunk-pull
> > > > > > > > > * Docker-Toolchain-Trunk
> > > > > > > > > * Docker-Toolchain-Trunk-pull
> > > > > > > > >
> > > > > > > > > * Bigtop-trunk-packages
> > > > > > > > > * Bigtop-trunk-repos
> > > > > > > > >
> > > > > > > > > * Remove-All-Docker-Containers-Except-Nexus
> > > > > > > > > * Remove-Dangling-Docker-Images
> > > > > > > > > * Remove-Inactive-Containers
> > > > > > > > >
> > > > > > > > > Kengo Seki 
> > > > > > > > >
> > > > > > > > > On Wed, Sep 16, 2020 at 7:35 PM Evans Ye <
> evan...@apache.org
> > >
> > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Awesome! Nice to hear from you, buddy!
> > > > > > > > > >
> > > > > > > > > > MrAsanjar  於 2020年9月16日 週三 上午3:54寫道：
> > > > > > > > > >
> > > > > > > > > > > Hi Evans,
> > > > > > > > > > > Let me see what I can do. Give me 24 hr :)
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Sep 15, 2020 at 10:51 AM Evans Ye <
> > > evan...@apache.org>
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Yes. I think the action is correct. However [2] might
> > be
> > > a
> > > > > > > > different
> > > > > > > > > > > thing
> > > > > > > > > > > > for PPC integration in Hadoop.
> > > > > > > > > > > >
> > > > > > > > > > > > Amir,
> > > > > > > > > > > > Could you confirm?
> > > > > > > > > > > >
> > > > > > > > > > > > Kengo Seki  於 2020年9月14日 週一
> > 下午9:56寫道：
> > > > > > > > > > > >
> > > > > > > > > > > >> Thank you for the advice, Evans!
> > > > > > > > > > > >> Let me confirm about "PPC machine owners". According
> > to
> > > > > Amir's
> > > > > > > > JIRA
> > > > > > > > > > > >> issues [1][2] and the powered-by list in the OSU
> site
> > > [3],
> > > > > > we're
> > > > > > > > > using
> > > > > > > > > > > >> a VM hosted by OSU OSL, right?
> > > > > > > > > > > >> If it's correct, I'm going to ask them for help via
> > > > > > > > > > > >> powerdev-requ...@osuosl.org.
> > > > > > > > > > > >>
> > > > > > > > > > > >> [1]:
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://issues.apache.org/jira/browse/INFRA-11467?focusedCommentId=15300982=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-15300982
> > > > > > > > > > > >> [2]:
> > https://issues.apache.org/jira/browse/INFRA-12014
> > > > > > > > > > > >> [3]:
> > > > > > > > > > >
> > > > > > >
> > > https://osuosl.org/services/powerdev/current-projects/#foss-projects
> > > > > > > > > > > >>
> > > > > > > > > > > >> Kengo Seki 
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >> On Mon, Sep 14, 2020 at 2:06 PM Evans Ye <
> > > > > evan...@apache.org>
> > > > > > > > > wrote:
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > I'd suggest to reach out to PPC machine owners.
> > Worst
> > > case
> > > > > > Is
> > > > > > > we
> > > > > > > > > can
> > > > > > > > > > > >> > temporary  drop the PPC support to move the
> release
> > > > > forward.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Kengo 

[jira] [Created] (BIGTOP-3526) Remove phantomjs depdency from yar-ui

[Jun He (Jira)]

Jun He created BIGTOP-3526:
--

 Summary: Remove phantomjs depdency from yar-ui
 Key: BIGTOP-3526
 URL: https://issues.apache.org/jira/browse/BIGTOP-3526
 Project: Bigtop
  Issue Type: Bug
  Components: hadoop
Reporter: Jun He
Assignee: Jun He


Haoop Yarn-UI depends on phantomjs (optional) while there is not prebuilt 
binary other than x86 platform. And this will cause hadoop build failure as 
yarn-ui has been enabled by BIGTOP-3456.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (BIGTOP-3524) Hadoop 3.2.2 build failure on Arm

[Jun He (Jira)]

Jun He created BIGTOP-3524:
--

 Summary: Hadoop 3.2.2 build failure on Arm
 Key: BIGTOP-3524
 URL: https://issues.apache.org/jira/browse/BIGTOP-3524
 Project: Bigtop
  Issue Type: Bug
  Components: hadoop
Reporter: Jun He
Assignee: Jun He


Hadoop 3.2.2 build is failed on Arm due to missing protobuf header file.
{code:java}
[WARNING] make[3]: Leaving directory 
'/var/lib/jenkins/bigtop/output/hadoop/hadoop-3.2.2/hadoop-hdfs-project/hadoop-hdfs-native-client/target'
[WARNING] Makefile:140: recipe for target 'all' failed
[WARNING] make[2]: Leaving directory 
'/var/lib/jenkins/bigtop/output/hadoop/hadoop-3.2.2/hadoop-hdfs-project/hadoop-hdfs-native-client/target'
[WARNING] In file included from 
/usr/local/include/google/protobuf/stubs/once.h:81:0,
[WARNING]  from 
/var/lib/jenkins/bigtop/output/hadoop/hadoop-3.2.2/hadoop-hdfs-project/hadoop-hdfs-native-client/target/main/native/libhdfspp/lib/proto/IpcConnectionContext.pb.cc:10:
[WARNING] /usr/local/include/google/protobuf/stubs/atomicops.h:188:10: fatal 
error: google/protobuf/stubs/atomicops_internals_generic_gcc.h: No such file or 
directory
[WARNING]  #include 
[WARNING]   ^
[WARNING] compilation terminated.
[WARNING] make[4]: *** 
[main/native/libhdfspp/lib/proto/CMakeFiles/proto_obj.dir/IpcConnectionContext.pb.cc.o]
 Error 1
[WARNING] make[4]: *** Waiting for unfinished jobs
[WARNING] In file included from 
/usr/local/include/google/protobuf/stubs/once.h:81:0,
[WARNING]  from 
/var/lib/jenkins/bigtop/output/hadoop/hadoop-3.2.2/hadoop-hdfs-project/hadoop-hdfs-native-client/target/main/native/libhdfspp/lib/proto/ProtobufRpcEngine.pb.cc:10:
[WARNING] /usr/local/include/google/protobuf/stubs/atomicops.h:188:10: fatal 
error: google/protobuf/stubs/atomicops_internals_generic_gcc.h: No such file or 
directory
[WARNING]  #include 
[WARNING]   ^
[WARNING] compilation terminated.
[WARNING] make[4]: *** 
[main/native/libhdfspp/lib/proto/CMakeFiles/proto_obj.dir/ProtobufRpcEngine.pb.cc.o]
 Error 1
[WARNING] In file included from 
/usr/local/include/google/protobuf/stubs/once.h:81:0,
[WARNING]  from 
/var/lib/jenkins/bigtop/output/hadoop/hadoop-3.2.2/hadoop-hdfs-project/hadoop-hdfs-native-client/target/main/native/libhdfspp/lib/proto/xattr.pb.cc:10:
[WARNING] /usr/local/include/google/protobuf/stubs/atomicops.h:188:10: fatal 
error: google/protobuf/stubs/atomicops_internals_generic_gcc.h: No such file or 
directory
[WARNING]  #include 
[WARNING]   ^
[WARNING] compilation terminated.
[WARNING] make[4]: *** 
[main/native/libhdfspp/lib/proto/CMakeFiles/proto_obj.dir/xattr.pb.cc.o] Error 1
[WARNING] In file included from 
/usr/local/include/google/protobuf/stubs/once.h:81:0,
[WARNING]  from 
/var/lib/jenkins/bigtop/output/hadoop/hadoop-3.2.2/hadoop-hdfs-project/hadoop-hdfs-native-client/target/main/native/libhdfspp/lib/proto/erasurecoding.pb.cc:10:
[WARNING] /usr/local/include/google/protobuf/stubs/atomicops.h:188:10: fatal 
error: google/protobuf/stubs/atomicops_internals_generic_gcc.h: No such file or 
directory
[WARNING]  #include 
[WARNING]   ^
[WARNING] compilation terminated.
[WARNING] make[4]: *** 
[main/native/libhdfspp/lib/proto/CMakeFiles/proto_obj.dir/erasurecoding.pb.cc.o]
 Error 1
[WARNING] In file included from 
/usr/local/include/google/protobuf/stubs/once.h:81:0,
[WARNING]  from 
/var/lib/jenkins/bigtop/output/hadoop/hadoop-3.2.2/hadoop-hdfs-project/hadoop-hdfs-native-client/target/main/native/libhdfspp/lib/proto/Security.pb.cc:10:
[WARNING] /usr/local/include/google/protobuf/stubs/atomicops.h:188:10: fatal 
error: google/protobuf/stubs/atomicops_internals_generic_gcc.h: No such file or 
directory
[WARNING]  #include 
[WARNING]   ^
[WARNING] compilation terminated.
[WARNING] make[4]: *** 
[main/native/libhdfspp/lib/proto/CMakeFiles/proto_obj.dir/Security.pb.cc.o] 
Error 1
[WARNING] In file included from 
/usr/local/include/google/protobuf/stubs/once.h:81:0,
[WARNING]  from 
/var/lib/jenkins/bigtop/output/hadoop/hadoop-3.2.2/hadoop-hdfs-project/hadoop-hdfs-native-client/target/main/native/libhdfspp/lib/proto/HAServiceProtocol.pb.cc:10:
[WARNING] /usr/local/include/google/protobuf/stubs/atomicops.h:188:10: fatal 
error: google/protobuf/stubs/atomicops_internals_generic_gcc.h: No such file or 
directory
[WARNING]  #include 
[WARNING]   ^
[WARNING] compilation terminated.
[WARNING] make[4]: *** 
[main/native/libhdfspp/lib/proto/CMakeFiles/proto_obj.dir

[jira] [Created] (BIGTOP-3507) Solr: CVE-2020-13957 mitigation backport

[Jun He (Jira)]

Jun He created BIGTOP-3507:
--

 Summary: Solr: CVE-2020-13957 mitigation backport
 Key: BIGTOP-3507
 URL: https://issues.apache.org/jira/browse/BIGTOP-3507
 Project: Bigtop
  Issue Type: Improvement
  Components: solr
Reporter: Jun He


There is a [CVE security 
beach|https://lucene.apache.org/solr/security.html#cve-2020-13957-the-checks-added-to-unauthenticated-configset-uploads-in-apache-solr-can-be-circumvented]
 reported for solr on quite a few versions, where v6.6.6 is also affected.

Will backport upstream fix 
[SOLR-14663|https://issues.apache.org/jira/browse/SOLR-14663]  for this to 
v6.6.6 in solr component.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (BIGTOP-3485) Bump Solr to v8.7.0

[Jun He (Jira)]

Jun He created BIGTOP-3485:
--

 Summary: Bump Solr to v8.7.0
 Key: BIGTOP-3485
 URL: https://issues.apache.org/jira/browse/BIGTOP-3485
 Project: Bigtop
  Issue Type: Bug
  Components: solr
Reporter: Jun He


To bump version to v8.7.0 which is released on 2020-11-03.

This version has a lot of bugfix and security mitigation patches. Should be 
used to replace current v6.6.6



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] New Bigtop PMC member: Yuqi Gu

[Jun HE]

Welcome on board, Yuqi!
Thanks for those contributions and looking forward to more to come, :)

Evans Ye  于2021年1月7日周四 上午1:48写道：

> On behalf of the Apache Bigtop PMC, I am pleased to announce that
> Yuqi Gu has been elected to the Bigtop Project Management
> Committee. We appreciate Yuqi's contributions thus far, and look
> forward to his continued involvement with his new role at Apache Bigtop.
>
> Please join me in congratulating Yuqi Gu!
>

One Arm node in CI is done

[Jun HE]

Hi folks,

The node docker-slave-arm-4 is temporary offline for some SW issues. We're
working on this. It might need reinstall OS. If that is the case, it may
take a bit longer to bring it back online as Jenkins reconfiguration is
needed.

Regards,

Jun

Re: [ANNOUNCE] Apache Bigtop 1.5.0 released

[Jun HE]

Awesome!
Thanks to Kengo for leading the release process, and all the contributors
to make this happen!

Kengo Seki  于2020年12月16日周三 下午9:01写道：

> On behalf of the Apache Bigtop team, I'd love to announce the general
> availability of the Bigtop 1.5.0 release.
>
> The release is available here:
>   https://bigtop.apache.org/download.html#releases
>
> A few highlights of this release include:
>   * Four different Linux distributions are newly supported: CentOS 8,
> Debian 10, Fedora 31, and Ubuntu 18.04
>   * Bigtop Mpack is added as a new feature, that enables users to
> deploy Bigtop components via Apache Ambari
>   * Livy and ELK stack (Elasticsearch, Logstash, Kibana) are newly
> added as components
>   * Many component upgrades, e.g., Hadoop 2.10.1, Spark 2.4.5, HBase
> 1.5.0, Hive 2.3.6, Kafka 2.4.0, Zeppelin 0.8.2
>
> With Bigtop 1.5.0 the community continues to deliver the most advanced
> big data stack to date. More details about 1.5.0 release are here:
>   https://bigtop.apache.org/release-notes.html
>
> Deploying Bigtop is easy: grab the repo/list file for your favorite
> Linux distribution:
>   https://www.apache.org/dyn/closer.lua/bigtop/bigtop-1.5.0/repos/
> and you'll be running your very own bigdata cluster in no time!
>
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at:
>   https://bigtop.apache.org
>
> Lastly, I want to emphasize that this is a collaborative work done by
> project contributors and other communities,
> who continue to devote time to make Bigtop a better software. Thank
> you all for making this release possible!
>
> Regards,
> Kengo Seki, Bigtop 1.5.0 Release Manager
>

Re: [VOTE] Release Bigtop version 1.5.0

[Jun HE]

Verified following items on Arm64:
* Verified bigtop-puppet and bigtop-slaves images build
* Manually run deployment of hadoop, hbase, kafka, spark, zookeeper, solr,
flink, elasticsearch on Ubuntu-18.04/Debian-10
* Run smoke tests of hadoop, hbase, kafka, spark, zookeeper, solr, flink,
elasticsearch using ./docker-hadoop.sh on Ubuntu-18.04/Debian-10

I'm +1(non-binding) to accept RC0 as 1.5.0 release.

Thanks a lot for Kengo leading this release and all folks contributed to it!

Regards,

Jun

Evans Ye  于2020年12月14日周一 下午1:29写道：

> To unlock the release. I'm +1(binding) to accept RC0 as 1.5.0 release:
>
>- Checked gpg signature, sha512 sum, sha256 sum
>- build
>   - Manual test zookeeper-pkg-ind (with default bigtop-slave images)
>- provision
>   - Manual test docker provisioner with default centos-7 and debian-10
>   config (with bigtop-puppet images)
>
> Since build, provision, and smoke test features are guarded by our CI
> already. I think just running through the checking as an end user is
> sufficient.
>
> Evans
>
> Kengo Seki  於 2020年12月14日 週一 上午9:28寫道：
>
> > Luca,
> >
> > Thank you for the detailed checks!
> > That's really helpful and informative, so I added some of your
> > findings to the cwiki.
> >
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Overview+of+Bigtop+1.5.0+Support+Matrix
> >
> > Kengo Seki 
> >
> > On Sun, Dec 13, 2020 at 5:40 PM Luca Toscano 
> > wrote:
> > >
> > > On Mon, Dec 7, 2020 at 12:47 AM Kengo Seki  wrote:
> > > >
> > > > This is the vote for release 1.5.0 of Apache Bigtop.
> > > >
> > > > [x] +1, accept RC0 as the official 1.5.0 release of Apache Bigtop
> > >
> > > (Community vote)
> > >
> > > I completed my tests, the new release didn't raise any blocking issue.
> > > My tests were for this use case:
> > > - Hadoop test cluster with Hive/Oozie/Spark/Jupyter/Hue on bare metal
> > hosts.
> > > - HDFS and Yarn HA setup (including journal nodes).
> > > - Bigtop 1.4 for Debian 9 for the majority of the nodes, and a client
> > > node on Debian 10.
> > > - In place upgrade from Bigtop 1.4 to 1.5 (upgrading the packages
> > > without removing the old ones first).
> > >
> > > I have tested an in place upgrade, trying to rollback/re-deploy before
> > > the HDFS finalize step, all good. Some interesting things that I
> > > noticed or had to fix in my config after the upgrade:
> > > - the HDFS Namenodes seem to be smarter when dealing with a rollback.
> > > While testing CDH -> Bigtop 1.4 (hdfs 2.6.0 -> 2.8.5) I had to follow
> > > the upstream guidelines and explicitly issue commands to rollback the
> > > hdfs state, but when testing this release it was not needed anymore.
> > > I'll dig a big more into this, but overall the upgrade/rollback
> > > procedure seemed smoother.
> > > - our Hadoop clusters never used the
> > > "yarn.resourcemanager.webapp.address.$rm-id" settings in yarn-site.xml
> > > (that seem to have a good default), but on Bigtop 1.5 it lead to NPE
> > > while running map-reduce jobs, like described in
> > > https://issues.apache.org/jira/browse/YARN-8056. Once I set the value,
> > > everything worked as expected.
> > > - while upgrading the packages, I had a little hiccup with oozie since
> > > the version on Bigtop 1.4 is higher than 1.5 (4.3.0-3 vs 4.3.0-1), so
> > > it needed a downgrade. Nothing really major, just mentioning it.
> > >
> > > About https://dist.apache.org/repos/dist/dev/bigtop/bigtop-1.5.0-RC0:
> > > - verified .asc/.shaXXX, all good.
> > > - all debian packages (9 and 10 versions) worked fine, and Kengo
> > > Seki's gpg signature was ok as well.
> > >
> > > Thanks a lot for this release!
> > >
> > > Luca
> >
>

Re: 1.5.0 release update and concern about ppc64le

[Jun HE]

Hi Kengo,

It's back online now. Sorry for any inconvenience.

Regards,

Jun

Jun HE  于2020年11月27日周五 下午1:45写道：

> Hi Kengo,
>
> Let me have a look and get back to you then.
>
> Regards,
>
> Jun
>
> Kengo Seki  于2020年11月27日周五 下午12:49写道：
>
>> We should do so for now unfortunately. Once the PPC server is back,
>> I'm going to proceed the release process for ppc ASAP.
>> BTW, docker-slave-arm-5 is required to build packages for ARM but
>> seems to be down. Would you take a look, Jun and Yuqi?
>>
>> Kengo Seki 
>>
>>
>> On Fri, Nov 27, 2020 at 10:03 AM Evans Ye  wrote:
>> >
>> > Hi Kengo,
>> >
>> > Seems that there's no response from IBM. I guess we can only move on w/o
>> > PPC supported in 1.5. What do you say?
>> >
>> > Evans
>> >
>> > Evans Ye  於 2020年11月17日 週二 下午12:15寫道：
>> >
>> > > Let's wait for 2~3 days to see whether IBM guys respond.
>> > > If no, unfortunately we can only release w/o PPC supported.
>> > >
>> > > Kengo Seki  於 2020年11月17日 週二 上午7:49寫道：
>> > >
>> > >> We've finally resolved all issues for the 1.5.0 release [1].
>> > >> For a few components (e.g., Kibana (BIGTOP-3428), and maybe also
>> > >> Livy), their smoke tests didn't succeed with all platforms/distros on
>> > >> CI yet, but we confirmed that they worked on our local environment.
>> > >> So I think they are caused by a CI-specific environmental problem and
>> > >> not blockers for this release.
>> > >>
>> > >> I'm going to do the following items this week.
>> > >>
>> > >> * Recreate CI worker nodes following Evans' advice (thanks a lot!),
>> > >> because some hosts are frequently running short of disk capacity and
>> > >> going down. It makes the build unstable.
>> > >> * Run the packaging and deployment/smoke test CI jobs, and ensure the
>> > >> current status is as listed in [2].
>> > >> * Then continue the release process from step 3 in the "How to
>> > >> release" document [3], and hopefully cut a release candidate within
>> > >> this week.
>> > >>
>> > >> My concern is about the ppc64le platform, because I can't build
>> > >> packages without that CI server.
>> > >> Is there any response from the IBM management, Amir? And if it seems
>> > >> difficult to get it back in the near future, is it OK that we publish
>> > >> this release without ppc64le (and do it later once the server comes
>> > >> back in the future)?
>> > >>
>> > >> [1]:
>> > >>
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20BIGTOP%20AND%20resolution%20%3D%20Unresolved%20AND%20fixVersion%20%3D%201.5.0
>> > >> [2]:
>> > >>
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=165225375
>> > >> [3]:
>> https://cwiki.apache.org/confluence/display/BIGTOP/How+to+release
>> > >>
>> > >> Kengo Seki 
>> > >>
>> > >
>>
>

Re: 1.5.0 release update and concern about ppc64le

[Jun HE]

Hi Kengo,

Let me have a look and get back to you then.

Regards,

Jun

Kengo Seki  于2020年11月27日周五 下午12:49写道：

> We should do so for now unfortunately. Once the PPC server is back,
> I'm going to proceed the release process for ppc ASAP.
> BTW, docker-slave-arm-5 is required to build packages for ARM but
> seems to be down. Would you take a look, Jun and Yuqi?
>
> Kengo Seki 
>
>
> On Fri, Nov 27, 2020 at 10:03 AM Evans Ye  wrote:
> >
> > Hi Kengo,
> >
> > Seems that there's no response from IBM. I guess we can only move on w/o
> > PPC supported in 1.5. What do you say?
> >
> > Evans
> >
> > Evans Ye  於 2020年11月17日 週二 下午12:15寫道：
> >
> > > Let's wait for 2~3 days to see whether IBM guys respond.
> > > If no, unfortunately we can only release w/o PPC supported.
> > >
> > > Kengo Seki  於 2020年11月17日 週二 上午7:49寫道：
> > >
> > >> We've finally resolved all issues for the 1.5.0 release [1].
> > >> For a few components (e.g., Kibana (BIGTOP-3428), and maybe also
> > >> Livy), their smoke tests didn't succeed with all platforms/distros on
> > >> CI yet, but we confirmed that they worked on our local environment.
> > >> So I think they are caused by a CI-specific environmental problem and
> > >> not blockers for this release.
> > >>
> > >> I'm going to do the following items this week.
> > >>
> > >> * Recreate CI worker nodes following Evans' advice (thanks a lot!),
> > >> because some hosts are frequently running short of disk capacity and
> > >> going down. It makes the build unstable.
> > >> * Run the packaging and deployment/smoke test CI jobs, and ensure the
> > >> current status is as listed in [2].
> > >> * Then continue the release process from step 3 in the "How to
> > >> release" document [3], and hopefully cut a release candidate within
> > >> this week.
> > >>
> > >> My concern is about the ppc64le platform, because I can't build
> > >> packages without that CI server.
> > >> Is there any response from the IBM management, Amir? And if it seems
> > >> difficult to get it back in the near future, is it OK that we publish
> > >> this release without ppc64le (and do it later once the server comes
> > >> back in the future)?
> > >>
> > >> [1]:
> > >>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20BIGTOP%20AND%20resolution%20%3D%20Unresolved%20AND%20fixVersion%20%3D%201.5.0
> > >> [2]:
> > >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=165225375
> > >> [3]:
> https://cwiki.apache.org/confluence/display/BIGTOP/How+to+release
> > >>
> > >> Kengo Seki 
> > >>
> > >
>

Re: Update

[Jun HE]

Thanks a lot for the update, Olaf!

Olaf Flebbe  于2020年10月31日周六 上午3:24写道：

> Hi,
>
> All machines patched. Jenkins and it plugins are updated:
>
> Things to be noted:
>
> * Slave 2 seems to be in serious problems. The disk image seems to be
> corrupt, I would say:
> One of the problems: docker does not start any more.
> Is there anything important on it ? If yes please contact me. I would
> recommend to set up slave2 from scratch again.
>
> * There was a warning regarding Copy Artifacts Plugin. It now imposes
> stricter rules. Not sure if there is a job depending on it.
>
> * I removed the CVS plugin.
>
> Everything else seem to working as usual.
>
> Best,
> Olaf
>
>
>
>
> > Am 30.10.2020 um 19:09 schrieb Olaf Flebbe :
> >
> > Hi,
> >
> > I am doing an update of the machines in CI . Seems a couple of security
> fixes are to be applied.
> >
> > Olaf
>
>

Re: ARM CI server seems to be down

[Jun HE]

Hi Kengo,

Sorry for late response. Yuqi had recovered that node back to normal the
day before yesterday, but forgot to send out the notification.
It should be all fixed now. Pls let me know if there is any problem. Thanks
a lot.

Regards,

Jun

Kengo Seki  于2020年10月24日周六 上午7:57写道：

> Hi Jun and Yuqi,
>
> I've just noticed that the docker-slave-arm-4 server is down.
> https://ci.bigtop.apache.org/computer/docker-slave-arm-4/
>
> I couldn't log in to that node using the same SSH key with which I can
> do to docker-slave-arm-5.
> Would you take a look?
>
> Kengo Seki 
>

Re: Current status of the next release

[Jun HE]

 The status looks very promising.
Thanks for leading this, Kengo, and thanks for the contributions from all
of the developers!

I can help to check those failed smoke tests this weekend and maybe take
some. :)

Evans Ye  于2020年9月24日周四 下午2:31写道：

> Great! Thanks for the update.
>
> For the release there is still a lot of manual effort needed. Let me know
> when you'd like to start the process so I can help.
>
> Kengo Seki  於 2020年9月23日 週三 22:49 寫道：
>
> > Hi everyone,
> >
> > Let me share the current status of the next release.
> > We've steadily moved forward and almost reached the goal.
> > I really appreciate your cooperation and contribution.
> >
> > The attached image is the current packaging matrix [1].
> > As you can see, we've succeeded on building most of all combinations
> > between components and distros/platforms, though ppc64 is temporarily
> > disabled due to the machine problem.
> >
> > The only build failure is Spark on CentOS 7 for arm64 (and ppc64),
> > but it's tracked as BIGTOP-3397 and is supposed to be resolved
> > once the next version of snappy-java is released and we add a small patch
> > for Spark that uses it (or, if that release doesn't make it, we can
> > build it ourselves).
> >
> > So, our remaining task is to complete the deployment and smoke test
> matrix
> > [2].
> > Currently, the following components have not succeeded on any
> > distro/platform yet,
> > so I'm going to begin with these components.
> >
> > - GPDB
> > - Livy
> > - Oozie (already filed as BIGTOP-3406)
> > - QFS
> > - Spark
> >
> > I'm planning to cut the release branch once all components passed
> > their smoke test
> > on at least one distro/platform for deb and rpm respectively
> > (e.g., it's regarded as OK if the smoke test succeeds on x86_64/CentOS
> > 7 and arm64/Debian 9),
> > and hopefully, I'd like to release v1.5.0 within this October.
> > Let me know if this schedule is not convenient for you :)
> >
> > [1]: https://ci.bigtop.apache.org/job/Bigtop-trunk-packages/
> > [2]: https://ci.bigtop.apache.org/job/Bigtop-trunk-smoke-tests/
> >
> > Kengo Seki 
> >
>

Re: Updated the ci.bigtop.apache.org certificate

[Jun HE]

Thanks, Kengo!

To make the life easier, would it be possible to use automatically renewal,
like crontab, or something like this (
https://github.com/acmesh-official/acme.sh)?

Olaf Flebbe  于2020年6月11日周四 下午9:40写道：

> Thanks!
>
> > Am 11.06.2020 um 14:24 schrieb Kengo Seki :
> >
> > Just for your information, I've just renewed the certificate for
> > ci.bigtop.apache.org in accordance with [1].
> > Let me know if you find something wrong with it.
> >
> > [1]:
> https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+CI+Setup+Guide#BigtopCISetupGuide-Renewingthecert
> >
> > Kengo Seki 
>
>

New Committer: Yuqi Gu

[Jun HE]

The Project Management Committee (PMC) for Apache Bigtop has asked Yuqi Gu
to become a committer and we are pleased to announce that he has accepted!

His ASF account has been created as: guy...@apache.org

Being a committer enables easier contribution to the project since there is
no
need to go via the proxy patch submission process. This should enable better
productivity.

Apache Bigtop practices CTR (commit-then-review) development process which
means that you can checkin the code without a +1 from a committer. This
doesn't
mean that you're encouraged to do so. Instead, we strongly recommend you to
seek for feedback before you commit. Please read the document before you
act:

* https://cwiki.apache.org/confluence/display/BIGTOP/CTR+Model

If in doubt, exercise your best judgement and don't hesitate to ask
questions on
this very mailing list.

We're happy to have you on board and looking for many more contributions to
come. Please join me in congratulating Yuqi Gu!

Regards,

Jun (on behalf of the ASF BigTop PMC)

Re: [DISCUSS] [VOTE] Release Ambari-Mpack as a seperate top component in Bigtop

[Jun HE]

Thanks Olaf for raising those comments/questions. That really helps us to
think more about how this Bigtop-MPack thing could go further.

I think Yuqi is trying to decouple Bigtop-Mpack from Ambari component
itself in terms of code/patch/building/packaging, so developer can easily
contribute to it (MPack). The discussion on the PR thread comes to create a
new component just like bigtop-jsvc/bigtop-utils things. So for such change
(adding new component) it might worth discussing and voting in community
besides simply reviewing on some patches.

Those items Olaf listed are definitely important to push Bigtop forward,
especiall CI from my point of view,  and we will focus on them. While MPack
also gives Bigtop community possibility to extend usage and improve
influence as the only open source Big Data SW stack, and there are
potential contributors here, maybe we just get the ball rolling.

Then I have a question same as Evans for Yuqi, Matt, or others interested
in: are you willing to be the maintainer going forward? :)

Regards,

Jun


Matt Andruff  于2020年6月8日周一 下午10:32写道：

> What problem/case is this additional component trying to solve?
>
>  My opinion is that it's an enable meant tool to make it easy to use
> BigTop by having it's configuration managed by Ambari.  Now is a great time
> for making this move as HDP is no longer going to be managed by a company,
> so BigTop could replace that role in Ambari.
>
>  This mpack allows BigTop to be installed into Ambari, and allows
> Ambari to manage the hadoop packages.  Effectively, it would increase the
> reach of BigTop to be able to be installed into and managed by Ambari.
>
> I hear that you see a lot of other priorities and they sound like good
> priorities.  I'm not able to contribute to those priorities as I don't know
> enough in that area/do not understand what's required.  I would contribute
> right now to Ambari MPack as it's more in my wheelhouse.
>
> Hope that helps
>
> Matt
>
> On Sun, Jun 7, 2020 at 12:07 PM Olaf Flebbe  wrote:
>
> > Hi,
> >
> > Sorry, I am super lost about the implications of that vote.
> >
> > Why do we need a vote in the first place?
> > I am familiar with votes for additional committers, PMC additions and
> > releases. For these votes we have the concept of binding votes by PMC
> > members.
> >
> > Why should I care about this ?
> > Does it have legal implications?
> >
> > Who will maintain this ?
> > I am a bit surprised that we do anything with ambari, since we not even
> > have a maintainer for it
> > grep ambari MAINTAINERS.txt -> nothing
> >
> > What problem/case is this additional component trying to solve?
> > If you are asking for a vote I would like to have a more elaborate
> > explanation. I looked at the linked ticket: Sorry I am totally lost about
> > what is this about. If you would have simply done it, I wouldn’t have
> cared.
> >
> > As I understand it should somehow enable more contributions.
> > Really? MPack is a amabari only technology, that seems not to even be
> > defined properly. How could this ever end in more contributions to
> Bigtop ?
> >
> > What will happen if someone is -1 on this vote ?
> >
> > If a contributor cares about adopting Bigtop to new business cases,
> that’s
> > very fine to me. Please go ahead. But please don’t ask for a vote. All
> > committers still have the chance to reject patches.
> >
> > Generally I am concerned that we care about an (from Bigtop side)
> > unmaintained project while neglecting our core infrastructure and
> packages
> > at the same time.
> > 1) The ci.bigtop.apache.org certificate expired again
> > 2) The bigtop-trunk-packages do not reflect the architectures we ought to
> > support (ubuntu-18.04 for instance)
> > 3) Trunk-packages for amd64 looks like garbage
> > 4) Still no-one working on Hadoop3
> > 5) CI needs to be redone (with pipeline jobs) .
> > IMO we might have wrong priorities.
> >
> > Best,
> > Olaf
> >
> >
> >
> >
> > > Am 04.06.2020 um 19:04 schrieb Ganesh Raju :
> > >
> > > +1
> > >
> > > On Thu, Jun 4, 2020 at 12:01 PM Matt Andruff <
> m...@andruffsolutions.com>
> > > wrote:
> > >
> > >> +1
> > >>
> > >> On Thu., Jun. 4, 2020, 02:35 Yuqi Gu,  wrote:
> > >>
> > >>> Hi folks,
> > >>>
> > >>> As the discussion with Evans Ye, Jun He, and Matt Andruff (from
> Ambari)
> > >> on
> > >>> https://github.com/apache/bigtop/pull/555#issuecomment-635228180:
> > >>

Re: [DISCUSS] [VOTE] Release Ambari-Mpack as a seperate top component in Bigtop

[Jun HE]

+1

Youngwoo Kim (김영우)  于2020年6月4日周四 下午3:31写道：

> +1
>
> Youngwoo
>
> 2020년 6월 4일 (목) 오후 3:35, Yuqi Gu 님이 작성:
>
> > Hi folks,
> >
> > As the discussion with Evans Ye, Jun He, and Matt Andruff (from Ambari)
> on
> > https://github.com/apache/bigtop/pull/555#issuecomment-635228180:
> >
> > For users / Bigtopers could easily modify Mapck and make contributions to
> > it like bugfix and adding more services,
> > How about to decouple Mpack from Ambari and put Mpack as a standalone
> > component in Bigtop, like *bigtop-packages/src/common/bigtop-mpack*?
> > If so, it's convenient for users to get and install Mpack rpm/deb
> packages
> > in their own Ambari cluster.
> >
> > The VOTE:
> > [ ] +1, Agree to put Mpack as a standalone component.
> > [ ] +0, I don't care either way.
> > [ ] -1, do *not * Agree.
> >
> > Look forward to your comments and feedback, thanks.
> >
> > BRs,
> > Yuqi
> >
>

Re: [DISCUSS] [VOTE] Publish release-1.5.0 without ppc64le support for Zeppelin

[Jun HE]

I'm +1 for this.

Yuqi Gu  于2020年4月26日周日 下午1:18写道：

> Yes, the VOTE is to just drop Zeppelin support on PPC64LE.
>
> On Sun, 26 Apr 2020 at 12:39, Evans Ye  wrote:
>
> > So the vote is to just drop Zeppelin support on PPC64LE, am I right?
> > If so I'm a +1.
> > In 1.4.0 release, we've a support matrix[1] so that it's more clear to
> user
> > what's there in Bigtop.
> >
> > BTW PPC folks: if you'd like to work on the fix and make Zeppelin
> > supported. Feel free to -1 on this.
> > With limited resources, we'd like to focus on the core features. However
> if
> > someone would like to work on it its definitely welcomed.
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Overview+of+Bigtop+1.4.0+Support+Matrix
> >
> > Evans
> >
> >
> > Yuqi Gu  於 2020年4月26日 週日 上午10:32寫道：
> >
> > > Hi folks,
> > >
> > > Let's go back to the discussion on PR #631
> > >  for BIGTOP-3327.
> > >
> > > The Zeppelin build failed on ppc64le due to the dependency issue that
> the
> > > ppc64le binaries of protoc-gen-grpc-java are still not available.
> > >
> > > To make the new release move onward, How about to drop the ppc64le
> > support
> > > for Zeppelin in the next release?
> > >
> > > The VOTE:
> > > [ ] +1, accept dropping the ppc64le support,
> > > [ ] +0, I don't care either way,
> > > [ ] -1, do not accept dropping the ppc64le support, or wanna be a
> > > volunteer to
> > > work on it.
> > >
> > >
> > > BR,
> > > Yuqi
> > >
> >
>

Re: [DISCUSS] Next Generation Bigtop CI Infra

[Jun HE]

Thanks for starting this conversation on CI evolvement, Evans.
Yes, our CI is pretty manual managed with few automatic and version
controlled. That did bring me "special" experience during v1.3 release, ;)

For #1, as we need folks from Linaro and ppc to help the maintenance, the
manual management may not be easy changed.
For #2 and #3, these two things can be resolved with code
(scripts/makefiles/...) and pipeline job as Olaf mentioned, and they can be
hosted as a part of Bigtop, which user can easily duplicate our ci
environment. We have exprience to do smiliar things in Jenkins.
For the docker deployment stuff, it has various limitations. But to move
Bigtop to a container based world, I think it still worths considering.

Just my opinion. :)

One mroe question is: does anyone have experience on Github Actions as CI?
It looks interesting as it can link with your own node with github's CI
flow.

Olaf Flebbe  于2020年4月27日周一 上午1:35写道：

> Hi,
>
> have some limited experience with Jenkins2 aka Pipeline. IMO this resolves
> 3) quite nicely.
>
> Regarding 2) imo our deployment with docker compose is dead-end. Docker is
> simply not designed to support whole machine scenarios white systemd as a
> init process and predictive networking. while lxc seems to be more
> suitable, I think we should not hardcode a toy/test environment with lxc.
> Would propose Terraform here, that ideally should enable our users to use
> other infra as well.
>
> Regarding 1) If we have Terraform for 2) we can even manage our own build
> infra with that (minus the ppc64le and arm64 slaves).
>
> What do you think?
>
> Olaf
>
>
> Von meinem iPad gesendet
>
> > Am 26.04.2020 um 13:58 schrieb Evans Ye :
> >
> > Hi folks,
> >
> > I've been thinking about the revamp of bigtop's CI infra.
> > Currently what we have is purely a Jenkins cluster with manual managed:
> > 1. hardware/machine resource
> > 2. build env: docker, docker compose are all manual installed
> > 3. job definition: all jobs are manual created and the definition is not
> > tracked w/ version control system.
> >
> > I think for 2,3 there're some tools/solutions that can better manage and
> > automate the things. So that user can just take the code and build their
> CI
> > directly.
> >
> > At Yahoo! there's a tool called screwdriver[1] which is a framework to do
> > CICD, however I don't think it fits our scenario well as we're not facing
> > production CD. Probably Jenkins 2.0 is more suitable? Anyone has
> experience
> > and suggestions?
> >
> > [1] https://screwdriver.cd/
> >
> > Evans
>

Re: New Bigtop PMC member: Kengo Seki

[Jun HE]

So excited to have you on board, Kengo! Congratulations!

Jay Vyas  于2020年4月19日周日 下午8:07写道：

> Likewise thanks Kengo !
>
> > On Apr 19, 2020, at 1:54 AM, Konstantin Boudnik  wrote:
> >
> > Great to have you on board!
> >
> >> On 2020-04-18 02:05, Olaf Flebbe wrote:
> >> On behalf of the Apache Bigtop Project Management Committee, I am
> pleased
> >> to announce that Kengo Seki has accepted our invitation to join the
> >> Bigtop PMC.
> >>
> >> Please join me in congratulating Kengo!
> >>
> >> Regards,
> >>   Olaf  (on behalf of the ASF BigTop PMC)
> >>
> >>
> > --
> > With regards,
> >  Cos
> >
>

Re: New Committer: Masatake Iwasaki

[Jun HE]

Awesome! Welcome Masatake!

Konstantin Boudnik  于2020年4月19日周日 下午1:54写道：

> Well done, welcome! Please keep up your contributions, man!
>
> On 2020-04-18 01:58, Olaf Flebbe wrote:
> > The Project Management Committee (PMC) for Apache Bigtop has asked
> > Masatake Iwasaki to become a committer and we are pleased to announce
> > that he has accepted!
> >
> > Being a committer enables easier contribution to the project since there
> is
> > no need to go via the proxy patch submission process. This should enable
> better
> > productivity.
> >
> > Apache Bigtop practices CTR (commit-then-review) development process
> which
> > means that you can checkin the code without a +1 from a committer. This
> > doesn't mean that you're encouraged to do so. Instead, we strongly
> recommend you to
> > seek for feedback before you commit. Please read the document before you
> > act:
> >
> > * https://github.com/apache/bigtop#ctr-model
> >
> > If in doubt, exercise your best judgement and don't hesitate to ask
> > questions on the dev@bigtop.apache.org mailing list.
> >
> > We're happy to have you on board and looking for many more contributions
> to
> > come. Please join me in congratulating Masatake Iwasaki !
> >
> > Regards,
> >Olaf  (on behalf of the ASF BigTop PMC)
> >
> >
> --
> With regards,
>Cos
>
>

Re: Test your account

[Jun HE]

Welcome Masatake! It's great to have you on board.

To update the web page, you may follow description here:
https://cwiki.apache.org/confluence/display/BIGTOP/Deploying+Bigtop%27s+Apache+Website


Masatake Iwasaki  于2020年4月18日周六 上午8:56写道：

> Hi Olaf,
>
> Thanks for setting this up.
>
> I added myself to the member page and pushed.
>
> On Confluence, it looks like I can not edit existing pages,
> while I can create a new page.
>
> Regards,
> Masatake Iwasaki
>
> On Sat, Apr 18, 2020 at 4:20 AM Olaf Flebbe  wrote:
> >
> > Hi Masatake,
> >
> > please test your account by adding yourself to the member page (Like
> > BIGTOP-2622).
> >
> > You should be able to update the page (see
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Deploying+Bigtop%27s+Apache+Website
> )
> >
> > Best,
> > Olaf
>

Re: Arm4 build slave is down

[Jun HE]

Sorry for any inconveince caused by this. This node is back on line now.
Thanks a lot for help from Linaro folks. :)

Evans Ye  于2020年4月8日周三 上午3:50写道：

> Fellow Arm folks,
>
> It seems ARM4 build slave is down. Could you help to fix it? Thanks!
>
> Evans
>

[jira] [Created] (BIGTOP-3329) Bump ambari to v2.7.5

[Jun He (Jira)]

Jun He created BIGTOP-3329:
--

 Summary: Bump ambari to v2.7.5
 Key: BIGTOP-3329
 URL: https://issues.apache.org/jira/browse/BIGTOP-3329
 Project: Bigtop
  Issue Type: Improvement
  Components: general
Affects Versions: 1.4.0
Reporter: Jun He


Current ambari uses v2.6.1 which was released in Dec 2017.
Bump this to latest v2.7.5



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Drop Apache Tajo, Apache Hama, and Apache Apex

[Jun HE]

I'm +1 to remove these inactive projects.
I remember that there was a discussion to leave those projects to another
branch, and related dev work depends on volunteered maintainer/contributor.
So maybe we can start this after v1.5 is released.


Masatake Iwasaki  于2020年3月17日周二 上午7:56写道：

> I'm +1 on removing (effectively) retired products.
>
> While deprecating in the next release and
> removing in after the next might be usual way,
> I think it would be ok to remove them in 1.5.0.
> Each product in Bigtop has dedicated subdirectories.
> Removing (and reverting the removal) should be easy
> unless it is depended by other products.
>
> Thanks,
> Masatake Iwasaki
>
>
> On 3/16/20 15:13, Konstantin Boudnik wrote:
> > Thanks for bringing this up, Evans!
> >
> > I think this trimming makes total sense and we were through similar
> > discussions before. In fact there's a consensus that we simply can't
> > support all the projects that seem to be nice, especially those that
> > weren't been active in 3+ years.
> >
> > I am in favor of trimming the stack down. Unless, there's a maintainer
> > in the community who wants to step up and help.
> >
> > --
> > Regards,
> >   Cos
> >
> > On 3/16/20 1:05 PM, Evans Ye wrote:
> >> Hi folks,
> >>
> >> I'd like to raise the discussion about the cleanup of supported
> >> component
> >> matrix.
> >>
> >>  From [1] the latest release of Tajo is 0.11.3 which was out on
> >> 2016-05-18.
> >> Since then there're no further release. Similarly on Hama side the
> >> latest
> >> release is on Jan 28, 2016 [2]. Apex is on Attic so there's no chance to
> >> get new release further [3].
> >>
> >> Though we'd like to support as many projects as possible, there's simply
> >> not enough effort and resource to do it. Dropping them can also let
> >> us more
> >> focus on what's valuable to the users. What do you think?
> >>
> >> Kengo Seki
> >> As 1.5 RM what do you think? I think if the discussion is positive this
> >> will be in 1.5 release.
> >>
> >> [1] https://tajo.apache.org/
> >> [2] https://hama.apache.org/
> >> [3] https://apex.apache.org/
> >>
> >> Evans
> >>
>
>

[jira] [Created] (BIGTOP-3316) Fix HBase build failure on Arm platform

[Jun He (Jira)]

Jun He created BIGTOP-3316:
--

 Summary: Fix HBase build failure on Arm platform
 Key: BIGTOP-3316
 URL: https://issues.apache.org/jira/browse/BIGTOP-3316
 Project: Bigtop
  Issue Type: Bug
  Components: build, hbase
Reporter: Jun He
Assignee: Jun He


After HBase is bumped to 1.5.0, jruby patch command in build script doesn't 
apply to this version.
This task will refine patch command to make it more versatile to various HBase 
versions.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] Jun He is now officially the chair of Apache Bigtop

[Jun HE]

Thank you all so much!  And really appreciate Youngwoo's great work!

I'm thrilled to have opportunity to serve this very open community. Would
love to seek for tips and suggestions on this big chair. :)




Yuqi Gu  于2020年2月22日周六 下午3:25写道：

> Congratulations!
>
> On Sat, 22 Feb 2020 at 14:18, Evans Ye  wrote:
>
> > Hi all,
> >
> > With ASF board's approval on February's meeting, Jun He is now officially
> > appointed as the chair of Apache Bigtop. Please join me in congratulating
> > Jun He!
> >
> > Meanwhile, thanks to our ex-chair Youngwoo Kim's service, We really glad
> to
> > have you together building a great community at Bigtop. What a success!
> >
> > Evans
> >
> > Matt Sicker  於 2020年2月20日 週四 上午4:13寫道：
> >
> > > Dear new PMC chairs,
> > >
> > > Congratulations on your new role at Apache. I've changed your LDAP
> > > privileges to reflect your new status.
> > >
> > > Please read this and update the foundation records:
> > >
> > >
> >
> https://svn.apache.org/repos/private/foundation/officers/advice-for-new-pmc-chairs.txt
> > >
> > > Warm regards,
> > >
> > > Matt Sicker
> > >
> >
>

[jira] [Created] (BIGTOP-3314) Fix flnk build failure with upgraded hadoop version

[Jun He (Jira)]

Jun He created BIGTOP-3314:
--

 Summary: Fix flnk build failure with upgraded hadoop version
 Key: BIGTOP-3314
 URL: https://issues.apache.org/jira/browse/BIGTOP-3314
 Project: Bigtop
  Issue Type: Bug
Reporter: Jun He
Assignee: Jun He


Flink 1.6.4 failed to build with new hadoop 2.10.0.

Error log as following:
{code:java}
[ERROR] COMPILATION ERROR : 
[INFO] -
[ERROR] 
/bigtop/build/flink/rpm/BUILD/flink-1.6.4/flink-yarn/src/test/java/org/apache/flink/yarn/AbstractYarnClusterTest.java:[89,41]
 no suitable method found for 
newInstance(org.apache.hadoop.yarn.api.records.ApplicationId,org.apache.hadoop.yarn.api.records.ApplicationAttemptId,java.lang.String,java.lang.String,java.lang.String,java.lang.String,int,,org.apache.hadoop.yarn.api.records.YarnApplicationState,,,long,long,org.apache.hadoop.yarn.api.records.FinalApplicationStatus,,,float,,)
method 
org.apache.hadoop.yarn.api.records.ApplicationReport.newInstance(org.apache.hadoop.yarn.api.records.ApplicationId,org.apache.hadoop.yarn.api.records.ApplicationAttemptId,java.lang.String,java.lang.String,java.lang.String,java.lang.String,int,org.apache.hadoop.yarn.api.records.Token,org.apache.hadoop.yarn.api.records.YarnApplicationState,java.lang.String,java.lang.String,long,long,long,org.apache.hadoop.yarn.api.records.FinalApplicationStatus,org.apache.hadoop.yarn.api.records.ApplicationResourceUsageReport,java.lang.String,float,java.lang.String,org.apache.hadoop.yarn.api.records.Token)
 is not applicable
  (actual and formal argument lists differ in length)
method 
org.apache.hadoop.yarn.api.records.ApplicationReport.newInstance(org.apache.hadoop.yarn.api.records.ApplicationId,org.apache.hadoop.yarn.api.records.ApplicationAttemptId,java.lang.String,java.lang.String,java.lang.String,java.lang.String,int,org.apache.hadoop.yarn.api.records.Token,org.apache.hadoop.yarn.api.records.YarnApplicationState,java.lang.String,java.lang.String,long,long,long,long,org.apache.hadoop.yarn.api.records.FinalApplicationStatus,org.apache.hadoop.yarn.api.records.ApplicationResourceUsageReport,java.lang.String,float,java.lang.String,org.apache.hadoop.yarn.api.records.Token)
 is not applicable
  (actual and formal argument lists differ in length)
method 
org.apache.hadoop.yarn.api.records.ApplicationReport.newInstance(org.apache.hadoop.yarn.api.records.ApplicationId,org.apache.hadoop.yarn.api.records.ApplicationAttemptId,java.lang.String,java.lang.String,java.lang.String,java.lang.String,int,org.apache.hadoop.yarn.api.records.Token,org.apache.hadoop.yarn.api.records.YarnApplicationState,java.lang.String,java.lang.String,long,long,org.apache.hadoop.yarn.api.records.FinalApplicationStatus,org.apache.hadoop.yarn.api.records.ApplicationResourceUsageReport,java.lang.String,float,java.lang.String,org.apache.hadoop.yarn.api.records.Token,java.util.Set,boolean,org.apache.hadoop.yarn.api.records.Priority,java.lang.String,java.lang.String)
 is not applicable
  (actual and formal argument lists differ in length)
method 
org.apache.hadoop.yarn.api.records.ApplicationReport.newInstance(org.apache.hadoop.yarn.api.records.ApplicationId,org.apache.hadoop.yarn.api.records.ApplicationAttemptId,java.lang.String,java.lang.String,java.lang.String,java.lang.String,int,org.apache.hadoop.yarn.api.records.Token,org.apache.hadoop.yarn.api.records.YarnApplicationState,java.lang.String,java.lang.String,long,long,long,org.apache.hadoop.yarn.api.records.FinalApplicationStatus,org.apache.hadoop.yarn.api.records.ApplicationResourceUsageReport,java.lang.String,float,java.lang.String,org.apache.hadoop.yarn.api.records.Token,java.util.Set,boolean,org.apache.hadoop.yarn.api.records.Priority,java.lang.String,java.lang.String)
 is not applicable
  (actual and formal argument lists differ in length)
method 
org.apache.hadoop.yarn.api.records.ApplicationReport.newInstance(org.apache.hadoop.yarn.api.records.ApplicationId,org.apache.hadoop.yarn.api.records.ApplicationAttemptId,java.lang.String,java.lang.String,java.lang.String,java.lang.String,int,org.apache.hadoop.yarn.api.records.Token,org.apache.hadoop.yarn.api.records.YarnApplicationState,java.lang.String,java.lang.String,long,long,long,long,org.apache.hadoop.yarn.api.records.FinalApplicationStatus,org.apache.hadoop.yarn.api.records.ApplicationResourceUsageReport,java.lang.String,float,java.lang.String,org.apache.hadoop.yarn.api.records.Token,java.util.Set,boolean,org.apache.hadoop.yarn.api.records.Priority,java.lang.String,java.lang.String)
 is not applicable
  (actual and formal argument lists differ in length)
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (BIGTOP-3313) Bump solor to 6.6.6

[Jun He (Jira)]

Jun He created BIGTOP-3313:
--

 Summary: Bump solor to 6.6.6
 Key: BIGTOP-3313
 URL: https://issues.apache.org/jira/browse/BIGTOP-3313
 Project: Bigtop
  Issue Type: Bug
Reporter: Jun He
Assignee: Jun He


Bump solr to latest version of 6.x branch.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (BIGTOP-3312) Fix Ambari build failure due to maven repo requires https now

[Jun He (Jira)]

Jun He created BIGTOP-3312:
--

 Summary: Fix Ambari build failure due to maven repo requires https 
now
 Key: BIGTOP-3312
 URL: https://issues.apache.org/jira/browse/BIGTOP-3312
 Project: Bigtop
  Issue Type: Bug
Reporter: Jun He
Assignee: Jun He


Maven repo has been upgrade to accpet https access only.

The URLs defined in ambari pom.xml need to be changed to https.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (BIGTOP-3311) Alluxio build failure with upgraded hadoop version

[Jun He (Jira)]

Jun He created BIGTOP-3311:
--

 Summary: Alluxio build failure with upgraded hadoop version
 Key: BIGTOP-3311
 URL: https://issues.apache.org/jira/browse/BIGTOP-3311
 Project: Bigtop
  Issue Type: Bug
Reporter: Jun He
Assignee: Jun He


Alluxio build failed after hadoop is bumpped to 2.10.0. The reason is the 
version check regex in integration/yarn/pom.xml doesn't cover version higher 
that 2.9.x.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (BIGTOP-3310) URL of maven and ant package list page is changed

[Jun He (Jira)]

Jun He created BIGTOP-3310:
--

 Summary: URL of maven and ant package list page is changed
 Key: BIGTOP-3310
 URL: https://issues.apache.org/jira/browse/BIGTOP-3310
 Project: Bigtop
  Issue Type: Bug
Reporter: Jun He
Assignee: Jun He


bigtop_toolchain uses packages list page hosted on https://www.apache.org/dist/ 
to detect latest version of maven and ant.
This url has been moved to https://downloads.apache.org/ and latest_*_binary 
functions need to be updated to reflect this.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Arm5 build slave is down

[Jun HE]

Hi, Evans,

It's back online now. Sorry for any inconvenience caused by this.

Regards,

Jun

Jun HE  于2020年2月17日周一 上午9:30写道：

> Hi, Evans,
>
> I'll look into this and get back to you.
>
> Regards,
>
> Jun
>
> Evans Ye  于2020年2月17日周一 上午1:16写道：
>
>> Hi folks,
>>
>> Our Arm5 build slave is down[1]. I've lost the way to get into the server.
>> So not sure what's going on inside. Could any one help to fix it or grant
>> me the access right? Thanks!
>>
>> Evans
>>
>> [1] https://ci.bigtop.apache.org/computer/
>>
>> ---JENKINS LOG---
>>
>> SSHLauncher{host='213.146.141.103', port=22,
>> credentialsId='e4343a01-5ca9-4290-8f79-d04ec1a95be2', jvmOptions='',
>> javaPath='', prefixStartSlaveCmd='', suffixStartSlaveCmd='',
>> launchTimeoutSeconds=210, maxNumRetries=10, retryWaitTime=15,
>>
>> sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.NonVerifyingKeyVerificationStrategy,
>> tcpNoDelay=true, trackCredentials=true}
>> [02/09/20 16:35:05] [SSH] Opening SSH connection to 213.146.141.103:22.
>> No route to host (Host unreachable)
>> SSH Connection failed with IOException: "No route to host (Host
>> unreachable)", retrying in 15 seconds.  There are 10 more retries
>> left.
>>
>> No route to host (Host unreachable)
>> SSH Connection failed with IOException: "No route to host (Host
>> unreachable)", retrying in 15 seconds.  There are 9 more retries left.
>>
>> No route to host (Host unreachable)
>> SSH Connection failed with IOException: "No route to host (Host
>> unreachable)", retrying in 15 seconds.  There are 8 more retries left.
>>
>> No route to host (Host unreachable)
>> SSH Connection failed with IOException: "No route to host (Host
>> unreachable)", retrying in 15 seconds.  There are 7 more retries left.
>>
>> No route to host (Host unreachable)
>> SSH Connection failed with IOException: "No route to host (Host
>> unreachable)", retrying in 15 seconds.  There are 6 more retries left.
>>
>> No route to host (Host unreachable)
>> SSH Connection failed with IOException: "No route to host (Host
>> unreachable)", retrying in 15 seconds.  There are 5 more retries left.
>>
>> No route to host (Host unreachable)
>> SSH Connection failed with IOException: "No route to host (Host
>> unreachable)", retrying in 15 seconds.  There are 4 more retries left.
>>
>> No route to host (Host unreachable)
>> SSH Connection failed with IOException: "No route to host (Host
>> unreachable)", retrying in 15 seconds.  There are 3 more retries left.
>>
>> No route to host (Host unreachable)
>> SSH Connection failed with IOException: "No route to host (Host
>> unreachable)", retrying in 15 seconds.  There are 2 more retries left.
>>
>> No route to host (Host unreachable)
>> SSH Connection failed with IOException: "No route to host (Host
>> unreachable)", retrying in 15 seconds.  There are 1 more retries left.
>>
>> No route to host (Host unreachable)
>> SSH Connection failed with IOException: "No route to host (Host
>> unreachable)".
>> java.io.IOException: There was a problem while connecting to
>> 213.146.141.103:22
>> at com.trilead.ssh2.Connection.connect(Connection.java:758)
>> at
>> hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:1175)
>> at
>> hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:846)
>> at
>> hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:833)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>> Caused by: java.net.NoRouteToHostException: No route to host (Host
>> unreachable)
>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>> at java.net
>> .AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>> at java.net
>> .AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>> at java.net
>> .AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>> at java.net.Socket.connect(Socket.java:589)
>> at
>> com.trilead.ssh2.transport.TransportManager.establishConnection(TransportManager.java:367)
>> at
>> com.trilead.ssh2.transport.TransportManager.initialize(TransportManager.java:480)
>> at com.trilead.ssh2.Connection.connect(Connection.java:698)
>> ... 7 more
>> [02/09/20 16:38:07] Launch failed - cleaning up connection
>> [02/09/20 16:38:07] [SSH] Connection closed.
>>
>

Re: Arm5 build slave is down

[Jun HE]

Hi, Evans,

I'll look into this and get back to you.

Regards,

Jun

Evans Ye  于2020年2月17日周一 上午1:16写道：

> Hi folks,
>
> Our Arm5 build slave is down[1]. I've lost the way to get into the server.
> So not sure what's going on inside. Could any one help to fix it or grant
> me the access right? Thanks!
>
> Evans
>
> [1] https://ci.bigtop.apache.org/computer/
>
> ---JENKINS LOG---
>
> SSHLauncher{host='213.146.141.103', port=22,
> credentialsId='e4343a01-5ca9-4290-8f79-d04ec1a95be2', jvmOptions='',
> javaPath='', prefixStartSlaveCmd='', suffixStartSlaveCmd='',
> launchTimeoutSeconds=210, maxNumRetries=10, retryWaitTime=15,
>
> sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.NonVerifyingKeyVerificationStrategy,
> tcpNoDelay=true, trackCredentials=true}
> [02/09/20 16:35:05] [SSH] Opening SSH connection to 213.146.141.103:22.
> No route to host (Host unreachable)
> SSH Connection failed with IOException: "No route to host (Host
> unreachable)", retrying in 15 seconds.  There are 10 more retries
> left.
>
> No route to host (Host unreachable)
> SSH Connection failed with IOException: "No route to host (Host
> unreachable)", retrying in 15 seconds.  There are 9 more retries left.
>
> No route to host (Host unreachable)
> SSH Connection failed with IOException: "No route to host (Host
> unreachable)", retrying in 15 seconds.  There are 8 more retries left.
>
> No route to host (Host unreachable)
> SSH Connection failed with IOException: "No route to host (Host
> unreachable)", retrying in 15 seconds.  There are 7 more retries left.
>
> No route to host (Host unreachable)
> SSH Connection failed with IOException: "No route to host (Host
> unreachable)", retrying in 15 seconds.  There are 6 more retries left.
>
> No route to host (Host unreachable)
> SSH Connection failed with IOException: "No route to host (Host
> unreachable)", retrying in 15 seconds.  There are 5 more retries left.
>
> No route to host (Host unreachable)
> SSH Connection failed with IOException: "No route to host (Host
> unreachable)", retrying in 15 seconds.  There are 4 more retries left.
>
> No route to host (Host unreachable)
> SSH Connection failed with IOException: "No route to host (Host
> unreachable)", retrying in 15 seconds.  There are 3 more retries left.
>
> No route to host (Host unreachable)
> SSH Connection failed with IOException: "No route to host (Host
> unreachable)", retrying in 15 seconds.  There are 2 more retries left.
>
> No route to host (Host unreachable)
> SSH Connection failed with IOException: "No route to host (Host
> unreachable)", retrying in 15 seconds.  There are 1 more retries left.
>
> No route to host (Host unreachable)
> SSH Connection failed with IOException: "No route to host (Host
> unreachable)".
> java.io.IOException: There was a problem while connecting to
> 213.146.141.103:22
> at com.trilead.ssh2.Connection.connect(Connection.java:758)
> at
> hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:1175)
> at
> hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:846)
> at
> hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:833)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.net.NoRouteToHostException: No route to host (Host
> unreachable)
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net
> .AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
> at java.net
> .AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> at java.net
> .AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:589)
> at
> com.trilead.ssh2.transport.TransportManager.establishConnection(TransportManager.java:367)
> at
> com.trilead.ssh2.transport.TransportManager.initialize(TransportManager.java:480)
> at com.trilead.ssh2.Connection.connect(Connection.java:698)
> ... 7 more
> [02/09/20 16:38:07] Launch failed - cleaning up connection
> [02/09/20 16:38:07] [SSH] Connection closed.
>

Re: [DISCUSS] Target distros on the 1.5.0 release

[Jun HE]

Hi, Kengo,

I just noticed you updated the BOM of Bigtop v1.5 on BIGTOP-3123.

One thing I'd like to know your and other folks' thought on Elasticsearch
as Bigtop component.
There is a ticket (https://issues.apache.org/jira/browse/BIGTOP-3219) for
this. And I've finished most of the sub-works
(build/packaging/deployement). Patch could be found at:
https://github.com/apache/bigtop/pull/566
For the smoke test part I think it could be done in this weekend.

So if that Jira (BIGTOP-3219) can be done in next week, do you think it is
OK to include Elasticsearch-5.6.14 in v1.5?

Regards,

Jun

Kengo Seki  于2019年11月22日周五 上午12:09写道：

> Thanks for the comments, everyone!
> If there's still no objection in a few days, I'm going to update
> BIGTOP-3123's description.
>
> > 'contrib' module will be easier for us to maintain traditional distros
> and cnb.
>
> Agreed. I think that merging the cnb branch later will be a hard work too.
>
> > What do you think about the components?
> > Is there a list of components you'd like to upgrade?
>
> I'd like to upgrade the following components:
>
> - Zookeeper: 3.4.13
> - Hadoop: 3.2.1 (or 2.10.0 if packaging Hadoop 3 is too hard, as Olaf
> mentioned before)
> - HBase: 2.2.2
> - Hive: 3.1.2
> - Tez: 0.9.2
> - Spark: 2.4.4 (or 3.0.0, if GA is released before long)
> - Phoenix: 5.0.0
> - Kafka: 2.3.1
> - Ignite: 2.7.6
> - Zeppelin: 0.8.2
>
> My Teammates and I are trying to package them, and all of them
> are successfully built anyway. But we have not tested them yet,
> and I'm sure many problems will be found from now, just as Olaf
> already came across on Hadoop 3... :)
>
> > the community was lean to the direction of having important component
> better supported
> > instead of spending resources for 20~30 components.
>
> Totally agreed. If we succeed to package Hadoop 3,
> I'd like to drop inactive components which can't be built with it,
> at least for now.
>
> > Just one concern about the puppet recipes compatibility across multiple
> puppet versions
>
> Exactly. I think it's difficult to support Puppet 3, 4 and 5 with a
> single manifest or config file,
> so I'm thinking to create a new "puppet5" directory beside the
> existing "puppet" directory
> and put the manifests and config files for Puppet 5 into it (and when
> we drop the distros
> using Puppet 3 and 4 completely in the future, we can drop the
> existing "puppet" directory
> and promote "puppet5" to "puppet").
> If it doesn't work as expected, I'll ask you the possibility to drop
> old versions in the next release again.
>
> > one source of problems was using puppet-forge for instance for
> puppet-stdlib and puppet-apt,
>
> Yeah, puppet modules seem to be installed into /usr/share/puppet/modules
> via apt on Debian 9 and Ubuntu 16.04, while /etc/puppet/modules via
> `puppet module install` on CentOS 7, as you said.
> Maybe we have to consolidate them somehow, or specify both of them
> for `--modulepath` (I'm not sure if it works though), or choose either of
> them
> in accordance with the distro.
>
> Kengo Seki 
>
> On Thu, Nov 21, 2019 at 3:17 PM Olaf Flebbe  wrote:
> >
> > hi
> >
> > one source of problems was using puppet-forge for instance for
> puppet-stdlib and puppet-apt, since they require rather new versions. look
> out for 'puppet module install '.  While all distros using apt do have
> matching prepackaged versions in their repository.
> >
> > other was different search paths of all these versions. we never fixed
> that consistently.
> >
> > olaf
> >
> > Von meinem iPad gesendet
> >
> > > Am 21.11.2019 um 03:43 schrieb Jun HE :
> > >
> > > I'm fine with the new distros list. Just one concern about the puppet
> > > recipes compatibility across multiple puppet versions (3.8.5 for
> > > Ubuntu-16.04, 4.8.2 for Debian-9, and 5.x for other new distros). I
> didn't
> > > do any investigation yet. If such issues arise, I'll vote for drop
> distros
> > > with older puppet.
> > >
> > > Evans Ye  于2019年11月21日周四 上午1:51写道：
> > >
> > >> Fine by me for the OS side.
> > >> What do you think about the components? Is there a list of components
> you'd
> > >> like to upgrade?
> > >> We can target a subset of current supported matrix as we previously
> > >> discussed about this and the community was lean to the direction of
> having
> > >> important component better supported instead of spending resources for
> > >> 20~30 components.
> > >>
> > >> Youngwoo Kim (

[jira] [Created] (BIGTOP-3274) Protocbuf build failed due to git.savannah.gnu.org is not accessible

[Jun He (Jira)]

Jun He created BIGTOP-3274:
--

 Summary: Protocbuf build failed due to git.savannah.gnu.org is not 
accessible
 Key: BIGTOP-3274
 URL: https://issues.apache.org/jira/browse/BIGTOP-3274
 Project: Bigtop
  Issue Type: Bug
Reporter: Jun He


In BIGTOP-2698, protocbuf is set to build from source with patche and new 
config.guess from git.savannah.gnu.org.
Recently when I was trying to build toolchain locally, I noticed that 
confgi.guess was failed to download due to git.savannah.gnu.org was not 
accessible.

To fix this I think we can use a included config.guess, just like the patch 
(0001-Add-generic-GCC-support-for-atomic-operations.patch), instead of 
downloading it every time.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Target distros on the 1.5.0 release

[Jun HE]

I'm fine with the new distros list. Just one concern about the puppet
recipes compatibility across multiple puppet versions (3.8.5 for
Ubuntu-16.04, 4.8.2 for Debian-9, and 5.x for other new distros). I didn't
do any investigation yet. If such issues arise, I'll vote for drop distros
with older puppet.

Evans Ye  于2019年11月21日周四 上午1:51写道：

> Fine by me for the OS side.
> What do you think about the components? Is there a list of components you'd
> like to upgrade?
> We can target a subset of current supported matrix as we previously
> discussed about this and the community was lean to the direction of having
> important component better supported instead of spending resources for
> 20~30 components.
>
> Youngwoo Kim (김영우)  於 2019年11月20日 週三 上午9:41寫道：
>
> > Kengo,
> >
> > Looks good to me. I think puppet on CentOS 8 would be fine.
> >
> > On Cloud Native Bigtop, I believe we should consider that components as a
> > 'contrib' at this point.
> > I'm considering about Jay's idea, making 'CNB' on master as a contrib
> > module. A development branch is good but on our "two-tracks" development,
> > 'contrib' module will be easier for us to maintain traditional distros
> and
> > cnb.
> >
> > Thanks,
> > Youngwoo
> >
> > On Wed, Nov 20, 2019 at 9:28 AM Kengo Seki  wrote:
> >
> > > Hi folks,
> > >
> > > I'd like to discuss the target distros for the next 1.5.0 release [1],
> > > because over 1.5 years have passed since Ubuntu 18.04 was released
> > > and the next LTS will be released within half a year. In addition,
> > > Fedora 26 and openSUSE 42.3 have already been EOL'd.
> > >
> > > (I understand the "Cloud Native Bigtop" project is going on
> > > and am really looking forward to it, but my customers still requires
> > > the traditional software stack :)
> > >
> > > Based on the past discussion [2], here's my proposal:
> > >
> > > - Add Debian 10, Fedora 31 and Ubuntu 18.04 as the target distros
> > >   and use the puppet package provided by each distro, so that
> > >   we can support all CPU architectures (x86_64, aarch64, and ppc64le).
> > >   Their puppet versions are 5.4.0 (ubuntu) and 5.5.10 (debian and
> > fedora).
> > >
> > >   Keep Debian 9 and Ubuntu 16.04 since they are still in the support
> > > period.
> > >
> > >   Drop Fedora 26 since it has reached to the EOL on 2018-05-29.
> > >
> > > - Add CentOS 8. Unfortunately, that version doesn't seem to
> > >   provide the distro's puppet package, even including EPEL.
> > >   Even though, I'd like to support it since that distro
> > >   (and RHEL8) are widely used especially in enterprise systems.
> > >   So, as the next best option, how about using Puppet 5.5 provided by
> > >   Puppetlabs and only supporting the x86_64 architecture on this
> version?
> > >
> > >   Keep CentOS 7 since it's still in the support period.
> > >
> > > - Drop openSUSE 42.3 since it has reached to the EOL on 2019-07-01
> > >   and don't add a new version of that distro, as discussed in [2].
> > >
> > > To summarize the above, the supported distros and their versions
> > > in the 1.5.0 release are as follows:
> > >
> > > - CentOS 7, 8 (8 is only supported on x86_64)
> > > - Debian 9, 10
> > > - Fedora 31
> > > - Ubuntu 16.04, 18.04
> > >
> > > Does this sound reasonable? I'd appreciate any comments or suggestions.
> > >
> > > (Honestly, I'd actually like to drop CentOS 7, Debian 9, and Ubuntu
> > 16.04,
> > > so that we can consolidate the Puppet version to 5.x.
> > > But it may be too aggressive for users.)
> > >
> > > [1]: https://issues.apache.org/jira/browse/BIGTOP-3123
> > > [2]:
> > >
> >
> https://lists.apache.org/thread.html/26e14cf36e9cfd61e0de581ed83bf305565c2e65234f1ce3bfb97628@%3Cdev.bigtop.apache.org%3E
> > >
> > > Kengo Seki 
> > >
> >
>

[jira] [Created] (BIGTOP-3223) Add smoke tests for Elasticsearch

[Jun He (Jira)]

Jun He created BIGTOP-3223:
--

 Summary: Add smoke tests for Elasticsearch
 Key: BIGTOP-3223
 URL: https://issues.apache.org/jira/browse/BIGTOP-3223
 Project: Bigtop
  Issue Type: Sub-task
Reporter: Jun He


Implement smoke tests for Elasticsearch



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (BIGTOP-3222) Add puppet deployment for Elasticsearch

[Jun He (Jira)]

Jun He created BIGTOP-3222:
--

 Summary: Add puppet deployment for Elasticsearch
 Key: BIGTOP-3222
 URL: https://issues.apache.org/jira/browse/BIGTOP-3222
 Project: Bigtop
  Issue Type: Sub-task
  Components: deployment
Reporter: Jun He


Implement puppet deployment recipes for Elasticsearch



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (BIGTOP-3221) Add support for Elasticsearch rpm build

[Jun He (Jira)]

Jun He created BIGTOP-3221:
--

 Summary: Add support for Elasticsearch rpm build
 Key: BIGTOP-3221
 URL: https://issues.apache.org/jira/browse/BIGTOP-3221
 Project: Bigtop
  Issue Type: Sub-task
Reporter: Jun He


Add rpm packaging script for Elasticsearch artifacts



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (BIGTOP-3220) Add support for Elasticsearch deb build

[Jun He (Jira)]

Jun He created BIGTOP-3220:
--

 Summary: Add support for Elasticsearch deb build
 Key: BIGTOP-3220
 URL: https://issues.apache.org/jira/browse/BIGTOP-3220
 Project: Bigtop
  Issue Type: Sub-task
  Components: debian
Reporter: Jun He


Add debian packaging support for Elasticsearch artifacts.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (BIGTOP-3219) Add Elasticsearch as component

[Jun He (Jira)]

Jun He created BIGTOP-3219:
--

 Summary: Add Elasticsearch as component
 Key: BIGTOP-3219
 URL: https://issues.apache.org/jira/browse/BIGTOP-3219
 Project: Bigtop
  Issue Type: New Feature
  Components: blueprints, general
Reporter: Jun He


Elasticsearch is a search engine based on the Lucene library. It provides a 
distributed, multitenant-capable full-text search engine with an HTTP web 
interface and schema-free JSON documents.
The offcial website: https://www.elastic.co/
Source code repo: https://github.com/elastic/elasticsearch



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Re: Shall tajo

[Jun HE]

Super +1 for Bigtop 2.0 with modern components and K8S support! I cannot
wait to see Bigtop running on K8S...
So can someone help to create a branch-2.0 now? :P


Konstantin Boudnik  于2019年8月30日周五 上午2:24写道：

> Aah, k8s - thanks Jay!
>
> BTW, I believe we can do this in a less formal stuff (ie no VOTE): we have
> a
> discussion going on the version of Bigtop. How about we make it 2.0 and
> trim
> the BOM into the needed shape and form? If there's enough drive in the
> community to continue with 1.x line (1.4 and so on), it can be done as a
> parallel effort.
>
> Thoughts?
>   Cos
>
> On Fri, Aug 30, 2019 at 01:26AM, Evans Ye wrote:
> > I'm super +1 with K8S stuff and the core components idea!
> >
> > I'm with Cos and Jun. Removing those stuffs can ease our CI resource
> > utilization and yield more stable CI results. This is what I'm thinking
> how
> > to proceed based on the previous feedback that a drop of supporting
> > component should be carefully discussed
> >
> > 1. Spin up a vote to drop project XXX (if confident enough, put multiple
> > components here)
> > 2. If +3 binding votes with no -1 votes, it passes.
> > 3. Put up the PR for code removing
> > 4. Refine our CI settings (I can definitely help with this part)
> >
> > Best,
> > Evans
> >
> >
> >
> >
> > Jay Vyas  於 2019年8月29日 週四 下午7:26寫道：
> >
> > > First of all I’m all for dropping the old stuff.
> > >
> > > 1) My opinion - to further cos point - I think people running old stuff
> > > don’t really need version updates, or if they do, they don’t
> constitute a
> > > large audience , or should contribute them ok their own.
> > >
> > > 2) surprise surprise :):) here’s my k8s native. View - we should move
> the
> > > old components into sustaining mode, and rebuild a new bigtop focus
> solely
> > > on spark , NiFi , Presto (with a lot of attention to minimal standalone
> > > Hadoop w/ a Hive connector ) and target it at kubernetes native
> deployments
> > > :).
> > >
> > > I can make it so the k8s part is an impl detail so users can be mostly
> > > decoupled from it .
> > >
> > > I’ve been integration testing various things in the bigdata ecosystem
> on
> > > k8s and there is a lot of demand without anyone owning the integration.
> > >
> > > > On Aug 29, 2019, at 2:39 AM, Konstantin Boudnik 
> wrote:
> > > >
> > > > I am not sure about Giraph, Tajo and some others, but Sqoop seems to
> be
> > > user
> > > > around by people. So, if it isn't much of the burden for us - and it
> > > seems
> > > > pretty stable at the moment - I'd leave it.
> > > >
> > > > What I would think would makes sense to spend some of our efforts on
> is
> > > on
> > > > adding modern tooling like Nifi/Airflow into the mix. As you said -
> > > things
> > > > move forward pretty fast, and we seem to be sticking to the some of
> the
> > > old
> > > > stuff.
> > > >
> > > > Thanks for starting this discussion!
> > > >  Cos
> > > >
> > > >> On Wed, Aug 28, 2019 at 04:12PM, Jun HE wrote:
> > > >> Hi, folks,
> > > >>
> > > >> I went through current components Bigtop is supporting, and I
> noticed
> > > that
> > > >> these upstream projects haven't been released for quite a while:
> > > >> Apache Tajo:
> > > >>last release: release-0.11.3-rc0, May 11, 2016
> > > >> Apache Apex:
> > > >>last release: v3.7.0, Apr 27, 2018
> > > >> Apache Giraph:
> > > >>last release: rel/1.2.0-RC1, Oct 13 2016
> > > >> Apache Hama:
> > > >>last release: 0.7.1-RC2, Mar 12, 2016
> > > >> Apache Sqoop:
> > > >>last release: release-1.4.7-rc0, Dec 6, 2017;
> release-1.99.7-rc1, Jul
> > > >> 20, 2016
> > > >>
> > > >> And some of them seem to be in slow development:
> > > >> Apache Tajo:
> > > >>last commit: Jul 13, 2018
> > > >> Apache Apex:
> > > >>last commit: Jun 20, 2018
> > > >> Apache Hama:
> > > >>last commit: Jul 30, 2018
> > > >>
> > > >> So I'm wondering whether we should continue support for these
> components
> > > >> (or part of them) in next/future releases.
> > > >>
> > > >> I understand that similar topics were discussed before. But, you
> know,
> > > this
> > > >> is an quickly evolving world, maybe it worth another revisit now? ;)
> > > >>
> > > >> Regards,
> > > >>
> > > >> Jun
> > >
>

Shall tajo

[Jun HE]

Hi, folks,

I went through current components Bigtop is supporting, and I noticed that
these upstream projects haven't been released for quite a while:
Apache Tajo:
last release: release-0.11.3-rc0, May 11, 2016
Apache Apex:
last release: v3.7.0, Apr 27, 2018
Apache Giraph:
last release: rel/1.2.0-RC1, Oct 13 2016
Apache Hama:
last release: 0.7.1-RC2, Mar 12, 2016
Apache Sqoop:
last release: release-1.4.7-rc0, Dec 6, 2017; release-1.99.7-rc1, Jul
20, 2016

And some of them seem to be in slow development:
Apache Tajo:
last commit: Jul 13, 2018
Apache Apex:
last commit: Jun 20, 2018
Apache Hama:
last commit: Jul 30, 2018

So I'm wondering whether we should continue support for these components
(or part of them) in next/future releases.

I understand that similar topics were discussed before. But, you know, this
is an quickly evolving world, maybe it worth another revisit now? ;)

Regards,

Jun

Re: [ANNOUNCE] Apache Bigtop 1.4.0 released

[Jun HE]

Great! Thanks Evans for your hard work.

Evans Ye  于2019年6月19日周三 上午12:07写道：

> On behalf of the Apache Bigtop team, I'd love to announce the general
> availability of the Bigtop 1.4.0 release.
>
> The release is available here:
> https://bigtop.apache.org/download.html#releases
>
> A few highlights of this release include:
> - Integration Test Framework 2.0: one-stop integrated build and test
>   framework at a single entry: ./gradlew [1]
> - Newly developed Smoke Test CI Matrix to guard the quality of releases [2]
> - Hadoop 2.8.5, Spark 2.2.3, Flink 1.6.0, Alluxio 1.8.1 and more [3]
> - 100+ JIRAs are resolved in this release
>
> With Bigtop 1.4.0 the community continues to deliver the most advanced big
> data stack to date. More details about 1.4.0 release are here:
> https://bigtop.apache.org/release-notes.html
>
> Deploying Bigtop is easy: grab the repo/list file for your favorite Linux
> distribution:
>   https://www.apache.org/dyn/closer.lua/bigtop/bigtop-1.4.0/repos/
> and you'll be running your very own big data cluster in no time!
>
> We welcome your help and feedback. For more information on how to report
> problems, and to get involved, visit the project website at:
>   https://bigtop.apache.org
>
> Lastly, I want to emphasize that this is a collaborative work done by
> project
> contributors and other communities, who continue to devote time to make
> Bigtop a better software. Thank you all for making this release possible!
>
> Thanks,
> Evans Ye, Release Manager
>
> [1]
>
> https://cwiki.apache.org/confluence/display/BIGTOP/Quickstart+Guide%3A+Bigtop+Integration+Test+Framework+2.0
> [2]
> https://ci.bigtop.apache.org/view/Test/job/Bigtop-trunk-smoke-tests-1.4.0/
> [3] https://issues.apache.org/jira/browse/BIGTOP-3162
>

Re: [VOTE] Release Bigtop version 1.4.0 (Release Candidate 1)

[Jun HE]

+1.

Tested Ubuntu and Debian, and basically works.

Konstantin Boudnik  于2019年6月10日周一 上午4:55写道：

> Fellas, I just got back from a weekend trip and I didn't have time to do
> any release testing. I see there's enough votes already, but if it'd make
> sense to have an extra pair of eyes to look at it - I would be happy to. I
> need an extra day though.
>
> Thanks!
> --
> With regards,
>   Konstantin
>
> On June 5, 2019 1:29:00 AM GMT+03:00, Evans Ye  wrote:
> >Let's extend the vote to *Sunday, June 9th, 2019 at noon PDT*.
> >Let me know if it's still not enough time for you to evaluate the
> >release :)
> >
> >Youngwoo Kim (김영우)  於 2019年6月3日 週一 上午3:23寫道：
> >
> >> Sorry for the late, Evans.
> >>
> >> We need to extend time window for this vote. just a couple of days?
> >> I would like to test the release candidate when I come back to my
> >office.
> >>
> >> Thanks,
> >> Youngwoo
> >>
> >> On Mon, Jun 3, 2019 at 8:02 AM Evans Ye  wrote:
> >>
> >> > Hi folks,
> >> >
> >> > Any info I can provide and help you to evaluate the release
> >candidate?
> >> >
> >> > Evans Ye  於 2019年5月28日 週二 下午8:37寫道：
> >> >
> >> > > Hi folks,
> >> > >
> >> > > If you have voted RC0, this is pretty much the same except for
> >the fix
> >> to
> >> > > change docker image name from trunk to 1.4.0 :)
> >> > >
> >> > > Evans Ye 於 2019年5月23日 週四，下午4:12寫道：
> >> > >
> >> > >> BTW, binary artifacts for centos7, debian9, opensuse42.3 on PPC
> >are
> >> all
> >> > >> uploaded to S#. However only debian 9 seems good. Others
> >encountered
> >> > >> dependency issues when installing packages. We shall only mark
> >those
> >> > >> working Distros as supported. I'll make this clear in the
> >release
> >> note.
> >> > >>
> >> > >> Evans Ye  於 2019年5月23日 週四 下午8:59寫道：
> >> > >>
> >> > >>> Hi folks,
> >> > >>>
> >> > >>> This is the vote to release 1.4.0 of Apache Bigtop with Release
> >> > >>> Candidate 1.
> >> > >>>
> >> > >>> It fixes the following issues:
> >> > >>> *
> >> > >>>
> >> >
> >>
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12344113=12311420
> >> > >>>
> >> > >>> The vote will be going for at least 72 hours and will be closed
> >on
> >> > >>> Sunday,
> >> > >>> *May 26th, 2019 at noon PDT*.  Please download, test and vote
> >with
> >> > >>>
> >> > >>> [ ] +1, accept RC1 as the official 1.4.0 release of Apache
> >Bigtop
> >> > >>> [ ] +0, I don't care either way,
> >> > >>> [ ] -1, do not accept RC1 as the official 1.4.0 release of
> >Apache
> >> > >>> Bigtop, because...
> >> > >>>
> >> > >>> Change(s) compared to RC0:
> >> > >>> * Switch to 1.4.0 docker images for provisioner:
> >> > >>>
> >> >
> >>
> >
> https://github.com/apache/bigtop/commit/4921ebd737bacb48ad22c7e8b572185b572ad7b4
> >> > >>>
> >> > >>> Source and binary files:
> >> > >>> *
> >https://dist.apache.org/repos/dist/dev/bigtop/bigtop-1.4.0-RC1
> >> > >>>
> >> > >>> Maven staging repo:
> >> > >>> *
> >> > >>>
> >> >
> >https://repository.apache.org/content/repositories/orgapachebigtop-1023
> >> > >>>
> >> > >>> The git tag to be voted upon is release-1.4.0-RC1
> >> > >>>
> >> > >>> Bigtop's KEYS file containing PGP keys we use to sign the
> >release:
> >> > >>> * https://dist.apache.org/repos/dist/release/bigtop/KEYS
> >> > >>>
> >> > >>> Bigtop 1.4.0 CI result:
> >> > >>> * Packaging:
> >> > https://ci.bigtop.apache.org/view/Releases/job/Bigtop-1.4.0
> >> > >>> * Deploy & Test:
> >> > >>>
> >> >
> >>
> >https://ci.bigtop.apache.org/view/Test/job/Bigtop-trunk-smoke-tests-1.4.0
> >> > >>>
> >> > >>>
> >> > >>> Evans Ye, RM
> >> > >>>
> >> > >>
> >> >
> >>
>

Re: [VOTE] Release Bigtop version 1.4.0

[Jun HE]

I had run basic tests on both aarch64 and amd64. The major components look
fine.
Some smoke tests failed, like flink, but that is caused by smoke test it
self. So +1 from me.

Thanks for your great work, Evans.

Regards,

Jun

[jira] [Created] (BIGTOP-3216) Flink smoke test failed on Debian-9

[Jun He (JIRA)]

Jun He created BIGTOP-3216:
--

 Summary: Flink smoke test failed on Debian-9
 Key: BIGTOP-3216
 URL: https://issues.apache.org/jira/browse/BIGTOP-3216
 Project: Bigtop
  Issue Type: Bug
  Components: flink
Affects Versions: 1.4.0
Reporter: Jun He
Assignee: Jun He


Flink smoke test on debian-9 failed with following output:
{code:bash}
org.apache.bigtop.itest.hadoop.flink.TestFlink > testWordCountBatch 
STANDARD_ERROR
19/05/15 05:48:29 INFO lang.Object: flink run 
$FLINK_HOME/examples/batch/WordCount.jar --input hdfs:///flink/test.data 
--output hdfs:///tmp/result.txt

org.apache.bigtop.itest.hadoop.flink.TestFlink > testWordCountBatch SKIPPED

> Task :bigtop-tests:smoke-tests:flink:test FAILED
:bigtop-tests:smoke-tests:flink:test (Thread[Task worker for ':' Thread 
42,5,main]) completed. Took 5 mins 56.008 secs.

FAILURE: Build failed with an exception.
{code}

The investigation showed that flink cannot get defaultFS info. So explicitly 
add hdfs info could fix this test.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: R.I.P. mrdocs (1963–2019)

[Jun HE]

R.I.P, Peter.

Olaf Flebbe  于2019年3月23日周六 下午8:10写道：

> Ruhe in Frieden, Peter
>

Re: [DISCUSS] Jump to 1.4.0 release directly instead of 1.3.1

[Jun HE]

Awesome, and huge +1 to this!

Konstantin Boudnik  于2019年3月6日周三 下午9:11写道：

> Oh my, that's amazing! Thank you man!
> +1 on moving directly to 1.4
> --
> With regards,
>   Konstantin
>
> On March 5, 2019 12:58:44 AM GMT+03:00, Olaf Flebbe  wrote:
> >Hi Evans,
> >
> >You found and fixed a lot of bugs and made the smoke tests work, so a
> >big +1 .
> >
> >Olaf
> >
> >> Am 04.03.2019 um 08:12 schrieb Evans Ye :
> >>
> >> -1 welcomed as well !!!
> >>
> >> Evans Ye  於 2019年3月4日 週一 下午3:11寫道：
> >>
> >>> Hi all,
> >>>
> >>> The overhaul as well as development is almost completed. With
> >several new
> >>> features added, I'd like to propose to release Bigtop 1.4.0 directly
> >>> instead of doing 1.3.1 release(https://s.apache.org/2a8K). This
> >helps to
> >>> promote these new features:
> >>>
> >>> 1. Full support to build and test inside docker with one stop
> >seamlessly
> >>> integration.
> >>> Now just one command to build and test components:
> >>> Example:
> >>> $ ./gradlew spark-ind repo-ind docker-provisioner -POS=ubuntu-16.04
> >>> -Pnexus -Penable_local_repo -Pstack=spark-standalone
> >-Psmoke-tests=spark
> >>>
> >>> 2. Support to build from git commit hash
> >>> Example:
> >>> $ ./gradlew kafka-pkg-ind
> >-Pgit_repo=https://github.com/apache/kafka.git
> >>> -Pgit_ref=1.1 -Pgit_sha1=4dae083af486eaedd27c69c973c74605bffd416b
> >>> -Pbase_version=1.1.1
> >>>
> >>> 3. Version bumps and tests
> >>> Hadoop 2.8.5, Kafka 1.1.1, Spark 2.4.0, Alluxio 1.8.1
> >>> New smoke tests: Flink, Giraph (Crunch)
> >>>
> >>> 4. Lts of bug fixes!
> >>>
> >>> If agreed, please +1 and I'll still serve as the RM for 1.4 release,
> >if no
> >>> preemption ;)
> >>>
> >>>
> >>> Best,
> >>> Evans
> >>>
> >>>
> >>>
> >>>
>

Re: [DISCUSS] Next minor Release of Apache Bigtop

[Jun HE]

Late reply as I cannot access gmail for past 10 days...

Hi, Evans,

+1 for the proposal, and Happy new year to everyone! :)

Regards,

Jun

Olaf Flebbe  于2019年2月7日周四 下午2:39写道：

> Hi Evans,
>
> +1 to your proposal!
>
>
> Happy new year
> olaf
>
> > Am 06.02.2019 um 14:10 schrieb Evans Ye :
> >
> > Thank you all for supporting. I'll take the RM role and start driving the
> > release process.
> >
> > Due to limited development capacity of our community, my intention for
> this
> > release as well as the entire overhauling,
> > is to focus on automation/CI/Documentation. The goal is to spend less
> > amount of time to sustain the project and yield
> > more time for having fun with new stuffs :)
> >
> > BTW, right now we're celebrating Chinese New Year, so happy new year to
> the
> > community.
> >
> > - Evans in Taiwan
> >
> >
> > Konstantin Boudnik  於 2019年2月6日 週三 上午4:51寫道：
> >
> >> Great idea, I'll try to help as much as my time permits. Thanks for
> >> volunteering!
> >> --
> >> Regards,
> >>  Cos
> >>
> >> On February 3, 2019 2:04:34 PM GMT+03:00, Evans Ye 
> >> wrote:
> >>> Hi all,
> >>>
> >>> I'd like to propose a next minor release of Apache Bigtop, which is
> >>> 1.3.1.
> >>>
> >>> Although Bigtop 1.3.0 was released back in Nov. 2018. It's a good sign
> >>> for
> >>> a project to release early, release often.
> >>>
> >>> The release will contain the following features:
> >>> 1. Overhaul the deployment and testing modules
> >>> 2. Project Frontier: Bigtop Integration Test Framework 2.0
> >>> 3. Minor upgrade to components such as: Hadoop, Spark, Kafka, etc
> >>> 4. Other bug fixes
> >>>
> >>> This minor release is more focus on the quality end instead of catching
> >>> up
> >>> to latest version of components.
> >>> The plan is to have the RC out around mid February, 2019.
> >>>
> >>> If the community is up to the release, I volunteer to be the RM.
> >>> However I'll yield if someone would like to take the role :)
> >>>
> >>>
> >>> - Evans
> >>
>

[jira] [Created] (BIGTOP-3151) Add flink smoke test

[Jun He (JIRA)]

Jun He created BIGTOP-3151:
--

 Summary: Add flink smoke test
 Key: BIGTOP-3151
 URL: https://issues.apache.org/jira/browse/BIGTOP-3151
 Project: Bigtop
  Issue Type: Bug
Reporter: Jun He
Assignee: Jun He


Add flink to smoke tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Define the future of Apache Bigtop

[Jun HE]

Thanks for the summary, Evans.

>From Linaro's perspective, it would be great to see:
1. Simple and unified deployment/management for Bigtop SW stack across
various distros and architectures.
We've seen people like Bigtop stack (open and easy to customize) but
failed to deploy and manage in an easy way. While Ambari provides such
capabilities it combines tight with HDP and x86. And this prevents the
wider aoption for Bigtop. As some academies and industry players are
leveraging certain Bigdata workloads to non-x86 platforms it would give
them great help if we can enable Bigtop SW stack support in Ambari. Also I
think community can work with ODPi (https://www.odpi.org/) on this to move
things forward more effectively.
2. Add and improve tests (smoke tests/package tests) to improve Bigtop
stack quality.
There are quite a few components enabled in the stack but lack of
tests. And some existing smoke tests need improvement to align with new
versions/settings. This could be improved to achieve good tests quality and
results to give users pretty confident to use Bigtop in their daily work.
3. Native K8s support as Jay has mentioned
4. Hadoop 3.x supports.
This could take times and effort (I think most come from ecosystem
dependecies), but it could be our primary but not prioritized work. :)

That's things came up to me so far. Pls feel free to comment.

Thanks.

Jun

Evans Ye  于2019年1月4日周五 上午12:10写道：

> To sum up the discussion of this thread, there're two ideas proposed:
> * Native K8S packages - by Jay
> * Deploy Bigtop via Ambari - by Jun
>
> I think both are very good ideas. However I'd like to gather more info from
> the requirement point of view. For example, Jun what do you see from the
> Linaro perspective for user requirements on Big Data? I think that's a
> valuable input to the community since you guys are also becoming a
> significant contributors at the community.
>
> - Evans
>
>
> Jun HE  於 2018年12月17日 週一 上午9:17寫道：
>
> > +1 for this!
> >
> > And another thing is I'm not clear about the status of ambari mpack in
> > Bigtop, so can we deploy Bigtop SW stack using ambari now? If the answer
> is
> > no, maybe this is what end user would like to see.
> >
> > Jay Vyas  于2018年12月15日周六 上午1:11写道：
> >
> > > How about a Kubernetes native distribution :).
> > >
> > > > On Dec 14, 2018, at 11:53 AM, Evans Ye  wrote:
> > > >
> > > > Hi all,
> > > >
> > > > We've just released 1.3.0 back in Nov. 2018 with RM Jun He and the
> > > > community's great help. Now it's time to look forward and setup a
> goal
> > > for
> > > > the next stage. We've a doc established back in 2017 to record the
> > ideas
> > > of
> > > > Bigtop. See:
> > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1F2Gxu8GARQDZXgqHn12LKkQ5wCV_AF4b_tVmjYB6YfA/edit#
> > > >
> > > > Align with Project Frontier's goal, I'll work on the testing part and
> > > > improve our testing, deployment, CI pipelines as a whole. So welcome
> to
> > > > work on this together!
> > > > But as time flows by, I'd more like to call out for your thoughts.
> What
> > > is
> > > > the most valuable feature to add in Bigtop, which ultimately helps
> you,
> > > > your company, and the big data users in the world. Please join the
> > > > discussion and shape the future of our project.
> > > >
> > > > Best,
> > > > Evans Ye
> > >
> >
>

[jira] [Created] (BIGTOP-3125) sbin/init is missed in puppet:opensuse-42.3

[Jun He (JIRA)]

Jun He created BIGTOP-3125:
--

 Summary: sbin/init is missed in puppet:opensuse-42.3
 Key: BIGTOP-3125
 URL: https://issues.apache.org/jira/browse/BIGTOP-3125
 Project: Bigtop
  Issue Type: Bug
Reporter: Jun He
Assignee: Jun He


The docker image, bigtop/puppet:opensuse-42.3 doesn't have /sbin/init installed.
This caused docker provisioner failed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] Move to gitbox

[Jun HE]

+1 for this.

jay vyas  于2019年1月4日周五 上午4:41写道：

> Was hoping we'd have convinced the ASF to be 100% github by now :) but i
> guess im +1 if someone has time to do it :)
>
> On Thu, Jan 3, 2019 at 3:33 PM Olaf Flebbe  wrote:
>
> > Hi,
> >
> > let see if we have consensus to move to gitbox early.
> >
> > More information on thread
> >
> >
> https://lists.apache.org/thread.html/b3868092e86d3ffa3abec7f97dbc8272518c63e57232e5386c103f19@%3Cdev.bigtop.apache.org%3E
> >
> > Please vote by reply to this email until  Sun., 6th of January 9pm CET
> >
> > +1: Let's move to gitbox early
> > -1: Let's wait
> > 0 : don't care
> >
> >
> > Olaf Flebbe
> >
> >
> >
>
> --
> jay vyas
>

[jira] [Created] (BIGTOP-3123) Define v1.4 release

[Jun He (JIRA)]

Jun He created BIGTOP-3123:
--

 Summary: Define v1.4 release
 Key: BIGTOP-3123
 URL: https://issues.apache.org/jira/browse/BIGTOP-3123
 Project: Bigtop
  Issue Type: Task
Affects Versions: 1.4.0
Reporter: Jun He


There is an existing doc: 
[https://docs.google.com/document/d/1F2Gxu8GARQDZXgqHn12LKkQ5wCV_AF4b_tVmjYB6YfA/edit#|https://docs.google.com/document/d/1F2Gxu8GARQDZXgqHn12LKkQ5wCV_AF4b_tVmjYB6YfA/edit]
 which describes ideas for the roadmap.

Some of them could be included in the v1.34 release and suggestions/thoughts 
are welcomed.

As usual BOMs and features are listed as placeholder.
 * BOM
{code}
PLACEHOLDER
{code}

 * Distros
{code}
PLACEHOLDER
{code}
 * Archs
{code}
x86_64, aarch64, ppc64le
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Define the future of Apache Bigtop

[Jun HE]

+1 for this!

And another thing is I'm not clear about the status of ambari mpack in
Bigtop, so can we deploy Bigtop SW stack using ambari now? If the answer is
no, maybe this is what end user would like to see.

Jay Vyas  于2018年12月15日周六 上午1:11写道：

> How about a Kubernetes native distribution :).
>
> > On Dec 14, 2018, at 11:53 AM, Evans Ye  wrote:
> >
> > Hi all,
> >
> > We've just released 1.3.0 back in Nov. 2018 with RM Jun He and the
> > community's great help. Now it's time to look forward and setup a goal
> for
> > the next stage. We've a doc established back in 2017 to record the ideas
> of
> > Bigtop. See:
> >
> >
> https://docs.google.com/document/d/1F2Gxu8GARQDZXgqHn12LKkQ5wCV_AF4b_tVmjYB6YfA/edit#
> >
> > Align with Project Frontier's goal, I'll work on the testing part and
> > improve our testing, deployment, CI pipelines as a whole. So welcome to
> > work on this together!
> > But as time flows by, I'd more like to call out for your thoughts. What
> is
> > the most valuable feature to add in Bigtop, which ultimately helps you,
> > your company, and the big data users in the world. Please join the
> > discussion and shape the future of our project.
> >
> > Best,
> > Evans Ye
>

Re: Request privilege to edit confluence page

[Jun HE]

Thanks, Olaf! :)

Olaf Flebbe  于2018年11月30日周五 上午1:58写道：

> Hi Jun,
>
> You can now edit the space.
>
> Thanks for your efforts
> Olaf
>
> PS:  I enabled our chair Youngwoo Kim to be administrator of that
> confluence space as well, oops.
>
>
> > Am 29.11.2018 um 08:39 schrieb Jun HE :
> >
> > Hi, folks,
> >
> > I'm trying to update HOWTo-release page (
> > https://cwiki.apache.org/confluence/display/BIGTOP/How+to+release), and
> > then found out I didn't have edit permissions.
> > Could anyone help to grant me the privileges?
> > The username is junhe. Thanks a lot!
> >
> > Regards,
> >
> > Jun
>
>

Request privilege to edit confluence page

[Jun HE]

Hi, folks,

I'm trying to update HOWTo-release page (
https://cwiki.apache.org/confluence/display/BIGTOP/How+to+release), and
then found out I didn't have edit permissions.
Could anyone help to grant me the privileges?
The username is junhe. Thanks a lot!

Regards,

Jun

[jira] [Created] (BIGTOP-3106) Update links in download page

[Jun He (JIRA)]

Jun He created BIGTOP-3106:
--

 Summary: Update links in download page
 Key: BIGTOP-3106
 URL: https://issues.apache.org/jira/browse/BIGTOP-3106
 Project: Bigtop
  Issue Type: Task
Reporter: Jun He


Links in download page should:
 * be links to release artifacts, not links to directories.
 * links to the asc and sha files should refer to archives only for non-current 
releases
 * current release links should be to [apache.org/dist|http://apache.org/dist]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: MODERATE for annou...@apache.org

[Jun HE]

Good question, and I don't know the actual info. :(
Anyway, I've sent mail to Craig to info the download page is updated.

Evans Ye  于2018年11月27日周二 下午3:10写道：

> I think the risk of direct code change for the download page is OK when
> under the emergency circumstance.
> But I'm not quite sure what's the timeout, specifically?
>
> Jun HE  於 2018年11月27日 週二 上午10:31寫道：
>
> > Hi, folks,
> >
> > I'm going to directly update the download page based on Craig's feedback
> to
> > avoid time out. And then I will create JIRA and post patch for review.
> > Sorry for this non-compliance action.
> >
> > Regards,
> >
> > Jun
> >
> > Private LIst Moderation 
> > 于2018年11月27日周二 上午7:47写道：
> >
> > > Hi Bigtoppers,
> > >
> > > This announcement is ok but the download page needs some work.
> > >
> > > The links on the page need to be links to release artifacts, not links
> to
> > > directories. The way it is set up now, the user has to go to the mirror
> > and
> > > look around for the artifact to download, and there is no obvious place
> > to
> > > find the corresponding asc and sha512.
> > >
> > > The links to the asc and sha files should refer to archives only for
> > > non-current releases. The current release links should be to
> > > apache.org/dist, not archive.apache.org/dist.
> > >
> > > If you can fix the download page timely, please let us know so we can
> > > moderate this announcement before it times out.
> > >
> > > Regards,
> > >
> > > Craig
> > >
> > > > Begin forwarded message:
> > > >
> > > > From:
> announce-reject-1543201406.6614.jnjjdcppdglpglcai...@apache.org
> > > > Subject: MODERATE for annou...@apache.org
> > > > Date: November 25, 2018 at 7:03:26 PM PST
> > > > To: Recipient list not shown: ;
> > > > Cc: announce-allow-tc.1543201406.oophecaiafipiinhpbcp-junhe=
> > > apache@apache.org
> > > > Reply-To:
> > > announce-accept-1543201406.6614.jnjjdcppdglpglcai...@apache.org
> > > >
> > > >
> > > > To approve:
> > > >   announce-accept-1543201406.6614.jnjjdcppdglpglcai...@apache.org
> > >  announce-accept-1543201406.6614.jnjjdcppdglpglcai...@apache.org>
> > > > To reject:
> > > >   announce-reject-1543201406.6614.jnjjdcppdglpglcai...@apache.org
> > >  announce-reject-1543201406.6614.jnjjdcppdglpglcai...@apache.org>
> > > > To give a reason to reject:
> > > > %%% Start comment
> > > > %%% End comment
> > > >
> > > >
> > > > From: Jun HE mailto:ju...@apache.org>>
> > > > Subject: [ANNOUNCE] Apache Bigtop 1.3.0 released
> > > > Date: November 25, 2018 at 7:03:13 PM PST
> > > > To: annou...@apache.org <mailto:annou...@apache.org>
> > > >
> > > >
> > > > The release is available here:
> > > > https://bigtop.apache.org/download.html#releases <
> > > https://bigtop.apache.org/download.html#releases>
> > > >
> > > > A few highlights of this release include:
> > > > AArch64 architecture is added to the support matrix
> > > > updated distributions (centos-7, fedora-26, debian-9, ubuntu-16.04,
> > > opensuse-42.3)
> > > > new docker images are available for distributions on different
> > > architectures
> > > > upgraded toolchain to include new tools and latest versions updates
> > > > many upgrades to the latest versions of the ecosystem projects
> (Hadoop,
> > > HBase, Hive, Spark, Solr, Flume, Kafka, GPDB and many others)
> > > > improved multi-arch support in several components (hadoop, hbase,
> qfs,
> > > ignite-hadoop and etc)
> > > > improvements in the deployment for unattended cluster
> > > > improvements in the smoke-tests coverage
> > > > With Bigtop 1.3.0 the community continues to deliver the most
> advanced
> > > big data stack to date. More details about 1.3.0 release are here:
> > > > https://bigtop.apache.org/release-notes.html <
> > > https://bigtop.apache.org/release-notes.html>
> > > >
> > > > Deploying Bigtop is easy: grab the repo/list file for your favorite
> > > Linux distribution:
> > > >   https://www.apache.org/dyn/closer.lua/bigtop/bigtop-1.3.0/repos/ <
> > > https://www.apache.org/dyn/closer.lua/bigtop/bigtop-1.3.0/repos/>
> > > > and you'll be running your very own bigdata cluster in no time!
> > > >
> > > > We welcome your help and feedback. For more information on how to
> > report
> > > problems, and to get involved, visit the project website at:
> > > > https://bigtop.apache.org <https://bigtop.apache.org/>
> > > >
> > > > I want to emphasize that this is a collaborative work done by project
> > > contributors and other communities, who continue to devote time to make
> > > Bigtop a better software. Thank you all for making this release
> possible!
> > > >
> > > > Thanks,
> > > >
> > > > Jun He (Bigtop 1.3.0 Release Manager)
> > >
> > > Craig L Russell
> > > Secretary, Apache Software Foundation
> > > c...@apache.org <mailto:c...@apache.org> http://db.apache.org/jdo <
> > > http://db.apache.org/jdo>
> > >
> >
>

Re: MODERATE for annou...@apache.org

[Jun HE]

Hi, folks,

I'm going to directly update the download page based on Craig's feedback to
avoid time out. And then I will create JIRA and post patch for review.
Sorry for this non-compliance action.

Regards,

Jun

Private LIst Moderation 
于2018年11月27日周二 上午7:47写道：

> Hi Bigtoppers,
>
> This announcement is ok but the download page needs some work.
>
> The links on the page need to be links to release artifacts, not links to
> directories. The way it is set up now, the user has to go to the mirror and
> look around for the artifact to download, and there is no obvious place to
> find the corresponding asc and sha512.
>
> The links to the asc and sha files should refer to archives only for
> non-current releases. The current release links should be to
> apache.org/dist, not archive.apache.org/dist.
>
> If you can fix the download page timely, please let us know so we can
> moderate this announcement before it times out.
>
> Regards,
>
> Craig
>
> > Begin forwarded message:
> >
> > From: announce-reject-1543201406.6614.jnjjdcppdglpglcai...@apache.org
> > Subject: MODERATE for annou...@apache.org
> > Date: November 25, 2018 at 7:03:26 PM PST
> > To: Recipient list not shown: ;
> > Cc: announce-allow-tc.1543201406.oophecaiafipiinhpbcp-junhe=
> apache@apache.org
> > Reply-To:
> announce-accept-1543201406.6614.jnjjdcppdglpglcai...@apache.org
> >
> >
> > To approve:
> >   announce-accept-1543201406.6614.jnjjdcppdglpglcai...@apache.org
> <mailto:announce-accept-1543201406.6614.jnjjdcppdglpglcai...@apache.org>
> > To reject:
> >   announce-reject-1543201406.6614.jnjjdcppdglpglcai...@apache.org
> <mailto:announce-reject-1543201406.6614.jnjjdcppdglpglcai...@apache.org>
> > To give a reason to reject:
> > %%% Start comment
> > %%% End comment
> >
> >
> > From: Jun HE mailto:ju...@apache.org>>
> > Subject: [ANNOUNCE] Apache Bigtop 1.3.0 released
> > Date: November 25, 2018 at 7:03:13 PM PST
> > To: annou...@apache.org <mailto:annou...@apache.org>
> >
> >
> > The release is available here:
> > https://bigtop.apache.org/download.html#releases <
> https://bigtop.apache.org/download.html#releases>
> >
> > A few highlights of this release include:
> > AArch64 architecture is added to the support matrix
> > updated distributions (centos-7, fedora-26, debian-9, ubuntu-16.04,
> opensuse-42.3)
> > new docker images are available for distributions on different
> architectures
> > upgraded toolchain to include new tools and latest versions updates
> > many upgrades to the latest versions of the ecosystem projects (Hadoop,
> HBase, Hive, Spark, Solr, Flume, Kafka, GPDB and many others)
> > improved multi-arch support in several components (hadoop, hbase, qfs,
> ignite-hadoop and etc)
> > improvements in the deployment for unattended cluster
> > improvements in the smoke-tests coverage
> > With Bigtop 1.3.0 the community continues to deliver the most advanced
> big data stack to date. More details about 1.3.0 release are here:
> > https://bigtop.apache.org/release-notes.html <
> https://bigtop.apache.org/release-notes.html>
> >
> > Deploying Bigtop is easy: grab the repo/list file for your favorite
> Linux distribution:
> >   https://www.apache.org/dyn/closer.lua/bigtop/bigtop-1.3.0/repos/ <
> https://www.apache.org/dyn/closer.lua/bigtop/bigtop-1.3.0/repos/>
> > and you'll be running your very own bigdata cluster in no time!
> >
> > We welcome your help and feedback. For more information on how to report
> problems, and to get involved, visit the project website at:
> > https://bigtop.apache.org <https://bigtop.apache.org/>
> >
> > I want to emphasize that this is a collaborative work done by project
> contributors and other communities, who continue to devote time to make
> Bigtop a better software. Thank you all for making this release possible!
> >
> > Thanks,
> >
> > Jun He (Bigtop 1.3.0 Release Manager)
>
> Craig L Russell
> Secretary, Apache Software Foundation
> c...@apache.org <mailto:c...@apache.org> http://db.apache.org/jdo <
> http://db.apache.org/jdo>
>

[jira] [Created] (BIGTOP-3104) Bring back changes in 1.3 to master

[Jun He (JIRA)]

Jun He created BIGTOP-3104:
--

 Summary: Bring back changes in 1.3 to master
 Key: BIGTOP-3104
 URL: https://issues.apache.org/jira/browse/BIGTOP-3104
 Project: Bigtop
  Issue Type: Task
Reporter: Jun He
Assignee: Jun He


Bring back changes happened on branch-1.3 into master.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [ANNOUNCE] Apache Bigtop 1.3.0 released

[Jun HE]

Thanks, Evans.

My username is junhe.

Evans Ye  于2018年11月25日周日 上午2:26写道：

> Great work, Jun!
>
> Let's also make some noise for the release on Apache Bigtop blog[1].
> Please let me know your username and I'll then grant you the permission.
>
> [1] https://blogs.apache.org/bigtop/
>
>
> Jun HE  於 2018年11月21日 週三 下午12:58寫道：
>
> > On behalf of the Apache Bigtop team, I'd love to announce the general
> > availability of the Bigtop 1.3.0 release.
> >
> > The release is available here:
> > https://www.apache.org/dyn/closer.lua/bigtop/bigtop-1.3.0/
> >
> > A few highlights of this release include:
> >
> >- AArch64 architecture is added to the support matrix
> >- updated distributions (centos-7, fedora-26, debian-9, ubuntu-16.04,
> >opensuse-42.3)
> >- new docker images are available for distributions on different
> >architectures
> >- upgraded toolchain to include new tools and latest versions updates
> >- many upgrades to the latest versions of the ecosystem projects
> >(Hadoop, HBase, Hive, Spark, Solr, Flume, Kafka, GPDB and many others)
> >- improved multi-arch support in several components (hadoop, hbase,
> qfs,
> >ignite-hadoop and etc)
> >- improvements in the deployment for unattended cluster
> >- improvements in the smoke-tests coverage
> >
> > With Bigtop 1.3.0 the community continues to deliver the most advanced
> big
> > data stack to date. More details about 1.3.0 release are here:
> > http://bigtop.apache.org/release-notes.html
> >
> > Deploying Bigtop is easy: grab the repo/list file for your favorite Linux
> > distribution:
> >   https://www.apache.org/dyn/closer.lua/bigtop/bigtop-1.3.0/repos/
> > and you'll be running your very own bigdata cluster in no time!
> >
> > We welcome your help and feedback. For more information on how to report
> > problems, and to get involved, visit the project website at:
> > http://bigtop.apache.org
> >
> > I want to emphasize that this is a collaborative work done by project
> > contributors and other communities, who continue to devote time to make
> > Bigtop a better software. Thank you all for making this release possible!
> >
> > Thanks,
> >
> > Jun He (Bigtop 1.3.0 Release Manager)
> >
>

[jira] [Created] (BIGTOP-3103) Update download page to comply with Apache announcement requirements

[Jun He (JIRA)]

Jun He created BIGTOP-3103:
--

 Summary: Update download page to comply with Apache announcement 
requirements
 Key: BIGTOP-3103
 URL: https://issues.apache.org/jira/browse/BIGTOP-3103
 Project: Bigtop
  Issue Type: Improvement
Affects Versions: 1.2.1
Reporter: Jun He
Assignee: Jun He
 Fix For: 1.4.0


To comply with announcement requirement, the download page should contain links 
to all current and archived releases along with links to KEYS, checksums, and 
signatures for all releases.
Current download page only provides link to latest one.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Returned post for annou...@apache.org

[Jun HE]

Hi, Evans,

I'm just start working on it now. Yesterday I have connection issue.
Patch should be created shortly.

Regards,

Jun

Evans Ye  于2018年11月25日周日 上午2:27写道：

> Jun, are you already working on this? I might be able to help on this one.
> Let me know whether you need me.
>
> Konstantin Boudnik  於 2018年11月23日 週五 上午2:31寫道：
>
> > Agree.
> >
> > --
> >   With regards,
> > Konstantin (Cos) Boudnik
> > 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
> >
> > Disclaimer: Opinions expressed in this email are those of the author,
> > and do not necessarily represent the views of any company the author
> > might be affiliated with at the moment of writing.
> >
> > On Thu, Nov 22, 2018 at 9:20 PM Evans Ye  wrote:
> > >
> > > Let me add this to the dev list cause I see this can be public :)
> > >
> > > Yes I think updating download page as what you referred is the best
> > choice.
> > > More specifically, having something similar to Hadoop[1] and slightly
> > > simplified is enough.
> > > I don't think we need to list all the releases from the very beginning
> of
> > > Bigtop.
> > > We can list available releases started from 1.1.0.
> > >
> > > [1] https://hadoop.apache.org/releases.htm
> > >
> > > Jun HE  於 2018年11月22日 週四 上午11:03寫道：
> > >
> > > > Hi, folks,
> > > >
> > > > Seems Apache release process has been updated, resulted in download
> > page (
> > > > http://bigtop.apache.org/download.html#releases) needs to be updated
> > to
> > > > comply with these requirements.
> > > > For 1.3.0 I'd like to modify
> > > >
> >
> https://github.com/apache/bigtop/blob/branch-1.3/src/site/xdoc/download.xml
> > > > directly with all required fields. Meanwhile we can discuss how to
> > reflect
> > > > these mandatory requirements to master branch.
> > > > What's your thought?
> > > >
> > > > Regards,
> > > >
> > > > Jun
> > > >
> > > > -- Forwarded message -
> > > > From: 
> > > > Date: 2018年11月21日周三 下午7:06
> > > > Subject: Returned post for annou...@apache.org
> > > > To: 
> > > >
> > > >
> > > >
> > > > Hi! This is the ezmlm program. I'm managing the
> > > > annou...@apache.org mailing list.
> > > >
> > > > I'm sorry, your message (enclosed) was not accepted by the moderator.
> > > > If the moderator has made any comments, they are shown below.
> > > >
> > > > >>>>>  >>>>>
> > > > This announcement was rejected because it does not conform to
> > standards.
> > > >
> > > > The announcement should contain a link to the bigtop download page,
> > not to
> > > > the dyn/closer direct download.
> > > > The download page should contain links to all current and archived
> > > > releases along with links to KEYS, checksums, and signatures for all
> > > > releases.
> > > >
> > > > Once the download page has been fixed and the announcement changed to
> > > > refer to the download page, please resubmit the announcement.
> > > > <<<<<  <<<<<
> > > >
> > > >
> > > >
> > > >
> > > > -- Forwarded message --
> > > > From: Jun HE 
> > > > To: annou...@apache.org, u...@bigtop.apache.org,
> dev@bigtop.apache.org
> > > > Cc:
> > > > Bcc:
> > > > Date: Wed, 21 Nov 2018 12:58:17 +0800
> > > > Subject: [ANNOUNCE] Apache Bigtop 1.3.0 released
> > > > On behalf of the Apache Bigtop team, I'd love to announce the general
> > > > availability of the Bigtop 1.3.0 release.
> > > >
> > > > The release is available here:
> > > > https://www.apache.org/dyn/closer.lua/bigtop/bigtop-1.3.0/
> > > >
> > > > A few highlights of this release include:
> > > >
> > > >- AArch64 architecture is added to the support matrix
> > > >- updated distributions (centos-7, fedora-26, debian-9,
> > ubuntu-16.04,
> > > >opensuse-42.3)
> > > >- new docker images are available for distributions on different
> > > >architectures
> > > >- upgraded toolchain to include new 

Re: [ANNOUNCE] YoungWoo Kim is now the chair of Apache Bigtop

[Jun HE]

Congrats Young Woo!

Regards,

Jun

Olaf Flebbe  于2018年11月23日周五 上午4:03写道：

> Congrats YoungWoo !
>
> Olaf
>
> Von meinem iPad gesendet
>
> > Am 22.11.2018 um 19:01 schrieb Evans Ye :
> >
> > Hi all,
> >
> > With ASF board's approval on November's meeting, YoungWoo Kim is now
> officially appointed as the chair of Apache Bigtop. Please join me in
> congratulating YoungWoo. Meanwhile, thanks to our previous chair Peter's
> service which makes the community better than ever.
> >
> > Evans
>

[ANNOUNCE] Apache Bigtop 1.3.0 released

[Jun HE]

On behalf of the Apache Bigtop team, I'd love to announce the general
availability of the Bigtop 1.3.0 release.

The release is available here:
https://www.apache.org/dyn/closer.lua/bigtop/bigtop-1.3.0/

A few highlights of this release include:

   - AArch64 architecture is added to the support matrix
   - updated distributions (centos-7, fedora-26, debian-9, ubuntu-16.04,
   opensuse-42.3)
   - new docker images are available for distributions on different
   architectures
   - upgraded toolchain to include new tools and latest versions updates
   - many upgrades to the latest versions of the ecosystem projects
   (Hadoop, HBase, Hive, Spark, Solr, Flume, Kafka, GPDB and many others)
   - improved multi-arch support in several components (hadoop, hbase, qfs,
   ignite-hadoop and etc)
   - improvements in the deployment for unattended cluster
   - improvements in the smoke-tests coverage

With Bigtop 1.3.0 the community continues to deliver the most advanced big
data stack to date. More details about 1.3.0 release are here:
http://bigtop.apache.org/release-notes.html

Deploying Bigtop is easy: grab the repo/list file for your favorite Linux
distribution:
  https://www.apache.org/dyn/closer.lua/bigtop/bigtop-1.3.0/repos/
and you'll be running your very own bigdata cluster in no time!

We welcome your help and feedback. For more information on how to report
problems, and to get involved, visit the project website at:
http://bigtop.apache.org

I want to emphasize that this is a collaborative work done by project
contributors and other communities, who continue to devote time to make
Bigtop a better software. Thank you all for making this release possible!

Thanks,

Jun He (Bigtop 1.3.0 Release Manager)

Re: [VOTE] Release Bigtop version 1.3.0

[Jun HE]

Hi, folks,

Just give a quick update here.

With Evans's help, maven artifacts, source packages and repos are in
position now. Tag (rel/1.3.0) is added.
Now I'm waiting for all the mirrors to sync. Hopefully I'll deploy site and
send out release announcment tomorrow.

This would not happen without your kind help and support during the whole
process. Really appreciated!

Regards,

Jun

Jun HE  于2018年11月15日周四 下午12:48写道：

> Thanks Evans for clarify this. :)
> Here is my vote:
> +1 [binding]: Checked signature, checksum, smoke tests fo basic components
> with several distros.
>
> So, here is the vote result:
> With four binding +1s, no non-binding +1, and no -1, or +-0 votes.
> The vote for 1.3.0 release PASSES.
>
> Binding +1s:
>   Olaf Flebbe
>   Konstantin Boudnik
>   Evans Ye
>   Jun He
>
>
> Thank you all for voting, testing, and giving feedback to make this
> release happen.
> I'll start to prepare the formal release and roll it out shortly.
>
> Regards,
>
> Jun
>
> Evans Ye  于2018年11月15日周四 上午10:20写道：
>
>> Hi Jun,
>>
>> I guess you are referring your own vote for 3+1 votes. Yes, your vote is
>> also a binding one.
>> Please wrap up the vote formally as any vote we have done previously.
>> Thanks!
>>
>> Best,
>> Evans
>>
>> Jun HE  於 2018年11月14日 週三 下午7:10寫道：
>>
>> > Hi, folks,
>> >
>> > Thanks for your prompt response and info. I think I missed Olaf's vote.
>> > Sorry.
>> > Yes, we do have 3 +1 now and I'll star to proceed formal release. Hope
>> it
>> > can be finished in this week.
>> >
>> > Thanks again!
>> >
>> > Regards,
>> >
>> > Jun
>> >
>> > Konstantin Boudnik  于2018年11月14日周三 下午4:38写道：
>> >
>> > > Huge +1 here! More people are looking into the testing of the bits
>> > > we're offering to our users - better the quality will be (speak like
>> > > Yoda I do ;)
>> > > --
>> > >   With regards,
>> > > Konstantin (Cos) Boudnik
>> > > 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>> > >
>> > > Disclaimer: Opinions expressed in this email are those of the author,
>> > > and do not necessarily represent the views of any company the author
>> > > might be affiliated with at the moment of writing.
>> > >
>> > >
>> > > On Wed, Nov 14, 2018 at 9:38 AM, Evans Ye  wrote:
>> > > > Yes. I also encourage not only PMC, but also committers and users to
>> > test
>> > > > out the release candidate.
>> > > > No matter binding or non-binding, the input are valuable.
>> > > >
>> > > > BTW, I checked the rule[1]. We've reached the ASF criterial for a
>> > > release,
>> > > > which is at least three votes from PMC.
>> > > > So, if you as the RM want to hear more from the community, we can
>> set
>> > > > another deadline.
>> > > > Afterwards we can proceed to the official release and the
>> announcement.
>> > > >
>> > > > [1]
>> > http://www.apache.org/legal/release-policy.html#approving-a-release
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > Jun HE  於 2018年11月14日 週三 下午1:09寫道：
>> > > >
>> > > >> Thanks for testing and voting, Cos and Evans.
>> > > >>
>> > > >> So far this release still didn't get enough votes. Could others pls
>> > > help to
>> > > >> check and vote? Thanks a lot for your kind support!
>> > > >>
>> > > >> Regards,
>> > > >>
>> > > >> Jun
>> > > >>
>> > > >> Konstantin Boudnik 于2018年11月12日 周一02:36写道：
>> > > >>
>> > > >> > Thanks for waiting...
>> > > >> >
>> > > >> > +1 [binding]
>> > > >> >
>> > > >> > Did basic consistency checks  - everything looks ok.
>> > > >> > --
>> > > >> >   With regards,
>> > > >> > Konstantin (Cos) Boudnik
>> > > >> > 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>> > > >> >
>> > > >> > Disclaimer: Opinions expressed in this email are those of the
>> > author,
>> > > >> > and do not necessarily represent the views of any company the
>> author
>>

Re: [VOTE] Release Bigtop version 1.3.0

[Jun HE]

Thanks Evans for clarify this. :)
Here is my vote:
+1 [binding]: Checked signature, checksum, smoke tests fo basic components
with several distros.

So, here is the vote result:
With four binding +1s, no non-binding +1, and no -1, or +-0 votes.
The vote for 1.3.0 release PASSES.

Binding +1s:
  Olaf Flebbe
  Konstantin Boudnik
  Evans Ye
  Jun He


Thank you all for voting, testing, and giving feedback to make this release
happen.
I'll start to prepare the formal release and roll it out shortly.

Regards,

Jun

Evans Ye  于2018年11月15日周四 上午10:20写道：

> Hi Jun,
>
> I guess you are referring your own vote for 3+1 votes. Yes, your vote is
> also a binding one.
> Please wrap up the vote formally as any vote we have done previously.
> Thanks!
>
> Best,
> Evans
>
> Jun HE  於 2018年11月14日 週三 下午7:10寫道：
>
> > Hi, folks,
> >
> > Thanks for your prompt response and info. I think I missed Olaf's vote.
> > Sorry.
> > Yes, we do have 3 +1 now and I'll star to proceed formal release. Hope it
> > can be finished in this week.
> >
> > Thanks again!
> >
> > Regards,
> >
> > Jun
> >
> > Konstantin Boudnik  于2018年11月14日周三 下午4:38写道：
> >
> > > Huge +1 here! More people are looking into the testing of the bits
> > > we're offering to our users - better the quality will be (speak like
> > > Yoda I do ;)
> > > --
> > >   With regards,
> > > Konstantin (Cos) Boudnik
> > > 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
> > >
> > > Disclaimer: Opinions expressed in this email are those of the author,
> > > and do not necessarily represent the views of any company the author
> > > might be affiliated with at the moment of writing.
> > >
> > >
> > > On Wed, Nov 14, 2018 at 9:38 AM, Evans Ye  wrote:
> > > > Yes. I also encourage not only PMC, but also committers and users to
> > test
> > > > out the release candidate.
> > > > No matter binding or non-binding, the input are valuable.
> > > >
> > > > BTW, I checked the rule[1]. We've reached the ASF criterial for a
> > > release,
> > > > which is at least three votes from PMC.
> > > > So, if you as the RM want to hear more from the community, we can set
> > > > another deadline.
> > > > Afterwards we can proceed to the official release and the
> announcement.
> > > >
> > > > [1]
> > http://www.apache.org/legal/release-policy.html#approving-a-release
> > > >
> > > >
> > > >
> > > >
> > > > Jun HE  於 2018年11月14日 週三 下午1:09寫道：
> > > >
> > > >> Thanks for testing and voting, Cos and Evans.
> > > >>
> > > >> So far this release still didn't get enough votes. Could others pls
> > > help to
> > > >> check and vote? Thanks a lot for your kind support!
> > > >>
> > > >> Regards,
> > > >>
> > > >> Jun
> > > >>
> > > >> Konstantin Boudnik 于2018年11月12日 周一02:36写道：
> > > >>
> > > >> > Thanks for waiting...
> > > >> >
> > > >> > +1 [binding]
> > > >> >
> > > >> > Did basic consistency checks  - everything looks ok.
> > > >> > --
> > > >> >   With regards,
> > > >> > Konstantin (Cos) Boudnik
> > > >> > 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
> > > >> >
> > > >> > Disclaimer: Opinions expressed in this email are those of the
> > author,
> > > >> > and do not necessarily represent the views of any company the
> author
> > > >> > might be affiliated with at the moment of writing.
> > > >> >
> > > >> >
> > > >> > On Tue, Oct 30, 2018 at 10:31 AM, Jun HE 
> wrote:
> > > >> > > This is the vote for release 1.3.0 of Apache Bigtop.
> > > >> > >
> > > >> > > It fixes the following issues:
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12338976=12311420
> > > >> > >
> > > >> > > The vote will be going for at least 72 hours and will be closed
> on
> > > >> > Saturday,
> > > >> > > November 3, 2018 at noon PDT.  Please download, test and vote
> with
> > > >> > >
> > > >> > > [ ] +1, accept RC2 as the official 1.3.0 release of Apache
> Bigtop
> > > >> > > [ ] +0, I don't care either way,
> > > >> > > [ ] -1, do not accept RC2 as the official 1.3.0 release of
> Apache
> > > >> Bigtop,
> > > >> > > because...
> > > >> > >
> > > >> > > Source and binary files:
> > > >> > >
> > > >> https://dist.apache.org/repos/dist/dev/bigtop/bigtop-1.3.0-RC2/
> > > >> > >
> > > >> > > Maven staging repo:
> > > >> > >
> > > >> > >
> > > >>
> > https://repository.apache.org/content/repositories/orgapachebigtop-1020
> > > >> > >
> > > >> > > The git tag to be voted upon is release-1.3.0
> > > >> > >
> > > >> > > Bigtop's KEYS file containing PGP keys we use to sign the
> release:
> > > >> > > https://dist.apache.org/repos/dist/release/bigtop/KEYS
> > > >> >
> > > >>
> > >
> >
>

Re: [VOTE] Release Bigtop version 1.3.0

[Jun HE]

Hi, folks,

Thanks for your prompt response and info. I think I missed Olaf's vote.
Sorry.
Yes, we do have 3 +1 now and I'll star to proceed formal release. Hope it
can be finished in this week.

Thanks again!

Regards,

Jun

Konstantin Boudnik  于2018年11月14日周三 下午4:38写道：

> Huge +1 here! More people are looking into the testing of the bits
> we're offering to our users - better the quality will be (speak like
> Yoda I do ;)
> --
>   With regards,
> Konstantin (Cos) Boudnik
> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>
> Disclaimer: Opinions expressed in this email are those of the author,
> and do not necessarily represent the views of any company the author
> might be affiliated with at the moment of writing.
>
>
> On Wed, Nov 14, 2018 at 9:38 AM, Evans Ye  wrote:
> > Yes. I also encourage not only PMC, but also committers and users to test
> > out the release candidate.
> > No matter binding or non-binding, the input are valuable.
> >
> > BTW, I checked the rule[1]. We've reached the ASF criterial for a
> release,
> > which is at least three votes from PMC.
> > So, if you as the RM want to hear more from the community, we can set
> > another deadline.
> > Afterwards we can proceed to the official release and the announcement.
> >
> > [1] http://www.apache.org/legal/release-policy.html#approving-a-release
> >
> >
> >
> >
> > Jun HE  於 2018年11月14日 週三 下午1:09寫道：
> >
> >> Thanks for testing and voting, Cos and Evans.
> >>
> >> So far this release still didn't get enough votes. Could others pls
> help to
> >> check and vote? Thanks a lot for your kind support!
> >>
> >> Regards,
> >>
> >> Jun
> >>
> >> Konstantin Boudnik 于2018年11月12日 周一02:36写道：
> >>
> >> > Thanks for waiting...
> >> >
> >> > +1 [binding]
> >> >
> >> > Did basic consistency checks  - everything looks ok.
> >> > --
> >> >   With regards,
> >> > Konstantin (Cos) Boudnik
> >> > 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
> >> >
> >> > Disclaimer: Opinions expressed in this email are those of the author,
> >> > and do not necessarily represent the views of any company the author
> >> > might be affiliated with at the moment of writing.
> >> >
> >> >
> >> > On Tue, Oct 30, 2018 at 10:31 AM, Jun HE  wrote:
> >> > > This is the vote for release 1.3.0 of Apache Bigtop.
> >> > >
> >> > > It fixes the following issues:
> >> > >
> >> > >
> >> >
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12338976=12311420
> >> > >
> >> > > The vote will be going for at least 72 hours and will be closed on
> >> > Saturday,
> >> > > November 3, 2018 at noon PDT.  Please download, test and vote with
> >> > >
> >> > > [ ] +1, accept RC2 as the official 1.3.0 release of Apache Bigtop
> >> > > [ ] +0, I don't care either way,
> >> > > [ ] -1, do not accept RC2 as the official 1.3.0 release of Apache
> >> Bigtop,
> >> > > because...
> >> > >
> >> > > Source and binary files:
> >> > >
> >> https://dist.apache.org/repos/dist/dev/bigtop/bigtop-1.3.0-RC2/
> >> > >
> >> > > Maven staging repo:
> >> > >
> >> > >
> >> https://repository.apache.org/content/repositories/orgapachebigtop-1020
> >> > >
> >> > > The git tag to be voted upon is release-1.3.0
> >> > >
> >> > > Bigtop's KEYS file containing PGP keys we use to sign the release:
> >> > > https://dist.apache.org/repos/dist/release/bigtop/KEYS
> >> >
> >>
>

Re: [VOTE] Release Bigtop version 1.3.0

[Jun HE]

Thanks for testing and voting, Cos and Evans.

So far this release still didn't get enough votes. Could others pls help to
check and vote? Thanks a lot for your kind support!

Regards,

Jun

Konstantin Boudnik 于2018年11月12日 周一02:36写道：

> Thanks for waiting...
>
> +1 [binding]
>
> Did basic consistency checks  - everything looks ok.
> --
>   With regards,
> Konstantin (Cos) Boudnik
> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>
> Disclaimer: Opinions expressed in this email are those of the author,
> and do not necessarily represent the views of any company the author
> might be affiliated with at the moment of writing.
>
>
> On Tue, Oct 30, 2018 at 10:31 AM, Jun HE  wrote:
> > This is the vote for release 1.3.0 of Apache Bigtop.
> >
> > It fixes the following issues:
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12338976=12311420
> >
> > The vote will be going for at least 72 hours and will be closed on
> Saturday,
> > November 3, 2018 at noon PDT.  Please download, test and vote with
> >
> > [ ] +1, accept RC2 as the official 1.3.0 release of Apache Bigtop
> > [ ] +0, I don't care either way,
> > [ ] -1, do not accept RC2 as the official 1.3.0 release of Apache Bigtop,
> > because...
> >
> > Source and binary files:
> > https://dist.apache.org/repos/dist/dev/bigtop/bigtop-1.3.0-RC2/
> >
> > Maven staging repo:
> >
> > https://repository.apache.org/content/repositories/orgapachebigtop-1020
> >
> > The git tag to be voted upon is release-1.3.0
> >
> > Bigtop's KEYS file containing PGP keys we use to sign the release:
> > https://dist.apache.org/repos/dist/release/bigtop/KEYS
>

Re: [VOTE] Release Bigtop version 1.3.0

[Jun HE]

Hi, Cos,

Sure. So far we didn't receive enough votes yet. Would it work for you if
the vote is extended to Nov 10?

Regards,

Jun

Konstantin Boudnik  于2018年11月4日周日 下午2:04写道：

> Can we give it a couple more days? I am away during the weekend, and won't
> be
> able to look at the release.
>
> Thanks!
>   Cos
>
> On Tue, Oct 30, 2018 at 03:31PM, Jun HE wrote:
> > This is the vote for release 1.3.0 of Apache Bigtop.
> >
> > It fixes the following issues:
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12338976=12311420
> >
> > The vote will be going for at least 72 hours and will be closed on
> Saturday,
> > November 3, 2018 at noon PDT.  Please download, test and vote with
> >
> > [ ] +1, accept RC2 as the official 1.3.0 release of Apache Bigtop
> > [ ] +0, I don't care either way,
> > [ ] -1, do not accept RC2 as the official 1.3.0 release of Apache Bigtop,
> > because...
> >
> > Source and binary files:
> > https://dist.apache.org/repos/dist/dev/bigtop/bigtop-1.3.0-RC2/
> >
> > Maven staging repo:
> >
> > https://repository.apache.org/content/repositories/orgapachebigtop-1020
> >
> > The git tag to be voted upon is release-1.3.0
> >
> > Bigtop's KEYS file containing PGP keys we use to sign the release:
> > https://dist.apache.org/repos/dist/release/bigtop/KEYS
>

Re: Testing 1.3.0RC2

[Jun HE]

Hi, Olaf,

Thanks for point this out. I can reproduce this on x86. Might be the
resitriction not set properly on S3.

I've re-publiced all directories under 1.3.0 again. The smoke test shows
fine.

Could you pls have a try again? Thanks.

Regards,

Jun


Olaf Flebbe  于2018年11月1日周四 下午5:53写道：

> *Hi Jun,*
>
> *Can u open access to the S3 bucket ? I get a 403 Forbidden for the first
> file to install.*
>
> *http://repos.bigtop.apache.org/releases/1.3.0/debian/9/amd64/pool/contrib/h/hadoop/libhdfs0-dev_2.8.4-1_amd64.deb
> *
>
> *Regards,*
> *Olaf*
>

[VOTE] Release Bigtop version 1.3.0

[Jun HE]

This is the vote for release 1.3.0 of Apache Bigtop.

It fixes the following issues:

https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12338976=12311420

The vote will be going for at least 72 hours and will be closed on Saturday,
November 3, 2018 at noon PDT.  Please download, test and vote with

[ ] +1, accept RC2 as the official 1.3.0 release of Apache Bigtop
[ ] +0, I don't care either way,
[ ] -1, do not accept RC2 as the official 1.3.0 release of Apache Bigtop,
because...

Source and binary files:
https://dist.apache.org/repos/dist/dev/bigtop/bigtop-1.3.0-RC2/

Maven staging repo:

https://repository.apache.org/content/repositories/orgapachebigtop-1020

The git tag to be voted upon is release-1.3.0

Bigtop's KEYS file containing PGP keys we use to sign the release:
https://dist.apache.org/repos/dist/release/bigtop/KEYS

Re: [VOTE] Release Bigtop version 1.3.0

[Jun HE]

Hi, Olaf,

These downloading failures were observed on other archs as well. Just
randomly happend.
And I saw several times the failure happened in zeppelin's zeppelin-web.

Regards,

Jun

Olaf Flebbe  于2018年10月26日周五 下午12:52写道：

> Hi Jun,
>
> these random artifact failures made me build the nexus artifact cache,
> which I had to disable because it finally tripped over other problems, I
> was aware but didn't show up before. I don't think just enabling it again
> will help. I will work on a new solution. Will create a jira to track the
> efforts.
>
> for the time please do cheat like copying together independent runs, I
> would recommend.
>
> one question: does this random download failures still mostly show on
> amd64 or do you observe it on ppc64le and aarch64 as well?
>
> Olaf
>
> Von meinem iPad gesendet
>
> > Am 26.10.2018 um 03:27 schrieb Jun HE :
> >
> > Sorry for not updating for a while.
> >
> > Actually I was struggling in building the 1.3 RC2 for last two weeks. I
> > cannot get the Bigtop-1.3.0 job successfully finished in one shot.
> There've
> > always been some random artifacts downloading issue, happened for
> different
> > arch and distros, caused some subjob failed. See fedora-26 and
> > opensuse-42.3 on amd64 (
> https://ci.bigtop.apache.org/job/Bigtop-1.3.0/25/)
> >
> > I tried these two subjob in job-26 and they succeeded. And I'm intended
> to
> > use their outputs (job-25/26) to prepare RC2. Do you see any risk or
> > problem it could be?
> >
> > Thanks,
> >
> > Jun
> >
> > Evans Ye  于2018年10月26日周五 上午1:21写道：
> >
> >> Hi Jun,
> >>
> >> Could you share the status of 1.3.0 release. Anything you'd like to call
> >> for help?
> >>
> >> Thanks,
> >> Evans
> >>
> >> Jun HE  於 2018年10月10日 週三 下午6:13寫道：
> >>
> >>> Hi, folks,
> >>>
> >>> I'm planning to start RC2 preparation on 10-12. Please kindly let me
> know
> >>> if you:
> >>> * need more time to test
> >>> * found problems in RC1, other than provisioners failure
> >>> (centos-7/debian-9)
> >>> * have anything which should be included in RC2
> >>>
> >>> The plan is to start build on 10-12 20:00, Beijing time if no further
> >>> comments are raised.
> >>>
> >>> Thanks & regards,
> >>>
> >>> Jun
> >>>
> >>> Jun HE  于2018年10月8日周一 上午8:41写道：
> >>>
> >>>>
> >>>> Sure, I will go to work on RC2.
> >>>>
> >>>> Before doing that, I'd like to know if there are other issues you
> found
> >>> in
> >>>> branch-1.3? So I can fix them before baking RC2. :)
> >>>>
> >>>> Konstantin Boudnik  于2018年10月8日周一 上午2:38写道：
> >>>>
> >>>>> So, looks like the vote needs to be stopped until RC2 is ready,
> right?
> >>>>> Otherwise, things will become confusing...
> >>>>>
> >>>>> Cos
> >>>>>
> >>>>>> On Wed, Oct 03, 2018 at 07:07PM, Jun HE wrote:
> >>>>>> I've tested CentOS-7 with 4.12.0 puppetlabs-stdlib, will submit
> >> patch
> >>>>> soon.
> >>>>>> For Debian, I'll update the config file as you suggested, and
> >>>>> centos/fedora
> >>>>>> config as well.
> >>>>>> They should be ready in today for review.
> >>>>>>
> >>>>>> Evans Ye  于2018年10月3日周三 下午1:48写道：
> >>>>>>
> >>>>>>> For provisioner part, I tested CentOS, Ubuntu, and Debian.
> >>>>>>> Currently only Ubuntu works.
> >>>>>>> * CentOS: Need to pin down the puppetlabs-stdlib to 4.12.0 as Jun
> >> He
> >>>>>>> suggested
> >>>>>>> * Debian: Just need to update config file from debian 8 to debian
> >> 9.
> >>>>>>>
> >>>>>>> Jun are you already working on this or you need my help fixing
> >> them?
> >>>>>>>
> >>>>>>>
> >>>>>>> Jun HE  於 2018年9月30日 週日 下午2:30寫道：
> >>>>>>>
> >>>>>>>> Hi Olaf,
> >>>>>>>>
> >>>>>>>> For debian provision test, use bigtop/puppet:1.3.0-debia

Re: [VOTE] Release Bigtop version 1.3.0

[Jun HE]

Sorry for not updating for a while.

Actually I was struggling in building the 1.3 RC2 for last two weeks. I
cannot get the Bigtop-1.3.0 job successfully finished in one shot. There've
always been some random artifacts downloading issue, happened for different
arch and distros, caused some subjob failed. See fedora-26 and
opensuse-42.3 on amd64 (https://ci.bigtop.apache.org/job/Bigtop-1.3.0/25/)

I tried these two subjob in job-26 and they succeeded. And I'm intended to
use their outputs (job-25/26) to prepare RC2. Do you see any risk or
problem it could be?

Thanks,

Jun

Evans Ye  于2018年10月26日周五 上午1:21写道：

> Hi Jun,
>
> Could you share the status of 1.3.0 release. Anything you'd like to call
> for help?
>
> Thanks,
> Evans
>
> Jun HE  於 2018年10月10日 週三 下午6:13寫道：
>
> > Hi, folks,
> >
> > I'm planning to start RC2 preparation on 10-12. Please kindly let me know
> > if you:
> > * need more time to test
> > * found problems in RC1, other than provisioners failure
> > (centos-7/debian-9)
> > * have anything which should be included in RC2
> >
> > The plan is to start build on 10-12 20:00, Beijing time if no further
> > comments are raised.
> >
> > Thanks & regards,
> >
> > Jun
> >
> > Jun HE  于2018年10月8日周一 上午8:41写道：
> >
> > >
> > > Sure, I will go to work on RC2.
> > >
> > > Before doing that, I'd like to know if there are other issues you found
> > in
> > > branch-1.3? So I can fix them before baking RC2. :)
> > >
> > > Konstantin Boudnik  于2018年10月8日周一 上午2:38写道：
> > >
> > >> So, looks like the vote needs to be stopped until RC2 is ready, right?
> > >> Otherwise, things will become confusing...
> > >>
> > >> Cos
> > >>
> > >> On Wed, Oct 03, 2018 at 07:07PM, Jun HE wrote:
> > >> > I've tested CentOS-7 with 4.12.0 puppetlabs-stdlib, will submit
> patch
> > >> soon.
> > >> > For Debian, I'll update the config file as you suggested, and
> > >> centos/fedora
> > >> > config as well.
> > >> > They should be ready in today for review.
> > >> >
> > >> > Evans Ye  于2018年10月3日周三 下午1:48写道：
> > >> >
> > >> > > For provisioner part, I tested CentOS, Ubuntu, and Debian.
> > >> > > Currently only Ubuntu works.
> > >> > > * CentOS: Need to pin down the puppetlabs-stdlib to 4.12.0 as Jun
> He
> > >> > > suggested
> > >> > > * Debian: Just need to update config file from debian 8 to debian
> 9.
> > >> > >
> > >> > > Jun are you already working on this or you need my help fixing
> them?
> > >> > >
> > >> > >
> > >> > > Jun HE  於 2018年9月30日 週日 下午2:30寫道：
> > >> > >
> > >> > > > Hi Olaf,
> > >> > > >
> > >> > > > For debian provision test, use bigtop/puppet:1.3.0-debian-9
> should
> > >> work.
> > >> > > > As for issue in centos-7, it should be the same of previous
> > >> puppet-stdlib
> > >> > > > dependency (See:
> > https://tickets.puppetlabs.com/browse/MODULES-3962
> > >> ).
> > >> > > Pin
> > >> > > > puppet-stdlib to 4.12.0 should fix this problem.
> > >> > > > I'll do test and submit fix if local verification is OK.
> > >> > > >
> > >> > > >
> > >> > > > Olaf Flebbe  于2018年9月30日周日 上午12:03写道：
> > >> > > >
> > >> > > > > Hi,
> > >> > > > >
> > >> > > > > I tried the docker provisioner and it failed for debian
> because
> > >> > > > /sbin/init
> > >> > > > > not present in bigtop/puppet:trunk-debian-9
> > >> > > > >
> > >> > > > > --- config --
> > >> > > > >
> > >> > > > > docker:
> > >> > > > > memory_limit: "4g"
> > >> > > > > image: "bigtop/puppet:trunk-debian-9"
> > >> > > > >
> > >> > > > > repo: "
> > >> http://repos.bigtop.apache.org/releases/1.3.0/debian/9/x86_64;
> > >> > > > > distro: debian
> > >> > > > > components: [hdfs, yarn, mapreduce]
> > >> > > > > enable_local_re

Re: Jenkins amd64 slaves are not working properly

[Jun HE]

Thank you, Evans.

Evans Ye  于2018年10月14日周日 下午11:50写道：

> Yeah It was me. I just hit the restart button w/o doing further action.
> Thanks Olaf!
>
> Jun HE 於 2018年10月14日 週日，下午3:19寫道：
>
> > Seeing the build is in profgress now.
> > Thanks a lot, Olaf!
> >
> >
> > Olaf Flebbe  于2018年10月14日周日 下午10:08写道：
> >
> > > Hi,
> > >
> > > Maintenance completed -- for now.
> > >
> > > Started bigtop-1.3 job
> > >
> > > Best,
> > > Olaf
> > >
> > > > Am 12.10.2018 um 16:35 schrieb Jun HE :
> > > >
> > > > When I was trying to start v1.3.0-RC2 building, amd4 slaves
> > > > (slave-06/slave-07) report error to start docker container. Same
> error
> > > > output as BIGTOP-3050.
> > > >
> > > > As I don't have access to these two nodes, could anybody help to
> > restart
> > > > them?
> > > > Thanks a lot!
> > >
> > >
> >
>

Re: Jenkins amd64 slaves are not working properly

[Jun HE]

Seeing the build is in profgress now.
Thanks a lot, Olaf!


Olaf Flebbe  于2018年10月14日周日 下午10:08写道：

> Hi,
>
> Maintenance completed -- for now.
>
> Started bigtop-1.3 job
>
> Best,
> Olaf
>
> > Am 12.10.2018 um 16:35 schrieb Jun HE :
> >
> > When I was trying to start v1.3.0-RC2 building, amd4 slaves
> > (slave-06/slave-07) report error to start docker container. Same error
> > output as BIGTOP-3050.
> >
> > As I don't have access to these two nodes, could anybody help to restart
> > them?
> > Thanks a lot!
>
>

Jenkins amd64 slaves are not working properly

[Jun HE]

When I was trying to start v1.3.0-RC2 building, amd4 slaves
(slave-06/slave-07) report error to start docker container. Same error
output as BIGTOP-3050.

As I don't have access to these two nodes, could anybody help to restart
them?
Thanks a lot!

Re: [VOTE] Release Bigtop version 1.3.0

[Jun HE]

Hi, folks,

I'm planning to start RC2 preparation on 10-12. Please kindly let me know
if you:
* need more time to test
* found problems in RC1, other than provisioners failure (centos-7/debian-9)
* have anything which should be included in RC2

The plan is to start build on 10-12 20:00, Beijing time if no further
comments are raised.

Thanks & regards,

Jun

Jun HE  于2018年10月8日周一 上午8:41写道：

>
> Sure, I will go to work on RC2.
>
> Before doing that, I'd like to know if there are other issues you found in
> branch-1.3? So I can fix them before baking RC2. :)
>
> Konstantin Boudnik  于2018年10月8日周一 上午2:38写道：
>
>> So, looks like the vote needs to be stopped until RC2 is ready, right?
>> Otherwise, things will become confusing...
>>
>> Cos
>>
>> On Wed, Oct 03, 2018 at 07:07PM, Jun HE wrote:
>> > I've tested CentOS-7 with 4.12.0 puppetlabs-stdlib, will submit patch
>> soon.
>> > For Debian, I'll update the config file as you suggested, and
>> centos/fedora
>> > config as well.
>> > They should be ready in today for review.
>> >
>> > Evans Ye  于2018年10月3日周三 下午1:48写道：
>> >
>> > > For provisioner part, I tested CentOS, Ubuntu, and Debian.
>> > > Currently only Ubuntu works.
>> > > * CentOS: Need to pin down the puppetlabs-stdlib to 4.12.0 as Jun He
>> > > suggested
>> > > * Debian: Just need to update config file from debian 8 to debian 9.
>> > >
>> > > Jun are you already working on this or you need my help fixing them?
>> > >
>> > >
>> > > Jun HE  於 2018年9月30日 週日 下午2:30寫道：
>> > >
>> > > > Hi Olaf,
>> > > >
>> > > > For debian provision test, use bigtop/puppet:1.3.0-debian-9 should
>> work.
>> > > > As for issue in centos-7, it should be the same of previous
>> puppet-stdlib
>> > > > dependency (See: https://tickets.puppetlabs.com/browse/MODULES-3962
>> ).
>> > > Pin
>> > > > puppet-stdlib to 4.12.0 should fix this problem.
>> > > > I'll do test and submit fix if local verification is OK.
>> > > >
>> > > >
>> > > > Olaf Flebbe  于2018年9月30日周日 上午12:03写道：
>> > > >
>> > > > > Hi,
>> > > > >
>> > > > > I tried the docker provisioner and it failed for debian because
>> > > > /sbin/init
>> > > > > not present in bigtop/puppet:trunk-debian-9
>> > > > >
>> > > > > --- config --
>> > > > >
>> > > > > docker:
>> > > > > memory_limit: "4g"
>> > > > > image: "bigtop/puppet:trunk-debian-9"
>> > > > >
>> > > > > repo: "
>> http://repos.bigtop.apache.org/releases/1.3.0/debian/9/x86_64;
>> > > > > distro: debian
>> > > > > components: [hdfs, yarn, mapreduce]
>> > > > > enable_local_repo: false
>> > > > > smoke_test_components: [hdfs, yarn, mapreduce]
>> > > > > -
>> > > > > I can live with it, since docker provisioner had many issues with
>> > > systemd
>> > > > > in the past:
>> > > > >
>> > > > > But it failed for centos-7  too.
>> > > > >
>> > > > > + docker exec
>> > > > > 8d3cf8113318d6b695a85742b985b65d6e146152a7019994dfb8fd8cf0017297
>> bash
>> > > -c
>> > > > > 'puppet apply --parser future
>> > > > >
>> > > >
>> > >
>> --modulepath=/bigtop-home/bigtop-deploy/puppet/modules:/etc/puppet/modules:/usr/share/puppet/modules
>> > > > > /bigtop-home/bigtop-deploy/puppet/manifests'
>> > > > > + future='--parser future'
>> > > > > + docker exec
>> > > > > c18b89c4cfd0847d71798b0bf5a86db01c6f3becb42b8ba293343227a71a8717
>> bash
>> > > -c
>> > > > > 'puppet apply --parser future
>> > > > >
>> > > >
>> > >
>> --modulepath=/bigtop-home/bigtop-deploy/puppet/modules:/etc/puppet/modules:/usr/share/puppet/modules
>> > > > > /bigtop-home/bigtop-deploy/puppet/manifests'
>> > > > > + future='--parser future'
>> > > > > + docker exec
>> > > > > 79b0a9ce1ae5d25f25cbf35c3e3b8e26131ca556a2b046eca56737020ffbd31e
>> bash
>> > > -c
>> > > > >

Re: [VOTE] Release Bigtop version 1.3.0

[Jun HE]

Sure, I will go to work on RC2.

Before doing that, I'd like to know if there are other issues you found in
branch-1.3? So I can fix them before baking RC2. :)

Konstantin Boudnik  于2018年10月8日周一 上午2:38写道：

> So, looks like the vote needs to be stopped until RC2 is ready, right?
> Otherwise, things will become confusing...
>
> Cos
>
> On Wed, Oct 03, 2018 at 07:07PM, Jun HE wrote:
> > I've tested CentOS-7 with 4.12.0 puppetlabs-stdlib, will submit patch
> soon.
> > For Debian, I'll update the config file as you suggested, and
> centos/fedora
> > config as well.
> > They should be ready in today for review.
> >
> > Evans Ye  于2018年10月3日周三 下午1:48写道：
> >
> > > For provisioner part, I tested CentOS, Ubuntu, and Debian.
> > > Currently only Ubuntu works.
> > > * CentOS: Need to pin down the puppetlabs-stdlib to 4.12.0 as Jun He
> > > suggested
> > > * Debian: Just need to update config file from debian 8 to debian 9.
> > >
> > > Jun are you already working on this or you need my help fixing them?
> > >
> > >
> > > Jun HE  於 2018年9月30日 週日 下午2:30寫道：
> > >
> > > > Hi Olaf,
> > > >
> > > > For debian provision test, use bigtop/puppet:1.3.0-debian-9 should
> work.
> > > > As for issue in centos-7, it should be the same of previous
> puppet-stdlib
> > > > dependency (See: https://tickets.puppetlabs.com/browse/MODULES-3962
> ).
> > > Pin
> > > > puppet-stdlib to 4.12.0 should fix this problem.
> > > > I'll do test and submit fix if local verification is OK.
> > > >
> > > >
> > > > Olaf Flebbe  于2018年9月30日周日 上午12:03写道：
> > > >
> > > > > Hi,
> > > > >
> > > > > I tried the docker provisioner and it failed for debian because
> > > > /sbin/init
> > > > > not present in bigtop/puppet:trunk-debian-9
> > > > >
> > > > > --- config --
> > > > >
> > > > > docker:
> > > > > memory_limit: "4g"
> > > > > image: "bigtop/puppet:trunk-debian-9"
> > > > >
> > > > > repo: "
> http://repos.bigtop.apache.org/releases/1.3.0/debian/9/x86_64;
> > > > > distro: debian
> > > > > components: [hdfs, yarn, mapreduce]
> > > > > enable_local_repo: false
> > > > > smoke_test_components: [hdfs, yarn, mapreduce]
> > > > > -
> > > > > I can live with it, since docker provisioner had many issues with
> > > systemd
> > > > > in the past:
> > > > >
> > > > > But it failed for centos-7  too.
> > > > >
> > > > > + docker exec
> > > > > 8d3cf8113318d6b695a85742b985b65d6e146152a7019994dfb8fd8cf0017297
> bash
> > > -c
> > > > > 'puppet apply --parser future
> > > > >
> > > >
> > >
> --modulepath=/bigtop-home/bigtop-deploy/puppet/modules:/etc/puppet/modules:/usr/share/puppet/modules
> > > > > /bigtop-home/bigtop-deploy/puppet/manifests'
> > > > > + future='--parser future'
> > > > > + docker exec
> > > > > c18b89c4cfd0847d71798b0bf5a86db01c6f3becb42b8ba293343227a71a8717
> bash
> > > -c
> > > > > 'puppet apply --parser future
> > > > >
> > > >
> > >
> --modulepath=/bigtop-home/bigtop-deploy/puppet/modules:/etc/puppet/modules:/usr/share/puppet/modules
> > > > > /bigtop-home/bigtop-deploy/puppet/manifests'
> > > > > + future='--parser future'
> > > > > + docker exec
> > > > > 79b0a9ce1ae5d25f25cbf35c3e3b8e26131ca556a2b046eca56737020ffbd31e
> bash
> > > -c
> > > > > 'puppet apply --parser future
> > > > >
> > > >
> > >
> --modulepath=/bigtop-home/bigtop-deploy/puppet/modules:/etc/puppet/modules:/usr/share/puppet/modules
> > > > > /bigtop-home/bigtop-deploy/puppet/manifests'
> > > > > Warning: Config file /etc/puppet/hiera.yaml not found, using Hiera
> > > > defaults
> > > > > Error: Could not find data item bigtop::hadoop_head_node in any
> Hiera
> > > > data
> > > > > file and no default supplied on node
> 8d3cf8113318.bigtop.apache.org
> > > > > Error: Could not find data item bigtop::hadoop_head_node in any
> Hiera
> > > > data
> > > > > file and no default supplied on node
> 8

[jira] [Created] (BIGTOP-3090) provisioner failed on fedora-26 when deploying jdk

[Jun He (JIRA)]

Jun He created BIGTOP-3090:
--

 Summary: provisioner failed on fedora-26 when deploying jdk
 Key: BIGTOP-3090
 URL: https://issues.apache.org/jira/browse/BIGTOP-3090
 Project: Bigtop
  Issue Type: Bug
  Components: provisioner
Affects Versions: 1.2.1
Reporter: Jun He
Assignee: Jun He
 Fix For: 1.3.0


Provisioner failed to deploy jdk on fedora-26. 
The error msg says:
{code:bash}
Error: Could not set 'link' on ensure: No such file or directory @ dir_chdir - 
/usr/lib/jvm/java-1.8.0-openjdk/jre/lib/security at 
64:/bigtop-home/bigtop-deploy/puppet/manifests/jdk.pp
{code}

The puppet recipe of jdk intends to setup cacerts link with pki/java/cacerts 
while jdk is not installed. This security failure will cause later bigtop 
components install failure.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BIGTOP-3089) Update provision config files with changes in 1.3.0 release

[Jun He (JIRA)]

Jun He created BIGTOP-3089:
--

 Summary: Update provision config files with changes in 1.3.0 
release
 Key: BIGTOP-3089
 URL: https://issues.apache.org/jira/browse/BIGTOP-3089
 Project: Bigtop
  Issue Type: Bug
  Components: provisioner
Affects Versions: 1.2.1
Reporter: Jun He
Assignee: Jun He
 Fix For: 1.3.0


v1.3.0 release will support 5 distros:
centos-7, fedora-26, debian-9, ubuntu-16.04, opensuse-42.3

config files in provisioner/docker should be updated to sync with these changes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BIGTOP-3088) provisioner failed to deploy puppet on CentOS-7

[Jun He (JIRA)]

Jun He created BIGTOP-3088:
--

 Summary: provisioner failed to deploy puppet on CentOS-7
 Key: BIGTOP-3088
 URL: https://issues.apache.org/jira/browse/BIGTOP-3088
 Project: Bigtop
  Issue Type: Bug
  Components: provisioner
Reporter: Jun He
Assignee: Jun He


Deploy with puppet on centos-7 failed with error:
{code:bash}
Error: Evaluation Error: Error while evaluating a Function Call, undefined 
method `repeated_param' for 
# at 
/bigtop-home/bigtop-deploy/puppet/manifests/site.pp:27:8 on node 
32d51c6d4736.bigtop.apache.org
Error: Evaluation Error: Error while evaluating a Function Call, undefined 
method `repeated_param' for 
# at 
/bigtop-home/bigtop-deploy/puppet/manifests/site.pp:27:8 on node 
32d51c6d4736.bigtop.apache.org
{code}
The root cause is puppet-stdlib (4.25) is not compatible with puppet <4.

Pin stdlib to 4.12.0 should fix this problem.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)