Re: [VOTE] Apache Hadoop Ozone 1.0.0 RC1

2020-08-31 Thread Xiaoyu Yao 
+1 (binding)

* Verify the checksums and signatures.
* Verify basic Ozone file system and S3 operations via CLI in secure docker
compose environment
* Run MR examples and teragen/terasort with ozone secure enabled.
* Verify EN/CN document rendering with hugo serve

Thanks Sammi for driving the release.

Regards,
Xiaoyu

On Mon, Aug 31, 2020 at 8:55 AM Shashikant Banerjee
 wrote:

> +1(binding)
>
> 1.Verified checksums
> 2.Verified signatures
> 3.Verified the output of `ozone version
> 4.Tried creating volume and bucket, write and read key, by Ozone shell
> 5.Verified basic Ozone Filesystem operations
>
> Thank you very much Sammi for putting up the release together.
>
> Thanks
> Shashi
>
> On Mon, Aug 31, 2020 at 4:35 PM Elek, Marton  wrote:
>
> > +1 (binding)
> >
> >
> > 1. verified signatures
> >
> > 2. verified checksums
> >
> > 3. verified the output of `ozone version` (includes the good git
> revision)
> >
> > 4. verified that the source package matches the git tag
> >
> > 5. verified source can be used to build Ozone without previous state
> > (docker run -v ... -it maven ... --> built from the source with zero
> > local maven cache during 16 minutes --> did on a sever at this time)
> >
> > 6. Verified Ozone can be used from binary package (cd compose/ozone &&
> > test.sh --> all tests were passed)
> >
> > 7. Verified documentation is included in SCM UI
> >
> > 8. Deployed to Kubernetes and executed Teragen on Yarn [1]
> >
> > 9. Deployed to Kubernetes and executed Spark (3.0) Word count (local
> > executor) [2]
> >
> > 10. Deployed to Kubernetes and executed Flink Word count [3]
> >
> > 11. Deployed to Kubernetes and executed Nifi
> >
> > Thanks very much Sammi, to drive this release...
> > Marton
> >
> > ps:  NiFi setup requires some more testing. Counters were not updated on
> > the UI and at some cases, I saw DirNotFound exceptions when I used
> > master. But during the last test with -rc1 it worked well.
> >
> > [1]: https://github.com/elek/ozone-perf-env/tree/master/teragen-ozone
> >
> > [2]: https://github.com/elek/ozone-perf-env/tree/master/spark-ozone
> >
> > [3]: https://github.com/elek/ozone-perf-env/tree/master/flink-ozone
> >
> >
> > On 8/25/20 4:01 PM, Sammi Chen wrote:
> > > RC1 artifacts are at:
> > > https://home.apache.org/~sammichen/ozone-1.0.0-rc1/
> > > 
> > >
> > > Maven artifacts are staged at:
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1278
> > > <
> https://repository.apache.org/content/repositories/orgapachehadoop-1277
> > >
> > >
> > > The public key used for signing the artifacts can be found at:
> > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >
> > > The RC1 tag in github is at:
> > > https://github.com/apache/hadoop-ozone/releases/tag/ozone-1.0.0-RC1
> > > 
> > >
> > > Change log of RC1, add
> > > 1. HDDS-4063. Fix InstallSnapshot in OM HA
> > > 2. HDDS-4139. Update version number in upgrade tests.
> > > 3. HDDS-4144, Update version info in hadoop client dependency readme
> > >
> > > *The vote will run for 7 days, ending on Aug 31th 2020 at 11:59 pm
> PST.*
> > >
> > > Thanks,
> > > Sammi Chen
> > >
> >
> > -
> > To unsubscribe, e-mail: ozone-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: ozone-dev-h...@hadoop.apache.org
> >
> >
>

Re: [VOTE] Apache Hadoop Ozone 0.5.0-beta RC2

2020-03-22 Thread Xiaoyu Yao 
+1 binding
Download source and verify signature.
Verify build and documents.
Deployed an 11 node cluster (3 om with ha, 6 datanodes, 1 scm and 1 s3g)
Verify multiple RATIS-3 pipelines are created as expected.
Tried ozone shell commands via o3 and o3fs, focus on security and HA
related.
 Only find a few minor issues that we can fix in followup JIRAs.
1) ozone getconf -ozonemanagers does not return all the om instances
bash-4.2$ ozone getconf -ozonemanagers
0.0.0.0
2) The document on specifying service/ID can be improved. More
specifically, the URI should give examples for the Service ID in HA.
Currently, it only mentions host/port.

ozone sh vol create /vol1
Service ID or host name must not be omitted when ozone.om.service.ids is
defined.
bash-4.2$ ozone sh vol create --help
Usage: ozone sh volume create [-hV] [--root] [-q=] [-u=]

Creates a volume for the specified user
 URI of the volume.
  Ozone URI could start with o3:// or without
prefix. URI
may contain the host and port of the OM server.
Both are
optional. If they are not specified it will be
identified from the config files.
3). ozone scmcli container list seems report incorrect numberOfKeys and
usedBytes
Also, container owner is set as the current leader om(om3), should we use
the om service id here instead?
bash-4.2$ ozone scmcli container list
{
  "state" : "OPEN",
  "replicationFactor" : "THREE",
  "replicationType" : "RATIS",
  "usedBytes" : 3813,
  "numberOfKeys" : 1,
...
bash-4.2$ ozone sh key list o3://id1/vol1/bucket1/
{
  "volumeName" : "vol1",
  "bucketName" : "bucket1",
  "name" : "k1",
  "dataSize" : 3813,
  "creationTime" : "2020-03-23T03:23:30.670Z",
  "modificationTime" : "2020-03-23T03:23:33.207Z",
  "replicationType" : "RATIS",
  "replicationFactor" : 3
}
{
  "volumeName" : "vol1",
  "bucketName" : "bucket1",
  "name" : "k2",
  "dataSize" : 3813,
  "creationTime" : "2020-03-23T03:18:46.735Z",
  "modificationTime" : "2020-03-23T03:20:15.005Z",
  "replicationType" : "RATIS",
  "replicationFactor" : 3
}


Run freon with random key generation.

Thanks Dinesh for driving the the release of Beta RC2.

Xiaoyu

On Sun, Mar 22, 2020 at 2:51 PM Aravindan Vijayan
 wrote:

> +1
> Deployed a 3 node cluster
> Tried ozone shell and filesystem commands
> Ran freon load generator
>
> Thanks Dinesh for working on the RC2.
>
> On Sun, Mar 15, 2020 at 7:27 PM Dinesh Chitlangia 
> wrote:
>
> > Hi Folks,
> >
> > We have put together RC2 for Apache Hadoop Ozone 0.5.0-beta.
> >
> > The RC artifacts are at:
> > https://home.apache.org/~dineshc/ozone-0.5.0-rc2/
> >
> > The public key used for signing the artifacts can be found at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > The maven artifacts are staged at:
> > https://repository.apache.org/content/repositories/orgapachehadoop-1262
> >
> > The RC tag in git is at:
> > https://github.com/apache/hadoop-ozone/tree/ozone-0.5.0-beta-RC2
> >
> > This release contains 800+ fixes/improvements [1].
> > Thanks to everyone who put in the effort to make this happen.
> >
> > *The vote will run for 7 days, ending on March 22nd 2020 at 11:59 pm
> PST.*
> >
> > Note: This release is beta quality, it’s not recommended to use in
> > production but we believe that it’s stable enough to try out the feature
> > set and collect feedback.
> >
> >
> > [1] https://s.apache.org/ozone-0.5.0-fixed-issues
> >
> > Thanks,
> > Dinesh Chitlangia
> >
>

[ANNOUNCE] New Apache Hadoop Committer - Siyao Meng

2020-03-20 Thread Xiaoyu Yao 
It's my pleasure to announce that Siyao Meng has been elected as committer
on the Apache Hadoop project recognizing his continued contributions to the
project.

Please join me in congratulating him.

Congratulations & Welcome aboard Siyao!

Xiaoyu Yao
(On behalf of the Hadoop PMC)

Re: [VOTE] Release Apache Hadoop Ozone 0.4.1-alpha

2019-10-12 Thread Xiaoyu Yao 
+1 binding. Verified
* Verify the signature.
* Build from source.
* Deploy docker compose in secure mode and verify ACL, sample MR jobs

Thanks,
Xiaoyu

On Fri, Oct 11, 2019 at 5:37 PM Hanisha Koneru 
wrote:

> Thank you Nanda for putting up the RC.
>
> +1 binding.
>
> Verified the following:
>   - Built from source
>   - Deployed to 5 node cluster and ran smoke tests.
>   - Ran sanity checks
>
> Thanks
> Hanisha
>
> > On Oct 4, 2019, at 10:42 AM, Nanda kumar  wrote:
> >
> > Hi Folks,
> >
> > I have put together RC0 for Apache Hadoop Ozone 0.4.1-alpha.
> >
> > The artifacts are at:
> > https://home.apache.org/~nanda/ozone/release/0.4.1/RC0/
> >
> > The maven artifacts are staged at:
> > https://repository.apache.org/content/repositories/orgapachehadoop-1238/
> >
> > The RC tag in git is at:
> > https://github.com/apache/hadoop/tree/ozone-0.4.1-alpha-RC0
> >
> > And the public key used for signing the artifacts can be found at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > This release contains 363 fixes/improvements [1].
> > Thanks to everyone who put in the effort to make this happen.
> >
> > *The vote will run for 7 days, ending on October 11th at 11:59 pm IST.*
> > Note: This release is alpha quality, it’s not recommended to use in
> > production but we believe that it’s stable enough to try out the feature
> > set and collect feedback.
> >
> >
> > [1] https://s.apache.org/yfudc
> >
> > Thanks,
> > Team Ozone
>
>

Re: VOTE: Hadoop Ozone 0.4.0-alpha RC2

2019-05-05 Thread Xiaoyu Yao 
+1 Binding. Thanks all who contributed to the release. 

+ Download sources and verify signature.
+ Build from source and ran docker-based ad-hot security tests.
++ From 1 datanode scale to 3 datanodes, verify certificates were correctly 
issued when security enabled
++ Smoke test for both non-secure and secure mode.
++ Put/Get/Delete/Rename Key with 
+++ Kerberos testing
+++ Delegation token testing with DTUtil CLI and MR jobs.
+++ S3 token.

Just have one minor question for the expanded source code which points to 
hadoop-3.3.0-SNAPSHOT-src-with-hdds/hadoop-ozone. But in hadoop-ozone/pom.xml, 
we explicitly declare dependency on Hadoop 3.2.0.
I understand we just take the trunk source code(3.3.0-SNAPSHOT up to the 
ozone-0.4 RC) here, should we fix this by giving the git hash of the trunk or 
clarify it to avoid confusion? 
This might be done by just updating the name of the binaries without reset the 
release itself. 

-Xiaoyu
 

On 5/3/19, 4:07 PM, "Dinesh Chitlangia"  wrote:

+1 (non-binding)

- Built from sources and ran smoke test
- Verified all checksums
- Toggled audit log and verified audit parser tool

Thanks Ajay for organizing the release.

Cheers,
Dinesh



On 5/3/19, 5:42 PM, "Eric Yang"  wrote:

+1

On 4/29/19, 9:05 PM, "Ajay Kumar"  wrote:

Hi All,



We have created the third release candidate (RC2) for Apache Hadoop 
Ozone 0.4.0-alpha.



This release contains security payload for Ozone. Below are some 
important features in it:



  *   Hadoop Delegation Tokens and Block Tokens supported for Ozone.
  *   Transparent Data Encryption (TDE) Support - Allows data 
blocks to be encrypted-at-rest.
  *   Kerberos support for Ozone.
  *   Certificate Infrastructure for Ozone  - Tokens use PKI 
instead of shared secrets.
  *   Datanode to Datanode communication secured via mutual TLS.
  *   Ability secure ozone cluster that works with Yarn, Hive, and 
Spark.
  *   Skaffold support to deploy Ozone clusters on K8s.
  *   Support S3 Authentication Mechanisms like - S3 v4 
Authentication protocol.
  *   S3 Gateway supports Multipart upload.
  *   S3A file system is tested and supported.
  *   Support for Tracing and Profiling for all Ozone components.
  *   Audit Support - including Audit Parser tools.
  *   Apache Ranger Support in Ozone.
  *   Extensive failure testing for Ozone.

The RC artifacts are available at 
https://home.apache.org/~ajay/ozone-0.4.0-alpha-rc2/



The RC tag in git is ozone-0.4.0-alpha-RC2 (git hash 
4ea602c1ee7b5e1a5560c6cbd096de4b140f776b)



Please try 
out,
 vote, or just give us feedback.



The vote will run for 5 days, ending on May 4, 2019, 04:00 UTC.



Thank you very much,

Ajay



-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org




-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: VOTE: Hadoop Ozone 0.4.0-alpha RC1

2019-04-20 Thread Xiaoyu Yao 

+1 (binding)

- Build from source
- Misc security tests with docker compose
- MR and Spark sample jobs with secure ozone cluster

—Xiaoyu

> On Apr 19, 2019, at 3:40 PM, Anu Engineer  
> wrote:
> 
> +1 (Binding)
> 
> -- Verified the checksums.
> -- Built from sources.
> -- Sniff tested the functionality.
> 
> --Anu
> 
> 
> On Mon, Apr 15, 2019 at 4:09 PM Ajay Kumar 
> wrote:
> 
>> Hi all,
>> 
>> We have created the second release candidate (RC1) for Apache Hadoop Ozone
>> 0.4.0-alpha.
>> 
>> This release contains security payload for Ozone. Below are some important
>> features in it:
>> 
>>  *   Hadoop Delegation Tokens and Block Tokens supported for Ozone.
>>  *   Transparent Data Encryption (TDE) Support - Allows data blocks to be
>> encrypted-at-rest.
>>  *   Kerberos support for Ozone.
>>  *   Certificate Infrastructure for Ozone  - Tokens use PKI instead of
>> shared secrets.
>>  *   Datanode to Datanode communication secured via mutual TLS.
>>  *   Ability secure ozone cluster that works with Yarn, Hive, and Spark.
>>  *   Skaffold support to deploy Ozone clusters on K8s.
>>  *   Support S3 Authentication Mechanisms like - S3 v4 Authentication
>> protocol.
>>  *   S3 Gateway supports Multipart upload.
>>  *   S3A file system is tested and supported.
>>  *   Support for Tracing and Profiling for all Ozone components.
>>  *   Audit Support - including Audit Parser tools.
>>  *   Apache Ranger Support in Ozone.
>>  *   Extensive failure testing for Ozone.
>> 
>> The RC artifacts are available at
>> https://home.apache.org/~ajay/ozone-0.4.0-alpha-rc1
>> 
>> The RC tag in git is ozone-0.4.0-alpha-RC1 (git hash
>> d673e16d14bb9377f27c9017e2ffc1bcb03eebfb)
>> 
>> Please try out<
>> https://cwiki.apache.org/confluence/display/HADOOP/Running+via+Apache+Release>,
>> vote, or just give us feedback.
>> 
>> The vote will run for 5 days, ending on April 20, 2019, 19:00 UTC.
>> 
>> Thank you very much,
>> 
>> Ajay
>> 
>> 
>>

Re: [VOTE] Propose to start new Hadoop sub project "submarine"

2019-02-04 Thread Xiaoyu Yao 
+1

-Xiaoyu

Sent from my iPhone

> On Feb 4, 2019, at 7:54 AM, Suma Shivaprasad  
> wrote:
> 
> +1 ( non binding)
> 
> On Mon, Feb 4, 2019 at 7:23 AM Shashikant Banerjee <
> sbaner...@hortonworks.com> wrote:
> 
>> +1 (non-binding)
>> 
>> Thanks
>> Shashi
>> 
>> On 2/4/19, 8:27 PM, "Elek, Marton"  wrote:
>> 
>>+1 (non-binding)
>> 
>>(my arguments are in the discuss thread. small move, huge benefit)
>> 
>>Thanks,
>>Marton
>> 
>>>On 2/1/19 11:15 PM, Wangda Tan wrote:
>>> Hi all,
>>> 
>>> According to positive feedbacks from the thread [1]
>>> 
>>> This is vote thread to start a new subproject named
>> "hadoop-submarine"
>>> which follows the release process already established for ozone.
>>> 
>>> The vote runs for usual 7 days, which ends at Feb 8th 5 PM PDT.
>>> 
>>> Thanks,
>>> Wangda Tan
>>> 
>>> [1]
>>> 
>> https://lists.apache.org/thread.html/f864461eb188bd12859d51b0098ec38942c4429aae7e4d001a633d96@%3Cyarn-dev.hadoop.apache.org%3E
>>> 
>> 
>>-
>>To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>>For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>> 
>> 
>> 
>> 

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [DISCUSS] Making submarine to different release model like Ozone

2019-02-01 Thread Xiaoyu Yao 
+1, thanks for bringing this up, Wangda. This will help expanding the Hadoop 
ecosystem by supporting new AI/ML workloads. 

Thanks,
Xiaoyu
On 2/1/19, 10:58 AM, "Dinesh Chitlangia"  wrote:

+1 This is a fantastic recommendation given the increasing interest in ML 
across the globe.

Thanks,
Dinesh



On 2/1/19, 1:54 PM, "Ajay Kumar"  wrote:

+1, Thanks for driving this. With rise of use cases running ML along 
with traditional applications this will be of great help.

Thanks,
Ajay   

On 2/1/19, 10:49 AM, "Suma Shivaprasad"  
wrote:

+1. Thanks for bringing this up Wangda.

Makes sense to have Submarine follow its own release cadence given 
the good
momentum/adoption so far. Also, making it run with older versions 
of Hadoop
would drive higher adoption.

Suma

On Fri, Feb 1, 2019 at 9:40 AM Eric Yang  
wrote:

> Submarine is an application built for YARN framework, but it does 
not have
> strong dependency on YARN development.  For this kind of 
projects, it would
> be best to enter Apache Incubator cycles to create a new 
community.  Apache
> commons is the only project other than Incubator that has 
independent
> release cycles.  The collection is large, and the project goal is
> ambitious.  No one really knows which component works with each 
other in
> Apache commons.  Hadoop is a much more focused project on 
distributed
> computing framework and not incubation sandbox.  For alignment 
with Hadoop
> goals, and we want to prevent Hadoop project to be overloaded 
while
> allowing good ideas to be carried forwarded in Apache incubator.  
Put on my
> Apache Member hat, my vote is -1 to allow more independent 
subproject
> release cycle in Hadoop project that does not align with Hadoop 
project
> goals.
>
> Apache incubator process is highly recommended for Submarine:
> https://incubator.apache.org/policy/process.html This allows 
Submarine to
> develop for older version of Hadoop like Spark works with 
multiple versions
> of Hadoop.
>
> Regards,
> Eric
>
> On 1/31/19, 10:51 PM, "Weiwei Yang"  wrote:
>
> Thanks for proposing this Wangda, my +1 as well.
> It is amazing to see the progress made in Submarine last 
year, the
> community grows fast and quiet collaborative. I can see the 
reasons to get
> it release faster in its own cycle. And at the same time, the 
Ozone way
> works very well.
>
> —
> Weiwei
> On Feb 1, 2019, 10:49 AM +0800, Xun Liu , 
wrote:
> > +1
> >
> > Hello everyone,
> >
> > I am Xun Liu, the head of the machine learning team at 
Netease
> Research Institute. I quite agree with Wangda.
> >
> > Our team is very grateful for getting Submarine machine 
learning
> engine from the community.
> > We are heavy users of Submarine.
> > Because Submarine fits into the direction of our big data 
team's
> hadoop technology stack,
> > It avoids the needs to increase the manpower investment in 
learning
> other container scheduling systems.
> > The important thing is that we can use a common YARN 
cluster to run
> machine learning,
> > which makes the utilization of server resources more 
efficient, and
> reserves a lot of human and material resources in our previous 
years.
> >
> > Our team have finished the test and deployment of the 
Submarine and
> will provide the service to our e-commerce department (
> http://www.kaola.com/) shortly.
> >
> > We also plan to provides the Submarine engine in our 
existing YARN
> cluster in the next six months.
> > Because we have a lot of product departments need to use 
machine
> learning services,
> > for example:
> > 1) Game department (http://game.163.com/) needs AI battle 
training,
> > 2) News department (http://www.163.com) needs news 
recommendation,
> > 3) Mailbox department (http://www.163.com) requires 
anti-spam and
> illegal detection,
> > 4) Music department (https://music.163.com/) requires music
> recommendation,
> > 5)

Re: HDFS/HDDS unit tests running failed in Jenkins precommit building

2018-11-26 Thread Xiaoyu Yao 
Hi Akira,

Thanks for the pointer. I pulled the change of HADOOP-15916 from trunk to 
HDDS-4 branch last week but we are still seeing similar failures from Jenkins 
like this one 
(https://builds.apache.org/job/PreCommit-HDDS-Build/1801/artifact/out/patch-unit-hadoop-ozone_common.txt).
 Can you help take a look?

Thanks,
Xiaoyu 
 

On 11/9/18, 1:29 AM, "Akira Ajisaka"  wrote:

+ common-dev, mapreduce-dev, yarn-dev

Probably this issue is caused by
https://issues.apache.org/jira/browse/SUREFIRE-1588 and fixed by Maven
surefire plugin 3.0.0-M1. I filed a JIRA and uploaded a patch to
upgrade the version of the plugin. Please check
https://issues.apache.org/jira/browse/HADOOP-15916

Regards,
Akira
2018年11月9日(金) 6:10 Lin,Yiqun(vip.com) :
>
> Hi developers,
>
> Recently, I found following error frequently appeared in HDFS/HDDS 
Jenkins building.
> The link: 
https://builds.apache.org/job/PreCommit-HDDS-Build/1632/artifact/out/patch-unit-hadoop-ozone_ozone-manager.txt
>
> [ERROR] ExecutionException The forked VM terminated without properly 
saying goodbye. VM crash or System.exit called?
> [ERROR] Command was /bin/sh -c cd 
/testptch/hadoop/hadoop-ozone/ozone-manager && 
/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xmx2048m 
-XX:+HeapDumpOnOutOfMemoryError -DminiClusterDedicatedDirs=true -jar 
/testptch/hadoop/hadoop-ozone/ozone-manager/target/surefire/surefirebooter6481080145571841952.jar
 /testptch/hadoop/hadoop-ozone/ozone-manager/target/surefire 
2018-11-07T22-53-35_334-jvmRun1 surefire2897373403289443808tmp 
surefire_42678601136131093095tmp
>
> And this error makes the unit tests not be real tested in Jenkins 
precommit buildings. Anyone who knows the root cause of this?
>
> Thanks
> Yiqun
> 
本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人，谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容，或将其用于其他任何目的或向任何人披露。谢谢您的合作！
 This communication is intended only for the addressee(s) and may contain 
information that is privileged and confidential. You are hereby notified that, 
if you are not an intended recipient listed above, or an authorized employee or 
agent of an addressee of this communication responsible for delivering e-mail 
messages to an intended recipient, any dissemination, distribution or 
reproduction of this communication (including any attachments hereto) is 
strictly prohibited. If you have received this communication in error, please 
notify us immediately by a reply e-mail addressed to the sender and permanently 
delete the original e-mail communication and any attachments from all storage 
devices without making or otherwise retaining a copy.

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-9046) Fix overlapping netty classes warnings when building YARN CSI module

2018-11-21 Thread Xiaoyu Yao (JIRA) 
Xiaoyu Yao created YARN-9046:


 Summary: Fix overlapping netty classes warnings when building YARN 
CSI module
 Key: YARN-9046
 URL: https://issues.apache.org/jira/browse/YARN-9046
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Xiaoyu Yao


{code}

[*WARNING*] netty-codec-4.1.27.Final.jar, netty-all-4.1.27.Final.jar define 197 
overlapping classes: 

[*WARNING*]   - io.netty.handler.codec.compression.ZlibUtil

[*WARNING*]   - io.netty.handler.codec.CharSequenceValueConverter

[*WARNING*]   - io.netty.handler.codec.compression.Bzip2Encoder$3

[*WARNING*]   - io.netty.handler.codec.DefaultHeaders$1

[*WARNING*]   - io.netty.handler.codec.compression.Bzip2BlockDecompressor

[*WARNING*]   - io.netty.handler.codec.compression.Lz4FrameDecoder$1

[*WARNING*]   - io.netty.handler.codec.compression.SnappyFrameDecoder

[*WARNING*]   - io.netty.handler.codec.compression.DecompressionException

[*WARNING*]   - io.netty.handler.codec.compression.Bzip2DivSufSort$TRBudget

[*WARNING*]   - io.netty.handler.codec.compression.Bzip2Decoder$State

[*WARNING*]   - 187 more...

{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop Ozone 0.2.1-alpha (RC0)

2018-09-26 Thread Xiaoyu Yao 
Thanks all for the efforts to reach the 1st ozone release.

+1 (binding).

1. Download src and verify Signature and Checksum.
2. Build with sources.
3. Run Acceptance tests.
3. Run docker cluster and scale tests. 
4. Verify ozone sh and ozone fs shell with basic volume/bucket/key inter-ops.
5. Run Freon with random key creations and verify the result on OM/SCM UI.
6. Review the document shipped with ozone UI.

Only found a minor issue in the doc where the acceptance test location need to 
be updated - tracked by HDDS-556. 

On 9/26/18, 8:51 AM, "Ajay Kumar"  wrote:

@Elek, Marton thanks  for working on this release.

+1 (non-binding)

1. Verified Signature and Checksum
2. Built from Sources
3. Ran docker cluster
5. Tested basic commands to create,delete volumes,buckets,keys.
6. Verified SCM and OM UI.


On 9/26/18, 5:32 AM, "Nandakumar Vadivelu"  
wrote:

Thanks Marton for all effort to make this release happen.

+1 (non-binding)

1. Verified Signature and Checksum
2. Build from Sources
3. Ran 5 node cluster using docker
4. Installed pseudo distributed cluster
5. Ran RPC and REST commands against the cluster
6. Verified SCM and OM UI

-Nanda

On 9/20/18, 3:19 AM, "Elek, Marton"  wrote:

Hi all,

After the recent discussion about the first Ozone release I've 
created 
the first release candidate (RC0) for Apache Hadoop Ozone 
0.2.1-alpha.

This release is alpha quality: it’s not recommended to use in 
production 
but we believe that it’s stable enough to try it out the feature 
set and 
collect feedback.

The RC artifacts are available from: 
https://home.apache.org/~elek/ozone-0.2.1-alpha-rc0/

The RC tag in git is: ozone-0.2.1-alpha-RC0 (968082ffa5d)

Please try the release and vote; the vote will run for the usual 5 
working days, ending on September 26, 2018 10pm UTC time.

The easiest way to try it out is:

1. Download the binary artifact
2. Read the docs at ./docs/index.html
3. TLDR; cd compose/ozone && docker-compose up -d


Please try it out, vote, or just give us feedback.

Thank you very much,
Marton

ps: At next week, we will have a BoF session at ApacheCon North 
Europe, 
Montreal on Monday evening. Please join, if you are interested, or 
need 
support to try out the package or just have any feedback.



-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org




-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org




-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org




-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: 答复: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

2017-11-03 Thread Xiaoyu Yao 
Hi Lei,

Thank you for your interest in Ozone. Let me answer each of the
specific questions.

> As the current state of Ozone implementation, what are the major
> benefits of using today’s Ozone over HDFS?


Scale -  HDFS tops out at 500/700 million keys; Ozone primary 
use case is to go beyond that limit. Ozone can scale block space 
and namespace independently. Ozone also provides object store 
semantics which is simpler and scales better


Enabling new workloads on HDFS -  For example, we see lots of 
customers moving to a cloud-like model, where they have a compute 
cluster and storage cluster. This allows them to move workloads to 
the cloud and back seamlessly. We see a marked increase of 
Docker/Kubernetes enabled workloads. The pain point for most 
Docker/VM deployments is lack of storage. Ozone is an excellent 
fit for Docker/Kubernetes deployments.

Ease of Management and use - Ozone has learned very 
valuable lessons from HDFS. It comes with a good set of 
management tools and tries to avoid very complicated setup.


> Giving that its missing features like HDFS-12680 and HDFS-12697, and
> the closing of Hadoop 3.0 release, should we wait for a late merge
> when Ozone is more mature ?

Both HDFS-12680 (lease manager) and HDFS-12697 (Ozone services 
stay disabled in secure setup) are resolved in the past weeks. 
We are targeting the merge for trunk not 3.0.

> Or more generally, why should this merge to a release branch happen
> now, when Ozone is not yet usable by users? Staying on a feature
> branch seems like it's still the right place to Me.

Let me repeat that we are not merging to the 3.0 branch. We are
merging to trunk and we do not intend to backport this to 3.0 code
base.

Ozone is certainly usable. We have written and read billions of keys
into ozone. I would think that it more like Erasure coding when we
merged. We want ozone to be used/tested when people start using 3.1
release. Yes, it is an Alpha feature, having an Alpha release out in
the community is the best way to mature Ozone.


> For the existing HDFS user, could you address the semantic gaps
> between Ozone / Ozone File System and HDFS.

Ozone file system offers a Hadoop compatible file system. For the
first release, we are targeting YARN, Hive, and Spark as the principle
workloads. These applications are functional with Ozone file system.

> It would be great to illustrate what is the expected use cases for
> Ozone giving its different architecture and design decisions?

We expect almost all real use case of ozone to come via Ozone File
System. Hence our focus has been to make sure that (YARN, Hive and
Spark) work well with this system. Ozone file system does the right
magic on behalf of the users for now.

> Like no append, no atomic rename and etc.

This is similar to S3 -- the rise of cloud-based object stores has made 
it very easy for ozone. In fact, the work done by other stacks (Hive, Spark 
etc.) 
to enable big data workload in cloud is extremely helpful for ozone.


> A follow question, was it able to run any of today’s Hadoop
> applications (MR, Spark, Impala, Presto and etc) on Ozone directly, or
> against OZoneFileSystem? I think a performance / scalability gain or
> extended functionality should be the prerequisites for the merge.
> Additionally, I believe such tests will reveal the potential caveats
> if any.

We have run, Mapreduce (pretty much all standard applications along
with Distcp works well), YARN, Hive and Spark against ozone with NO
Modifications to MR,YARN, Hive or Spark.

We have never tried out Impala or Presto, but if they are known to work
well against Hadoop compatible file systems, I am hopeful that they
will work as well. Please feel free to test and report if you run into
any issues.


> * Ozone’s architecture shows great potential to address NN
> scalability.  However it looks like a XXL effort to me, considering
> the fact that 1) the community had multiple unfinished attempts to
> simply separate namespace and block management within the same NN
> process,

You are absolutely correct. We have learned from those experiences. We
think that separating namespace and block space in the same NN process
does not address the core issue of NN scale. And, also as you clearly 
mentioned they are unfinished. 

With Ozone, we have separated out a block service. Once it is known 
to be stable, we will use that in Namenode, thus achieving the full 
separation. Ozone FS and Ozone object store are intermediate steps
 to solving the scale issue for HDFS.

> *and 2) many existing features like snapshot, append, erasure coding,
> and etc, are not straightforward to be implemented in today’s ozone
> design. Could you share your opinions on this matter?

Ozone is well prepared to implement each of these features. We have
many design documents for ozone posted in the sub-JIRAs. For example,
please take a look at the versioning doc to understand how Ozone’s block
layer really offers.

[jira] [Created] (YARN-2519) Credential Provider related unit tests failed on Windows

2014-09-06 Thread Xiaoyu Yao (JIRA) 
Xiaoyu Yao created YARN-2519:


 Summary: Credential Provider related unit tests failed on Windows
 Key: YARN-2519
 URL: https://issues.apache.org/jira/browse/YARN-2519
 Project: Hadoop YARN
  Issue Type: Test
  Components: webapp
Affects Versions: 2.4.1
Reporter: Xiaoyu Yao


Reported by: Xiaomara and investigated by Chris Nauroth.

The credential provider related unit tests failed on Windows. The tests try to 
set up a URI by taking the build test directory and concatenating it with other 
strings containing the rest of the URI format, i.e.:
{code}
  public void testFactory() throws Exception {
Configuration conf = new Configuration();
conf.set(CredentialProviderFactory.CREDENTIAL_PROVIDER_PATH,
UserProvider.SCHEME_NAME + :///, +
JavaKeyStoreProvider.SCHEME_NAME + ://file + tmpDir + 
/test.jks);
{code}

This logic is incorrect on Windows, because the file path separator will be 
'\', which violates URI syntax. Forward slash is not permitted.

The proper fix is to always do path/URI construction through the 
org.apache.hadoop.fs.Path class, specifically using the constructors that take 
explicit parent and child arguments.

The affected unit tests are:
{code}
* TestWebAppUtils
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)