Apache Hadoop qbt Report: branch2.10+JDK7 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86/687/ No changes - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17045) `Configuration` javadoc describes supporting environment variables, but the feature is not available
Nick Dimiduk created HADOOP-17045: - Summary: `Configuration` javadoc describes supporting environment variables, but the feature is not available Key: HADOOP-17045 URL: https://issues.apache.org/jira/browse/HADOOP-17045 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.10.0 Reporter: Nick Dimiduk In Hadoop 2.10.0, the javadoc on the `Configuration` class describes the ability to read values from environment variables. However, this feature isn't implemented until HADOOP-9642, which shipped in 3.0.0. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [DISCUSS] making Ozone a separate Apache project
+1. Thanks, Subru On Thu, May 14, 2020 at 10:00 PM Akira Ajisaka wrote: > +1 > > -Akira > > On Wed, May 13, 2020 at 4:53 PM Elek, Marton wrote: > > > > > > > I would like to start a discussion to make a separate Apache project for > > Ozone > > > > > > > > ### HISTORY [1] > > > > * Apache Hadoop Ozone development started on a feature branch of > > Hadoop repository (HDFS-7240) > > > > * In the October of 2017 a discussion has been started to merge it to > > the Hadoop main branch > > > > * After a long discussion it's merged to Hadoop trunk at the March of > > 2018 > > > > * During the discussion of the merge, it was suggested multiple times > > to create a separated project for the Ozone. But at that time: > > 1). Ozone was tightly integrated with Hadoop/HDFS > > 2). There was an active plan to use Block layer of Ozone (HDDS or > > HDSL at that time) as the block level of HDFS > > 3). The community of Ozone was a subset of the HDFS community > > > > * The first beta release of Ozone was just released. Seems to be a > > good time before the first GA to make a decision about the future. > > > > > > > > ### WHAT HAS BEEN CHANGED > > > > During the last years Ozone became more and more independent both at > > the community and code side. The separation has been suggested again and > > again (for example by Owen [2] and Vinod [3]) > > > > > > > > From COMMUNITY point of view: > > > > > >* Fortunately more and more new contributors are helping Ozone. > > Originally the Ozone community was a subset of HDFS project. But now a > > bigger and bigger part of the community is related to Ozone only. > > > >* It seems to be easier to _build_ the community as a separated > project. > > > >* A new, younger project might have different practices > > (communication, commiter criteria, development style) compared to old, > > mature project > > > >* It's easier to communicate (and improve) these standards in a > > separated projects with clean boundaries > > > >* Separated project/brand can help to increase the adoption rate and > > attract more individual contributor (AFAIK it has been seen in Submarine > > after a similar move) > > > > * Contribution process can be communicated more easily, we can make > > first time contribution more easy > > > > > > > > From CODE point of view Ozone became more and more independent: > > > > > > * Ozone has different release cycle > > > > * Code is already separated from Hadoop code base > > (apache/hadoop-ozone.git) > > > > * It has separated CI (github actions) > > > > * Ozone uses different (more strict) coding style (zero toleration of > > unit test / checkstyle errors) > > > > * The code itself became more and more independent from Hadoop on > > Maven level. Originally it was compiled together with the in-tree latest > > Hadoop snapshot. Now it depends on released Hadoop artifacts (RPC, > > Configuration...) > > > > * It starts to use multiple version of Hadoop (on client side) > > > > * Volume of resolved issues are already very high on Ozone side (Ozone > > had slightly more resolved issues than HDFS/YARN/MAPREDUCE/COMMON all > > together in the last 2-3 months) > > > > > > Summary: Before the first Ozone GA release, It seems to be a good time > > to discuss the long-term future of Ozone. Managing it as a separated TLP > > project seems to have more benefits. > > > > > > Please let me know what your opinion is... > > > > Thanks a lot, > > Marton > > > > > > > > > > > > [1]: For more details, see: > > https://github.com/apache/hadoop-ozone/blob/master/HISTORY.md > > > > [2]: > > > > > https://lists.apache.org/thread.html/0d0253f6e5fa4f609bd9b917df8e1e4d8848e2b7fdb3099b730095e6%40%3Cprivate.hadoop.apache.org%3E > > > > [3]: > > > > > https://lists.apache.org/thread.html/8be74421ea495a62e159f2b15d74627c63ea1f67a2464fa02c85d4aa%40%3Chdfs-dev.hadoop.apache.org%3E > > > > - > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > > > > >
[jira] [Resolved] (HADOOP-17044) Revert "HADOOP-8143. Change distcp to have -pb on by default
[ https://issues.apache.org/jira/browse/HADOOP-17044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-17044. - Resolution: Fixed > Revert "HADOOP-8143. Change distcp to have -pb on by default > > > Key: HADOOP-17044 > URL: https://issues.apache.org/jira/browse/HADOOP-17044 > Project: Hadoop Common > Issue Type: Bug > Components: tools/distcp >Affects Versions: 3.0.3, 3.3.0, 3.2.1, 3.1.3 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Fix For: 3.0.4, 3.2.2, 3.3.1, 3.1.5 > > > revert the HADOOP-8143. "distcp -pb as default" feature as it was > * breaking s3a uploads > * breaking incremental uploads to any object store -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16756) distcp -update to S3A; abfs, etc always overwrites due to block size mismatch
[
https://issues.apache.org/jira/browse/HADOOP-16756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-16756.
-
Fix Version/s: 3.1.5
3.3.1
3.2.2
3.0.4
Resolution: Fixed
> distcp -update to S3A; abfs, etc always overwrites due to block size mismatch
> -
>
> Key: HADOOP-16756
> URL: https://issues.apache.org/jira/browse/HADOOP-16756
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Daisuke Kobayashi
>Priority: Major
> Fix For: 3.0.4, 3.2.2, 3.3.1, 3.1.5
>
>
> Distcp over S3A always copies all source files no matter the files are
> changed or not. This is opposite to the statement in the doc below.
> [http://hadoop.apache.org/docs/current/hadoop-distcp/DistCp.html]
> {noformat}
> And to use -update to only copy changed files.
> {noformat}
> CopyMapper compares file length as well as block size before copying. While
> the file length should match, the block size does not. This is apparently
> because the returned block size from S3A is always 32MB.
> [https://github.com/apache/hadoop/blob/release-3.2.0-RC1/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java#L348]
> I'd suppose we should update the documentation or make code change.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17044) Revert "HADOOP-8143. Change distcp to have -pb on by default
Steve Loughran created HADOOP-17044: --- Summary: Revert "HADOOP-8143. Change distcp to have -pb on by default Key: HADOOP-17044 URL: https://issues.apache.org/jira/browse/HADOOP-17044 Project: Hadoop Common Issue Type: Bug Components: tools/distcp Affects Versions: 3.1.3, 3.2.1, 3.0.3, 3.3.0 Reporter: Steve Loughran Assignee: Steve Loughran Fix For: 3.0.4, 3.1.4, 3.2.2, 3.3.1 revert the HADOOP-8143. "distcp -pb as default" feature as it was * breaking s3a uploads * breaking incremental uploads to any object store -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
