date:20160815

[jira] [Created] (MAPREDUCE-6755) Report correct spill/output size for map in nativetask

2016-08-15 Thread He Tianyi (JIRA)

He Tianyi created MAPREDUCE-6755:


 Summary: Report correct spill/output size for map in nativetask
 Key: MAPREDUCE-6755
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6755
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: nativetask
Affects Versions: 2.6.0
Reporter: He Tianyi
Priority: Minor


Currently nativetask always report '-1' as size when calling methods in 
{{MapOutputFile}}.
This is an obstacle for custom {{MapOutputFile}} implementations where size 
matters (e.g. determine which disk to use).

The issue propose to estimate spill/output/index size in nativetask like java 
implementation does.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.3 RC1

2016-08-15 Thread Jason Lowe

+1 (binding)
- Verified signatures and digests- Built from source with native support- 
Deployed a pseudo-distributed cluster- Ran some sample jobs
Jason

  From: Vinod Kumar Vavilapalli 
 To: "common-...@hadoop.apache.org" ; 
hdfs-...@hadoop.apache.org; yarn-...@hadoop.apache.org; 
"mapreduce-dev@hadoop.apache.org"  
Cc: Vinod Kumar Vavilapalli 
 Sent: Friday, August 12, 2016 11:45 AM
 Subject: [VOTE] Release Apache Hadoop 2.7.3 RC1
   
Hi all,

I've created a release candidate RC1 for Apache Hadoop 2.7.3.

As discussed before, this is the next maintenance release to follow up 2.7.2.

The RC is available for validation at: 
http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ 


The RC tag in git is: release-2.7.3-RC1

The maven artifacts are available via repository.apache.org 
 at 
https://repository.apache.org/content/repositories/orgapachehadoop-1045/ 


The release-notes are inside the tar-balls at location 
hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html 
 for your 
quick perusal.

As you may have noted,
 - few issues with RC0 forced a RC1 [1]
 - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 
2.7.3 (along with every other Hadoop release) to slip by quite a bit. This 
release's related discussion thread is linked below: [2].

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[1] [VOTE] Release Apache Hadoop 2.7.3 RC0: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 

[2]: 2.7.3 release plan: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html

Re: [VOTE] Release Apache Hadoop 2.7.3 RC1

2016-08-15 Thread Vinod Kumar Vavilapalli

Thanks Marco. It was a Thursday late-night slip-up.

Fixed the dates and replaced the bits, so the voting can continue.

FYI, they aren’t binding though - as it all depends on how the release voting 
goes. One should usually only trust the release-date published on the website.

Thanks
+Vinod

> On Aug 13, 2016, at 1:35 PM, Marco Zühlke  > wrote:
> 
> Hi Vinod,
> 
> I'm not sure if this is relevant, but you changed the release date in the 
> CHANGES.txt 
> 
>  files to 2016-09-19.
> I guess you have meant 2016-08-19.
> 
> See: 
> https://github.com/apache/hadoop/commit/5474c9e736d4c44a603a3f6749130b67cd4da52f
>  
> 
> 
> 
> Thanks,
> Marco
> 
> 
> 
> 2016-08-12 18:45 GMT+02:00 Vinod Kumar Vavilapalli  >:
> Hi all,
> 
> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
> 
> As discussed before, this is the next maintenance release to follow up 2.7.2.
> 
> The RC is available for validation at: 
> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ 
>  
>  >
> 
> The RC tag in git is: release-2.7.3-RC1
> 
> The maven artifacts are available via repository.apache.org 
>   > at 
> https://repository.apache.org/content/repositories/orgapachehadoop-1045/ 
>  
>  >
> 
> The release-notes are inside the tar-balls at location 
> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
> this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html 
>  
>  > for 
> your quick perusal.
> 
> As you may have noted,
>  - few issues with RC0 forced a RC1 [1]
>  - a very long fix-cycle for the License & Notice issues (HADOOP-12893) 
> caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. 
> This release's related discussion thread is linked below: [2].
> 
> Please try the release and vote; the vote will run for the usual 5 days.
> 
> Thanks,
> Vinod
> 
> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: 
> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 
>  
>  >
> [2]: 2.7.3 release plan: 
> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 
>  
>  >
>

Re: [Release thread] 2.6.5 release activities

2016-08-15 Thread Allen Wittenauer

> On Aug 12, 2016, at 8:19 AM, Junping Du  wrote:
> 
>  In this community, we are so aggressive to drop Java 7 support in 3.0.x 
> release. Here, why we are so conservative to keep releasing new bits to 
> support Java 6?

I don't view a group of people putting bug fixes into a micro release 
as particularly conservative.  If a group within the community wasn't 
interested in doing it, 2.6.5 wouldn't be happening.

But let's put the releases into context, because I think it tells a 
more interesting story.

* hadoop 2.6.x = EOLed JREs (6,7) 
* hadoop 2.7 -> hadoop 2.x = transitional (7,8)
* hadoop 3.x = JRE 8
* hadoop 4.x = JRE 9 

There are groups of people still using JDK6 and they want bug fixes in 
a maintenance release.  Boom, there's 2.6.x.

Hadoop 3.x has been pushed off for years for "reasons".  So we still 
have releases coming off of branch-2.  If 2.7 had been released as 3.x, this 
chart would look less weird. But it wasn't thus 2.x has this weird wart in the 
middle of that supports JDK7 and JDK8.  Given the public policy and roadmaps of 
at least one major vendor at the time of this writing, we should expect to see 
JDK7 support for at least the next two years after 3.x appears. Bang, there's 
2.x, where x is some large number.

Then there is the future.  People using JRE 8 want to use newer 
dependencies.  A reasonable request. Some of these dependency updates won't 
work with JRE 7.   We can't do that in hadoop 2.x in any sort of compatible way 
without breaking the universe. (Tons of JIRAs on this point.) This means we can 
only do it in 3.x (re: Hadoop Compatibility Guidelines).  Kapow, there's 3.x

The log4j community has stated that v1 won't work with JDK9. In turn, 
this means we'll need to upgrade to v2 at some point.  Upgrading to v2 will 
break the log4j properties file (and maybe other things?). Another incompatible 
change and it likely won't appear until Apache Hadoop v4 unless someone takes 
the initiative to fix it before v3 hits store shelves.  This makes JDK9 the 
likely target for Apache Hadoop v4.  

Having major release cadences tied to JRE updates isn't necessarily a 
bad thing and definitely forces the community to a) actually stop beating 
around the bush on majors and b) actually makes it relatively easy to determine 
what the schedule looks like to some degree.

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2016-08-15 Thread Apache Jenkins Server

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/134/

[Aug 14, 2016 10:01:21 PM] (varunsaxena) YARN-5491. Fix random failure of




-1 overall


The following subsystems voted -1:
asflicense unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.yarn.logaggregation.TestAggregatedLogFormat 
   
hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager
 
   hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices 
   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization 
   hadoop.yarn.server.TestContainerManagerSecurity 
   hadoop.yarn.client.api.impl.TestYarnClient 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/134/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/134/artifact/out/diff-compile-javac-root.txt
  [172K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/134/artifact/out/diff-checkstyle-root.txt
  [16M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/134/artifact/out/diff-patch-pylint.txt
  [16K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/134/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/134/artifact/out/diff-patch-shelldocs.txt
  [16K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/134/artifact/out/whitespace-eol.txt
  [12M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/134/artifact/out/whitespace-tabs.txt
  [1.3M]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/134/artifact/out/diff-javadoc-javadoc-root.txt
  [2.2M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/134/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
  [24K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/134/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [36K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/134/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/134/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [268K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/134/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/134/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativetask.txt
  [124K]

   asflicense:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/134/artifact/out/patch-asflicense-problems.txt
  [4.0K]

Powered by Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org



-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-6755) Report correct spill/output size for map in nativetask

Re: [VOTE] Release Apache Hadoop 2.7.3 RC1

Re: [VOTE] Release Apache Hadoop 2.7.3 RC1

Re: [Release thread] 2.6.5 release activities

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

5 matches

Site Navigation

Mail list logo

Footer information