[jira] [Created] (HADOOP-13839) Fix outdated tracing documentation

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-13839:
-

 Summary: Fix outdated tracing documentation
 Key: HADOOP-13839
 URL: https://issues.apache.org/jira/browse/HADOOP-13839
 Project: Hadoop Common
  Issue Type: Bug
  Components: documentation, tracing
Affects Versions: 2.7.3
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Minor


Sample code in tracing doc is based on older version of SpanReceiverHost. The 
doc in branch-2 and trunk seems to be good.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.6.5 (RC1)

[Masatake Iwasaki]

+1(non-binding)

* verified signature and md5.
* built with -Pnative on CentOS6 and OpenJDK7.
* built documentation and skimmed the contents.
* built rpms by bigtop and ran smoke-tests of hdfs, yarn and mapreduce 
on 3-node cluster.


Thanks,
Masatake Iwasaki

On 10/3/16 09:12, Sangjin Lee wrote:

Hi folks,

I have pushed a new release candidate (R1) for the Apache Hadoop 2.6.5
release (the next maintenance release in the 2.6.x release line). RC1
contains fixes to CHANGES.txt, and is otherwise identical to RC0.

Below are the details of this release candidate:

The RC is available for validation at:
http://home.apache.org/~sjlee/hadoop-2.6.5-RC1/.

The RC tag in git is release-2.6.5-RC1 and its git commit is
e8c9fe0b4c252caf2ebf1464220599650f119997.

The maven artifacts are staged via repository.apache.org at:
https://repository.apache.org/content/repositories/orgapachehadoop-1050/.

You can find my public key at
http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS.

Please try the release and vote. The vote will run for the usual 5 days. I
would greatly appreciate your timely vote. Thanks!

Regards,
Sangjin




-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-13497) fix wrong command in CredentialProviderAPI.md

[Masatake Iwasaki (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki resolved HADOOP-13497.
---
   Resolution: Fixed
Fix Version/s: 3.0.0-alpha2
   2.8.0

Committed. Thanks for the contribution, [~yuanbo].

> fix wrong command in CredentialProviderAPI.md
> -
>
> Key: HADOOP-13497
> URL: https://issues.apache.org/jira/browse/HADOOP-13497
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: documentation
>Reporter: Yuanbo Liu
>Assignee: Yuanbo Liu
>Priority: Trivial
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HADOOP-13497.001.patch
>
>
> In CredentialProviderAPI.md line 122
> {quote}
> Example: `hadoop credential create ssl.server.keystore.password 
> jceks://file/tmp/test.jceks`
> {quote}
> should be
> {quote}
> Example: `hadoop credential create ssl.server.keystore.password -provider 
> jceks://file/tmp/test.jceks`
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13439) Fix race between TestMetricsSystemImpl and TestGangliaMetrics

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-13439:
-

 Summary: Fix race between TestMetricsSystemImpl and 
TestGangliaMetrics
 Key: HADOOP-13439
 URL: https://issues.apache.org/jira/browse/HADOOP-13439
 Project: Hadoop Common
  Issue Type: Bug
  Components: t, test
Reporter: Masatake Iwasaki
Priority: Minor


TestGangliaMetrics#testGangliaMetrics2 set *.period to 120 but 8 was used.
{noformat}
2016-06-27 15:21:31,480 INFO  impl.MetricsSystemImpl 
(MetricsSystemImpl.java:startTimer(375)) - Scheduled snapshot period at 8 
second(s).
{noformat}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.3 RC0

[Masatake Iwasaki]

Thanks for putting this up, Vinod.

+1 (non-binding)

* verified signature and mds of source and binary tarball
* built from source tarball on CentOS 6
* built site documentation
* deployed 3-node cluster with NN-HA and RM-HA, ran example jobs
* built rpms by using Bigtop, deployed 3-node cluster on docker and ran 
smoke-tests of hdfs, yarn and mapreduce


Masatake Iwasaki

On 7/23/16 11:15, Vinod Kumar Vavilapalli wrote:

Hi all,

I've created a release candidate RC0 for Apache Hadoop 2.7.3.

As discussed before, this is the next maintenance release to follow up 2.7.2.

The RC is available for validation at: 
http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ 
<http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>

The RC tag in git is: release-2.7.3-RC0

The maven artifacts are available via repository.apache.org 
<http://repository.apache.org/> at 
https://repository.apache.org/content/repositories/orgapachehadoop-1040/ 
<https://repository.apache.org/content/repositories/orgapachehadoop-1040/>

The release-notes are inside the tar-balls at location 
hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at 
http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/releasenotes.html 
<http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your 
quick perusal.

As you may have noted, a very long fix-cycle for the License & Notice issues 
(HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by 
quite a bit. This release's related discussion thread is linked below: [1].

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[1]: 2.7.3 release plan: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 
<http://markmail.org/thread/6yv2fyrs4jlepmmr>



-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Merge feature branch HADOOP-12930

[Masatake Iwasaki]

> Fix (with unit tests!) committed.

Thanks, Allen.

+1.

- checked out HADOOP-12930 and ran shelltest on my local
- ran 'mvn package -Pdist -Pnative -DskipTests'
- started pseudo-distributed cluster by start-dfs.sh and start-yarn.sh
- ran 'hadoop distcp' and 'mapred streaming' successfully
- created etc/hadoop/shellprofile.d/test.sh and added subcommands 
described in UnixShellGuide.md

- tested overriding built-in subcommands

Masatake Iwasaki

On 5/16/16 00:30, Allen Wittenauer wrote:

On May 15, 2016, at 7:27 AM, Allen Wittenauer <a...@apache.org> wrote:



On May 14, 2016, at 9:04 PM, Masatake Iwasaki <iwasak...@oss.nttdata.co.jp> 
wrote:

+  hadoop_debug "Calling dynamically: hadoop_subcommand_${HADOOP_SUBCMD} 
${HADOOP_SUBCMD_ARGS[$*]}”

Easy fix.  The $* should just be *.  I’ll open an issue and fix it here 
in a  sec.


Fix (with unit tests!) committed.

Thanks
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org




-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13152) Fix dynamic subcommands error on multiple arguments

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-13152:
-

 Summary: Fix dynamic subcommands error on multiple arguments
 Key: HADOOP-13152
 URL: https://issues.apache.org/jira/browse/HADOOP-13152
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: HADOOP-12930
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Merge feature branch HADOOP-12930

[Masatake Iwasaki]

Hi Allen,

I had a problem when I was testing HADOOP-12930 branch. (built by mvn 
package -Pdist)


Single argument is ok.::

  [iwasakims@centos7 HADOOP-12930]$ bin/hadoop hello foo
  foo


Error on multiple arguments.  ::

  [iwasakims@centos7 HADOOP-12930]$ bin/hadoop hello foo bar
  bin/hadoop: line 200: foo bar: syntax error in expression (error 
token is "bar")

  Error: Could not find or load main class

  [iwasakims@centos7 HADOOP-12930]$ bin/hadoop distcp /foo /bar
  bin/hadoop: line 200: /foo /bar: syntax error: operand expected 
(error token is "/foo /bar")

  Error: Could not find or load main class


The fix of HADOOP-13120 seems to be related.::

  diff --git a/hadoop-common-project/hadoop-common/src/main/bin/hadoop 
b/hadoop-common-project/hadoop-common/src/main/bin/hadoop

  index 61fdc2e..0f6982b 100755
  --- a/hadoop-common-project/hadoop-common/src/main/bin/hadoop
  +++ b/hadoop-common-project/hadoop-common/src/main/bin/hadoop
  @@ -197,6 +197,7 @@ shift
   HADOOP_SUBCMD_ARGS=("$@")

   if declare -f hadoop_subcommand_"${HADOOP_SUBCMD}" >/dev/null 2>&1; then
  +  hadoop_debug "Calling dynamically: 
hadoop_subcommand_${HADOOP_SUBCMD} ${HADOOP_SUBCMD_ARGS[$*]}"

 "hadoop_subcommand_${HADOOP_SUBCMD}" "${HADOOP_SUBCMD_ARGS[@]}"
   else
 hadoopcmd_case "${HADOOP_SUBCMD}" "${HADOOP_SUBCMD_ARGS[@]}"


Thanks,
Masatake Iwasaki


On 5/15/16 02:33, Allen Wittenauer wrote:

This vote closes in 2 days and the only response has been from a non-committer 
and one of the 137 other committers on the project…. it’d be great if some 
others could take a look.

Thanks!


On May 12, 2016, at 6:07 PM, Andrew Wang <andrew.w...@cloudera.com> wrote:

+1. I looked at the patches on the branch, wasn't too bad to review. As
Allen said, there's some code movement, assorted other nice doc and shell
fixups.

Found one extra typo, which I added to HADOOP-13129.

Best,
Andrew

On Wed, May 11, 2016 at 1:14 AM, Sean Busbey <bus...@cloudera.com> wrote:


+1 (non-binding)

reviewed everything, filed an additional subtask for a very trivial
typo in the docs. should be fine to make a full issue after close and
then fix.

tried merging locally, tried running through new shell tests (both
with and without bats installed), tried making an example custom
command (valid and malformed). everything looks great.

On Mon, May 9, 2016 at 1:26 PM, Allen Wittenauer <a...@apache.org> wrote:

Hey gang!

I’d like to call a vote to run for 7 days (ending May 16 at

13:30 PT) to merge the HADOOP-12930 feature branch into trunk. This branch
was developed exclusively by me as per the discussion two months ago as a
way to make what would be a rather large patch hopefully easier to review.
The vast majority of the branch is code movement in the same file,
additional license headers, maven assembly hooks for distribution, and
variable renames. Not a whole lot of new code, but a big diff file
none-the-less.

This branch modifies the ‘hadoop’, ‘hdfs’, ‘mapred’, and ‘yarn’

commands to allow for subcommands to be added or modified at runtime.  This
allows for individual users or entire sites to tweak the execution
environment to suit their local needs.  For example, it has been a practice
for some locations to change the distcp jar out for a custom one.  Using
this functionality, it is possible that the ‘hadoop distcp’ command could
run the local version without overwriting the bundled jar and for existing
documentation (read: results from Internet searches) to work as written
without modification. This has the potential to be a huge win, especially
for:

* advanced end users looking to supplement the Apache

Hadoop experience

* operations teams that may be able to leverage existing

documentation without having to remain local “exception” docs

* development groups wanting an easy way to trial

experimental features

Additionally, this branch includes the following, related

changes:

* Adds the first unit tests for the ‘hadoop’ command
* Adds the infrastructure for hdfs script testing and

the first unit test for the ‘hdfs’ command

* Modifies the hadoop-tools components to be dynamic

rather than hard coded

* Renames the shell profiles for hdfs, mapred, and yarn

to be consistent with other bundled profiles, including the ones introduced
in this branch

Documentation, including a ‘hello world’-style example, is in

the UnixShellGuide markdown file.  (Of course!)

 I am at ApacheCon this week if anyone wants to discuss in-depth.

Thanks!

P.S.,

There are still two open sub-tasks.  These are blocked by other

issues so that we may add unit testing to the shell code in those
respective areas.  I’ll covert to full issues after HADOOP-12930 is clo

Re: [DISCUSS] Set minimum version of Hadoop 3 to JDK8 (HADOOP-11858)

[Masatake Iwasaki]

+1

Masatake

On 5/12/16 13:11, Gangumalla, Uma wrote:

+1

Regards,
Uma

On 5/10/16, 2:24 PM, "Andrew Wang"  wrote:


+1

On Tue, May 10, 2016 at 12:36 PM, Ravi Prakash 
wrote:


+1. Thanks for driving this Akira

On Tue, May 10, 2016 at 10:25 AM, Tsuyoshi Ozawa 
wrote:


Before cutting 3.0.0-alpha RC, I'd like to drop JDK7 support in

trunk.

Sounds good. To do so, we need to check the blockers of 3.0.0-alpha
RC, especially upgrading all dependencies which use refractions at
first.

Thanks,
- Tsuyoshi

On Tue, May 10, 2016 at 8:32 AM, Akira AJISAKA
 wrote:

Hi developers,

Before cutting 3.0.0-alpha RC, I'd like to drop JDK7 support in

trunk.

Given this is a critical change, I'm thinking we should get the

consensus

first.

One concern I think is, when the minimum version is set to JDK8, we

need

to

configure Jenkins to disable multi JDK test only in trunk.

Any thoughts?

Thanks,
Akira



-

To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org




-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org




-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Replacing Commons-httpclient and bumping httpclient version

[Masatake Iwasaki]

Thanks for the suggestion, Wei-Chiu Chuang.

I'm +1 too to clean up dependency on commons-httpclient.

Your suggestion reminded me of HADOOP-12552 which
seems to depends on HADOOP-12710 and HADOOP-12711 now.
I will revisit it.

Masatake Iwasaki

On 2/17/16 03:59, Colin P. McCabe wrote:

+1 for updating the dependencies in trunk.

best,
Colin

On Tue, Feb 16, 2016 at 9:20 AM, Wei-Chiu Chuang <weic...@cloudera.com> wrote:

Fellow Hadoop developers,

Hadoop codebase depends on commons-httpclient, and its latest version, 3.1.2, 
is EOL nearly 5 years ago. But because its API is not compatible with its 
successor, httpclient 4, the community seem to have been reluctant to upgrade.
However, a lot of evidence indicates that commons-httpclient has a number of 
security vulnerabilities which are never addressed, including CVE-2012-6153. To 
make Hadoop less susceptible to existing and future vulnerabilities, we should 
seriously consider replacing commons-httpclient with httpclient 4.x.

There are a few Hadoop JIRAs that have patches available to address that, but 
they really need more attention to get them committed:
HADOOP-10105 <https://issues.apache.org/jira/browse/HADOOP-10105> (remove 
httpclient dependency) is the umbrella JIRA for all.
Other efforts includes HADOOP-11613 <https://issues.apache.org/jira/browse/HADOOP-11613> (Remove 
httpclient dependency from hadoop-azure), HADOOP-11614 
<https://issues.apache.org/jira/browse/HADOOP-11614> (Remove httpclient dependency from 
hadoop-openstack), HADOOP-12710 <https://issues.apache.org/jira/browse/HADOOP-12710> (Remove 
dependency on commons-httpclient for TestHttpServerLogs), HADOOP-12711 
<https://issues.apache.org/jira/browse/HADOOP-12711> (Remove dependency on commons-httpclient for 
ServletUtil). I’d also like to urge the community to reject patches that imports commons-httpclient in 
the future.

Additionally, Hadoop trunk depends on httpclient 4.2.5, which is known to suffer from 
several security vulnerabilities as well, including CVE-2012-6153, CVE-2011-4461, 
CVE-2014-3577, CVE-2015-5262. HADOOP-12767 
<https://issues.apache.org/jira/browse/HADOOP-12767> (update apache httpclient 
version to the latest 4.5 for security) has a patch that bumps the version to 4.5.1. 
But I’d like to ask the community whether we should do it or not, and the implication 
of bump the latest version.

Best regards,
Wei-Chiu Chuang
A very happy Clouderan

Re: [VOTE] Release Apache Hadoop 2.7.2 RC2

[Masatake Iwasaki]

+1(non-binding)

- verified mds and signature of source and binary tarball
- built from source tarball with -Pnative on CentOS 7/OpenJDK 7
- built rpm by using bigtop, launched 3 nodes cluster, ran smoke test of 
hdfs, mapreduce, yarn
- built rpm of hbase, hive, pig against 2.7.2-RC2, ran smoke test of 
hive, pig


Thanks,
Masatake Iwasaki


On 1/15/16 13:57, Vinod Kumar Vavilapalli wrote:

Hi all,

I've created an updated release candidate RC2 for Apache Hadoop 2.7.2.

As discussed before, this is the next maintenance release to follow up 2.7.1.

The RC is available for validation at: 
http://people.apache.org/~vinodkv/hadoop-2.7.2-RC2/

The RC tag in git is: release-2.7.2-RC2

The maven artifacts are available via repository.apache.org 
<http://repository.apache.org/> at 
https://repository.apache.org/content/repositories/orgapachehadoop-1027 
<https://repository.apache.org/content/repositories/orgapachehadoop-1027>

The release-notes are inside the tar-balls at location 
hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at 
http://people.apache.org/~vinodkv/hadoop-2.7.2-RC2/releasenotes.html 
<http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your 
quick perusal.

As you may have noted,
  - I terminated the RC1 related voting thread after finding out that we didn’t 
have a bunch of patches that are already in the released 2.6.3 version. After a 
brief discussion, we decided to keep the parallel 2.6.x and 2.7.x releases 
incremental, see [4] for this discussion.
  - The RC0 related voting thread got halted due to some critical issues. It 
took a while again for getting all those blockers out of the way. See the 
previous voting thread [3] for details.
  - Before RC0, an unusually long 2.6.3 release caused 2.7.2 to slip by quite a 
bit. This release's related discussion threads are linked below: [1] and [2].

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[1]: 2.7.2 release plan: http://markmail.org/message/oozq3gvd4nhzsaes 
<http://markmail.org/message/oozq3gvd4nhzsaes>
[2]: Planning Apache Hadoop 2.7.2 http://markmail.org/message/iktqss2qdeykgpqk 
<http://markmail.org/message/iktqss2qdeykgpqk>
[3]: [VOTE] Release Apache Hadoop 2.7.2 RC0: 
http://markmail.org/message/5txhvr2qdiqglrwc 
<http://markmail.org/message/5txhvr2qdiqglrwc>
[4] Retracted [VOTE] Release Apache Hadoop 2.7.2 RC1: 
http://markmail.org/thread/n7ljbsnquihn3wlw

[jira] [Reopened] (HADOOP-12588) Fix intermittent test failure of TestGangliaMetrics

[Masatake Iwasaki (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki reopened HADOOP-12588:
---

I saw this test failure again today. The fix seemed to be not enough.
{noformat}
java.lang.AssertionError: Missing metrics: test.s1rec.Xxx
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at 
org.apache.hadoop.metrics2.impl.TestGangliaMetrics.checkMetrics(TestGangliaMetrics.java:161)
at 
org.apache.hadoop.metrics2.impl.TestGangliaMetrics.testGangliaMetrics2(TestGangliaMetrics.java:139)
{noformat}


> Fix intermittent test failure of TestGangliaMetrics
> ---
>
> Key: HADOOP-12588
> URL: https://issues.apache.org/jira/browse/HADOOP-12588
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Tsuyoshi Ozawa
>        Assignee: Masatake Iwasaki
> Fix For: 2.8.0, 2.7.3
>
> Attachments: HADOOP-12588.001.patch
>
>
> Jenkins found this test failure on HADOOP-11149.
> {quote}
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.773 sec <<< 
> FAILURE! - in org.apache.hadoop.metrics2.impl.TestGangliaMetrics
> testGangliaMetrics2(org.apache.hadoop.metrics2.impl.TestGangliaMetrics)  Time 
> elapsed: 0.39 sec  <<< FAILURE!
> java.lang.AssertionError: Missing metrics: test.s1rec.Xxx
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.assertTrue(Assert.java:41)
>   at 
> org.apache.hadoop.metrics2.impl.TestGangliaMetrics.checkMetrics(TestGangliaMetrics.java:159)
>   at 
> org.apache.hadoop.metrics2.impl.TestGangliaMetrics.testGangliaMetrics2(TestGangliaMetrics.java:137)
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-12675) Fix description about retention period in usage of expunge command

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-12675:
-

 Summary: Fix description about retention period in usage of 
expunge command
 Key: HADOOP-12675
 URL: https://issues.apache.org/jira/browse/HADOOP-12675
 Project: Hadoop Common
  Issue Type: Bug
  Components: documentation
Affects Versions: 2.8.0
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-12672) Split RPC timeout from IPC ping

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-12672:
-

 Summary: Split RPC timeout from IPC ping
 Key: HADOOP-12672
 URL: https://issues.apache.org/jira/browse/HADOOP-12672
 Project: Hadoop Common
  Issue Type: Improvement
  Components: ipc
Reporter: Masatake Iwasaki






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-12661) Add trash APIs to the FileSystem specification

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-12661:
-

 Summary: Add trash APIs to the FileSystem specification
 Key: HADOOP-12661
 URL: https://issues.apache.org/jira/browse/HADOOP-12661
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: documentation
Affects Versions: 2.8.0
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki


HDFS-8831 added getTrashRoot and getTrashRoots.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-12627) Add user documentation about metrics system

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-12627:
-

 Summary: Add user documentation about metrics system
 Key: HADOOP-12627
 URL: https://issues.apache.org/jira/browse/HADOOP-12627
 Project: Hadoop Common
  Issue Type: Improvement
  Components: documentation, metrics
Affects Versions: 2.7.1
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki


Though there is [reference of metric 
items|https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/Metrics.html]
 and 
[Javadoc|https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-project/api/org/apache/hadoop/metrics2/package-summary.html],
 we have no user documentation to configure metrics system.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-12609) Fix intermittent failure of TestDecayRpcScheduler

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-12609:
-

 Summary: Fix intermittent failure of TestDecayRpcScheduler
 Key: HADOOP-12609
 URL: https://issues.apache.org/jira/browse/HADOOP-12609
 Project: Hadoop Common
  Issue Type: Bug
  Components: test
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (HADOOP-12605) Fix intermittent failure of TestIPC.testIpcWithReaderQueuing

[Masatake Iwasaki (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki reopened HADOOP-12605:
---

I rethought about this. It would be better to do follow-up work here because 
the patch of HADOOP-10406 was already committed. The patch was improvement but 
just turned out to be not enough.

> Fix intermittent failure of TestIPC.testIpcWithReaderQueuing
> 
>
> Key: HADOOP-12605
> URL: https://issues.apache.org/jira/browse/HADOOP-12605
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: test
>        Reporter: Masatake Iwasaki
>    Assignee: Masatake Iwasaki
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-12605) Fix intermittent failure of TestIPC.testIpcWithReaderQueuing

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-12605:
-

 Summary: Fix intermittent failure of 
TestIPC.testIpcWithReaderQueuing
 Key: HADOOP-12605
 URL: https://issues.apache.org/jira/browse/HADOOP-12605
 Project: Hadoop Common
  Issue Type: Bug
  Components: test
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-12561) Add documentation about metrics system

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-12561:
-

 Summary: Add documentation about metrics system
 Key: HADOOP-12561
 URL: https://issues.apache.org/jira/browse/HADOOP-12561
 Project: Hadoop Common
  Issue Type: Improvement
  Components: documentation
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki


Though there is [reference of metric 
items|https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/Metrics.html],
 we have no documentation about metrics system and configuration. Adding 
concise documentation about metrics system would help users, admins and 
developers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-12552) Fix undeclared/unused dependency to httpclient

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-12552:
-

 Summary: Fix undeclared/unused dependency to httpclient
 Key: HADOOP-12552
 URL: https://issues.apache.org/jira/browse/HADOOP-12552
 Project: Hadoop Common
  Issue Type: Bug
  Components: build
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Minor


hadoop-common uses httpclient as undeclared dependency and have unused 
dependency to commons-httpclient. Vise versa in hadoop-azure and 
hadoop-openstack.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-12539) GraphiteSink should have default value for server_host and server_port

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-12539:
-

 Summary: GraphiteSink should have default value for server_host 
and server_port
 Key: HADOOP-12539
 URL: https://issues.apache.org/jira/browse/HADOOP-12539
 Project: Hadoop Common
  Issue Type: Improvement
  Components: metrics
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Minor


GraphiteSink throws exception if {{server_port}} is not set in metrics 
configuration.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Release Apache Hadoop 2.6.2

[Masatake Iwasaki]

+1(non-binding)

- verified checksums and signature of source and binary tarball
- built from source tarball on CentOS 6.4 and OpenJDK 7
- built site documentation
- launched 4-nodes cluster and ran example jobs

Thanks,
Masatake Iwasaki


On 10/23/15 06:14, Sangjin Lee wrote:

Hi all,

I have created a release candidate (RC0) for Hadoop 2.6.2.

The RC is available at: http://people.apache.org/~sjlee/hadoop-2.6.2-RC0/

The RC tag in git is: release-2.6.2-RC0

The list of JIRAs committed for 2.6.2:
https://issues.apache.org/jira/browse/YARN-4101?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20AND%20fixVersion%20%3D%202.6.2

The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1022/

Please try out the release candidate and vote. The vote will run for 5 days.

Thanks,
Sangjin

[jira] [Created] (HADOOP-12511) Add support for InfluxDB as metrics sink

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-12511:
-

 Summary: Add support for InfluxDB as metrics sink
 Key: HADOOP-12511
 URL: https://issues.apache.org/jira/browse/HADOOP-12511
 Project: Hadoop Common
  Issue Type: New Feature
  Components: metrics
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-12477) Add test to make sure hadoop-minicluster provides enough dependencies

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-12477:
-

 Summary: Add test to make sure hadoop-minicluster provides enough 
dependencies
 Key: HADOOP-12477
 URL: https://issues.apache.org/jira/browse/HADOOP-12477
 Project: Hadoop Common
  Issue Type: Improvement
  Components: test
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki


Dependencies of test-jar is not transitive by design (MNG-1378). If mini 
cluster depends on artifacts depended by only test-jars, downstream projects 
using mini cluster need to add the dependencies in addition to 
hadoop-minicluster. It would be kind for downstream projects to make sure 
hadoop-minicluster provides enough dependencies.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-12470) In-page TOC of documentation should be automatically generated by doxia macro

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-12470:
-

 Summary: In-page TOC of documentation should be automatically 
generated by doxia macro
 Key: HADOOP-12470
 URL: https://issues.apache.org/jira/browse/HADOOP-12470
 Project: Hadoop Common
  Issue Type: Improvement
  Components: documentation
Affects Versions: 2.7.1
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki


In-page TOC of each documentation page is maintained by hand now. It should be 
automatically generated once doxia macro is supported by doxia-module-markdown.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] About the details of JDK-8 support

[Masatake Iwasaki]

Thanks for clear summary, Tsuyoshi.

I read some related past discussions.

  https://wiki.apache.org/hadoop/MovingToJdk7and8
  http://search-hadoop.com/m/uOzYtGSiCs1acRnh
  http://search-hadoop.com/m/uOzYthdWJqpGdSZ1

Though there seems to be no consensus about when to drop java 7 support yet,
it would not be 2.8 for which the preparation is already started.
If the works for making source compatible with java 8 does not result in
dropping java 7 support, it would be nice and easy to backport to branch-2.


> we need to upgrade grizzly to 2.2.16 to use
> jersey-test-framework-grizzly2. I’d like to discuss which version we
> will target this change. Can we do this in branch-2?

At lease, the newest grizzly, jersey and asm seems to support java 7 too
and HADOOP-11993 may work in branch-2.


Masatake Iwasaki


On 10/6/15 09:35, Tsuyoshi Ozawa wrote:
> Hi commiters and users of Hadoop stack,
>
> I’ll share the current status of JDK-8 support here. We can take a
> two-step approach to support JDK-8 - runtime-level support and
> source-level support.
>
> About runtime-level support, I’ve tested Hadoop stack with JDK-8  e.g.
> MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
> long as I tested, it works well completely since JDK-8 doesn’t have
> any incompatibility at binary level. We can say Hadoop has supported
> JDK8 runtime already. Do you have any concern about this? I’ve not
> tested with HBase yet. I need help of HBase community. I think only
> problem about runtime is HADOOP-11364, default value of
> colntainer-killer of YARN. After fixing the issue, we can declare the
> support of JDK on Hadoop Wiki to make it clear for users.
> https://wiki.apache.org/hadoop/HadoopJavaVersions
>
> About source-level, however, we have one big problem - upgrading
> dependency of asm and cglib. We need to upgrade all libraries which
> depends on asm to support new byte code introduced in JDK8[1]. The
> dependencies which uses asm are jersey-server for compile and provide
> scope, and cglib for test scope(I checked it with mvn dependency:tree
> command). HADOOP-9613 is addressing the problem.
>
> One complex problem I’ve faced is Jersey depends on grizzly - to
> upgrade jersey to 1.19, which supports JDK8,
>  we need to upgrade grizzly to 2.2.16 to use
> jersey-test-framework-grizzly2. I’d like to discuss which version we
> will target this change. Can we do this in branch-2? Should we take
> care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
> confirm whether HADOOP-11993 means to remove Jersey, which depends on
> asm, or not. I think we can collaborate with Yetus community here.
>
> Also, another simple problem is that source code cannot be compiled
> because javadoc format or variable identifier are illegal(e.g.
> HADOOP-12457, HADOOP-11875). I think this can be solved
> straightforwardly.
>
> Please share any concern I’ve missed. The opinions of users are also 
welcome :-)

> I'd like to go forward this step by step to make Hadoop user friendly.
>
> Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
> Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
> JDK-8.
>
> Best regards,
> - Tsuyoshi
>
> [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
> [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker

[jira] [Created] (HADOOP-12452) Fix tracing documention reflecting the update to htrace-4

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-12452:
-

 Summary: Fix tracing documention reflecting the update to htrace-4
 Key: HADOOP-12452
 URL: https://issues.apache.org/jira/browse/HADOOP-12452
 Project: Hadoop Common
  Issue Type: Bug
  Components: tracing
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki


The sample code in Tracing.md have some problems.
* compilation error by not importing Tracer.
* generic options are not reflected because Tracer is initialized before 
ToolRunner#run.
* it may be confusing to use FsShell in example because it has embedded Tracer 
now.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Release Apache Hadoop 2.6.1 RC1

[Masatake Iwasaki]

+1(non-binding)

- verified mds and signature of source and binary tarball
- built from source tarball with -Pnative on CentOS 7 and OpenJDK 7
- built site documentation
- started pseudo distributed cluster and ran example jobs

Masatake Iwasaki


On 9/17/15 11:10, Vinod Kumar Vavilapalli wrote:

Hi all,

After a nearly month long [1] toil, with loads of help from Sangjin Lee and
Akira Ajisaka, and 153 (RC0)+7(RC1) commits later, I've created a release
candidate RC1 for hadoop-2.6.1.

RC1 is RC0 [0] (for which I opened and closed a vote last week) + UI fixes
for the issue Sangjin raised (YARN-3171 and the dependencies YARN-3779,
YARN-3248), additional fix to avoid incompatibility (YARN-3740), other UI
bugs (YARN-1884, YARN-3544) and the MiniYARNCluster issue (right patch for
YARN-2890) that Jeff Zhang raised.

The RC is available at: http://people.apache.org/~vinodkv/hadoop-2.6.1-RC1/

The RC tag in git is: release-2.6.1-RC1

The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/orgapachehadoop-1021

Some notes from our release process
  -  - Sangjin and I moved out a bunch of items pending from 2.6.1 [2] -
non-committed but desired patches. 2.6.1 is already big as is and is late
by any standard, we can definitely include them in the next release.
  - The 2.6.1 wiki page [3] captures some (but not all) of the context of
the patches that we pushed in.
  - Given the number of fixes pushed [4] in, we had to make a bunch of
changes to our original plan - we added a few improvements that helped us
backport patches easier (or in many cases made backports possible), and we
dropped a few that didn't make sense (HDFS-7831, HDFS-7926, HDFS-7676,
HDFS-7611, HDFS-7843, HDFS-8850).
  - I ran all the unit tests which (surprisingly?) passed. (Except for one,
which pointed out a missing fix HDFS-7552).

As discussed before [5]
  - This release is the first point release after 2.6.0
  - I’d like to use this as a starting release for 2.6.2 in a few weeks and
then follow up with more of these.

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[0] Hadoop 2.6.1 RC0 vote: http://markmail.org/thread/ubut2rn3lodc55iy
[1] Hadoop 2.6.1 Release process thread:
http://markmail.org/thread/wkbgkxkhntx5tlux
[2] 2.6.1 Pending tickets:
https://issues.apache.org/jira/issues/?filter=12331711
[3] 2.6.1 Wiki page: https://wiki.apache.org/hadoop/Release-2.6.1
-Working-Notes
[4] List of 2.6.1 patches pushed:
https://issues.apache.org/jira/issues/?jql=fixVersion%20%3D%202.6.1
%20and%20labels%20%3D%20%222.6.1-candidate%22
[5] Planning Hadoop 2.6.1 release:
http://markmail.org/thread/sbykjn5xgnksh6wg

PS:
  - Note that branch-2.6 which will be the base for 2.6.2 doesn't have these
fixes yet. Once 2.6.1 goes through, I plan to rebase branch-2.6 based off
2.6.1.
  - The additional patches in RC1 that got into 2.6.1 all the way from 2.8
are NOT in 2.7.2 yet, this will be done as a followup.

[jira] [Created] (HADOOP-12280) Skip unit tests based on maven profile rather than NativeCodeLoader.isNativeCodeLoaded

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-12280:
-

 Summary: Skip unit tests based on maven profile rather than 
NativeCodeLoader.isNativeCodeLoaded
 Key: HADOOP-12280
 URL: https://issues.apache.org/jira/browse/HADOOP-12280
 Project: Hadoop Common
  Issue Type: Improvement
  Components: test
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-12241) Add specification of FileStatus

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-12241:
-

 Summary: Add specification of FileStatus
 Key: HADOOP-12241
 URL: https://issues.apache.org/jira/browse/HADOOP-12241
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-12242) Add TOC to filesystem.md

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-12242:
-

 Summary: Add TOC to filesystem.md
 Key: HADOOP-12242
 URL: https://issues.apache.org/jira/browse/HADOOP-12242
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-12240) Fix tests requiring native library to be skipped in non-native profile

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-12240:
-

 Summary: Fix tests requiring native library to be skipped in 
non-native profile
 Key: HADOOP-12240
 URL: https://issues.apache.org/jira/browse/HADOOP-12240
 Project: Hadoop Common
  Issue Type: Bug
  Components: test
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-12229) Fix inconsistent subsection titles in filesystem.md

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-12229:
-

 Summary: Fix inconsistent subsection titles in filesystem.md
 Key: HADOOP-12229
 URL: https://issues.apache.org/jira/browse/HADOOP-12229
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: documentation
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Minor


* Some API signatures does not have return value.
* Some API signatures have {{FileSystem.}} prefix.
* Some subsection has wrong level.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-12200) TestCryptoStreamsWithOpensslAesCtrCryptoCodec should be skipped in non-native profile

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-12200:
-

 Summary: TestCryptoStreamsWithOpensslAesCtrCryptoCodec should be 
skipped in non-native profile
 Key: HADOOP-12200
 URL: https://issues.apache.org/jira/browse/HADOOP-12200
 Project: Hadoop Common
  Issue Type: Bug
  Components: test
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-12181) TestZKSignerSecretProvider intermittently fails

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-12181:
-

 Summary: TestZKSignerSecretProvider intermittently fails
 Key: HADOOP-12181
 URL: https://issues.apache.org/jira/browse/HADOOP-12181
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-12183) Add annotation to tracing span of FsShell to record args

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-12183:
-

 Summary: Add annotation to tracing span of FsShell to record args
 Key: HADOOP-12183
 URL: https://issues.apache.org/jira/browse/HADOOP-12183
 Project: Hadoop Common
  Issue Type: Improvement
  Components: tracing
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-12175) FsShell must load SpanReceierHost to support tracing

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-12175:
-

 Summary: FsShell must load SpanReceierHost to support tracing
 Key: HADOOP-12175
 URL: https://issues.apache.org/jira/browse/HADOOP-12175
 Project: Hadoop Common
  Issue Type: Bug
  Components: tracing
Affects Versions: 2.8.0
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki


FsShell need to load SpanReceierHost to support tracing in addition to 
HADOOP-12124.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-12177) [Umbrella] Update and extend filesystem specification

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-12177:
-

 Summary: [Umbrella] Update and extend filesystem specification
 Key: HADOOP-12177
 URL: https://issues.apache.org/jira/browse/HADOOP-12177
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Masatake Iwasaki






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Release Apache Hadoop 2.7.1 RC0

[Masatake Iwasaki]

+1 (non-binding)

+ verified mds of source and binary tarball
+ built from source tarball
+ deployed binary tarball to 4 nodes cluster and run some 
hadoop-mapreduce-examples jobs


Thanks,
Masatake Iwasaki


On 6/29/15 17:45, Vinod Kumar Vavilapalli wrote:

Hi all,

I've created a release candidate RC0 for Apache Hadoop 2.7.1.

As discussed before, this is the next stable release to follow up 2.6.0,
and the first stable one in the 2.7.x line.

The RC is available for validation at:
*http://people.apache.org/~vinodkv/hadoop-2.7.1-RC0/
http://people.apache.org/~vinodkv/hadoop-2.7.1-RC0/*

The RC tag in git is: release-2.7.1-RC0

The maven artifacts are available via repository.apache.org at
*https://repository.apache.org/content/repositories/orgapachehadoop-1019/
https://repository.apache.org/content/repositories/orgapachehadoop-1019/*

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

PS: It took 2 months instead of the planned [1] 2 weeks in getting this
release out: post-mortem in a separate thread.

[1]: A 2.7.1 release to follow up 2.7.0
http://markmail.org/thread/zwzze6cqqgwq4rmw

[jira] [Created] (HADOOP-12167) Add usage of tracing from FsShell to documentation

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-12167:
-

 Summary: Add usage of tracing from FsShell to documentation
 Key: HADOOP-12167
 URL: https://issues.apache.org/jira/browse/HADOOP-12167
 Project: Hadoop Common
  Issue Type: Improvement
  Components: documentation
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-12037) Fix wrong classname in example configuration of hadoop-auth documentation

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-12037:
-

 Summary: Fix wrong classname in example configuration of 
hadoop-auth documentation
 Key: HADOOP-12037
 URL: https://issues.apache.org/jira/browse/HADOOP-12037
 Project: Hadoop Common
  Issue Type: Bug
  Components: documentation
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Trivial






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11971) Move test utilities for tracing from hadoop-hdfs to hadoo-common

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-11971:
-

 Summary: Move test utilities for tracing from hadoop-hdfs to 
hadoo-common
 Key: HADOOP-11971
 URL: https://issues.apache.org/jira/browse/HADOOP-11971
 Project: Hadoop Common
  Issue Type: Improvement
  Components: tracing
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Minor


Utilities used in TestTracing such as SetSpanReceiver should be moved to 
{{hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/tracing/}}
 in order to make it usable from yarn and other modules.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11967) SpanReceiverHost should be able to handle tracing configuration properties without prefix

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-11967:
-

 Summary: SpanReceiverHost should be able to handle tracing 
configuration properties without prefix
 Key: HADOOP-11967
 URL: https://issues.apache.org/jira/browse/HADOOP-11967
 Project: Hadoop Common
  Issue Type: Improvement
  Components: tracing
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11912) Extra configuration key used in TraceUtils should respect prefix

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-11912:
-

 Summary: Extra configuration key used in TraceUtils should respect 
prefix
 Key: HADOOP-11912
 URL: https://issues.apache.org/jira/browse/HADOOP-11912
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Minor


HDFS-8213 added prefix handling to configuration used by tracing but extra key 
value pairs in configuration returned by TraceUtils#wrapHadoopConf does not 
respect this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11894) Bump the version of HTrace to 3.2.0-incubating

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-11894:
-

 Summary: Bump the version of HTrace to 3.2.0-incubating
 Key: HADOOP-11894
 URL: https://issues.apache.org/jira/browse/HADOOP-11894
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki


* update pom.xml
* update documentation
* replace {{addKVAnnotation(byte[] key, byte[] value)}} with 
{{addKVAnnotation(String key, String value)}}
* replace {{SpanReceiverHost#getUniqueLocalTraceFileName}} with 
{{LocalFileSpanReceiver#getUniqueLocalTraceFileName}}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Release Apache Hadoop 2.7.0 RC0

[Masatake Iwasaki]

+1 (non-binding)

+ verified signature and mds of source and binary tarball
+ built from source tarball
+ deployed binary tarball to 3 nodes cluster and run some 
hadoop-mapreduce-examples jobs


Thanks,
Masatake Iwasaki

On 4/11/15 08:44, Vinod Kumar Vavilapalli wrote:

Hi all,

I've created a release candidate RC0 for Apache Hadoop 2.7.0.

  The RC is available at: http://people.apache.org/~vinodkv/hadoop-2.7.0-RC0/

The RC tag in git is: release-2.7.0-RC0

  The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/orgapachehadoop-1017/

As discussed before
  - This release will only work with JDK 1.7 and above
  - I’d like to use this as a starting release for 2.7.x [1], depending on
how it goes, get it stabilized and potentially use a 2.7.1 in a few
weeks as the stable release.

  Please try the release and vote; the vote will run for the usual 5 days.

  Thanks,
  Vinod

  [1]: A 2.7.1 release to follow up 2.7.0
http://markmail.org/thread/zwzze6cqqgwq4rmw

[jira] [Created] (HADOOP-11729) Fix link to cgroups doc in site.xml

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-11729:
-

 Summary: Fix link to cgroups doc in site.xml
 Key: HADOOP-11729
 URL: https://issues.apache.org/jira/browse/HADOOP-11729
 Project: Hadoop Common
  Issue Type: Bug
  Components: documentation
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Minor


s/NodeManagerCGroups/NodeManagerCgroups/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11663) Remove description about Java 6 from docs

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-11663:
-

 Summary: Remove description about Java 6 from docs
 Key: HADOOP-11663
 URL: https://issues.apache.org/jira/browse/HADOOP-11663
 Project: Hadoop Common
  Issue Type: Bug
  Components: documentation
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Minor


{{hadoop-auth/BuildingIt.md}} has:
{noformat}
Hadoop Auth, Java HTTP SPNEGO - Building It
Requirements
Java 6+
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11594) Improve the readability of site index of documentation

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-11594:
-

 Summary: Improve the readability of site index of documentation
 Key: HADOOP-11594
 URL: https://issues.apache.org/jira/browse/HADOOP-11594
 Project: Hadoop Common
  Issue Type: Improvement
  Components: documentation
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Minor


* change the order of items
* make redundant title shorter and fit it in single line as far as possible




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11559) Add links to RackAwareness and InterfaceClassification to site index

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-11559:
-

 Summary: Add links to RackAwareness and InterfaceClassification to 
site index
 Key: HADOOP-11559
 URL: https://issues.apache.org/jira/browse/HADOOP-11559
 Project: Hadoop Common
  Issue Type: Bug
  Components: documentation
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Minor


RackAwareness.html and InterfaceClassification.html are not linked from site 
index. Add link to them to site.xml if the contents are not outdated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11558) Fix dead links to doc of hadoop-tools

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-11558:
-

 Summary: Fix dead links to doc of hadoop-tools
 Key: HADOOP-11558
 URL: https://issues.apache.org/jira/browse/HADOOP-11558
 Project: Hadoop Common
  Issue Type: Bug
  Components: documentation
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Minor


Some links to dosc of hadoop-tools are dead by HADOOP-10976.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11544) Remove unused configuration keys for tracing

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-11544:
-

 Summary: Remove unused configuration keys for tracing
 Key: HADOOP-11544
 URL: https://issues.apache.org/jira/browse/HADOOP-11544
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Trivial


CommonConfigurationKeys.HADOOP_TRACE_SAMPLER* are no longer used.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11529) Fix findbugs warnings in hadoop-archives

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-11529:
-

 Summary: Fix findbugs warnings in hadoop-archives
 Key: HADOOP-11529
 URL: https://issues.apache.org/jira/browse/HADOOP-11529
 Project: Hadoop Common
  Issue Type: Bug
  Components: tools
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Minor


{code}
NP  Possible null pointer dereference of reader in 
org.apache.hadoop.tools.HadoopArchives$HArchiveInputFormat.getSplits(JobConf, 
int) on exception path
Dm  Found reliance on default encoding in 
org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.close(): 
String.getBytes()
Dm  Found reliance on default encoding in 
org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.configure(JobConf): 
String.getBytes()
Dm  Found reliance on default encoding in 
org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.reduce(IntWritable, 
Iterator, OutputCollector, Reporter): String.getBytes()
{code}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11498) Bump the version of HTrace to 3.1.0-incubating

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-11498:
-

 Summary: Bump the version of HTrace to 3.1.0-incubating
 Key: HADOOP-11498
 URL: https://issues.apache.org/jira/browse/HADOOP-11498
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki


The package is renamed from org.htrace to org.apache.htrace.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11405) Configuration should not change the behavior by the value of quietmode

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-11405:
-

 Summary: Configuration should not change the behavior by the value 
of quietmode
 Key: HADOOP-11405
 URL: https://issues.apache.org/jira/browse/HADOOP-11405
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Minor


FsShell throws RuntimeException if the Configuration can not find default 
resource such as core-default.xml and core-site.xml. Other tools and daemons 
ignore absence of default resources because only FsShell calls 
Configuration#setQuietMode(false).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Release Apache Hadoop 2.6.0

[Masatake Iwasaki]

+1 (non-binding)

+ built from source tarball
+ ran some example jobs in HiBench such as
  nutchindexing, wordcount, dfsioe, hivebench, kmeans, pagerank, bayes, sort, 
terasort
  on 3 node cluster.

I'm on the same side with Tsuyoshi and would like HDFS-6833 to be fixed in 
2.6.1 at least.

Masatake Iwasaki

(11/19/14, 7:41), Tsuyoshi OZAWA wrote:

+1 if once discussion about HDFS-6833 is done(non-binding).

* Verified checksum.
* Deployed cluster and ran some mr examples.
* Checked RM restart while running jobs.


  I think you have to pass along -Pdist too… afaik that is how the Jenkins we 
use for building release artifacts operates.


Thanks for your comment, Arun. It's still fails with -Pdist option. It
can be environment-depend problem. +1 for fixing at 2.6.1.

Thanks,
- Tsuyoshi

On Wed, Nov 19, 2014 at 5:04 AM, Karthik Kambatla ka...@cloudera.com wrote:

+1 (binding) on the source tarball, would be nice to redo the binary
tarball.

- Stood up a pseudo-dist cluster, and ran some HDFS and MR jobs.
- The binary is about 40 MB larger than the previous release; it appears
the docs are copied over twice - share/doc/hadoop and share/hadoop. The
binary tarball from the release build should have everything in the right
place, I can help with this if needed.
- index.html (under share/doc/hadoop) is stale, and could be updated. If we
ship the bits as is, we should at least put the new features on the
website. Also, highlight the alpha features (like YARN reservations) as
alpha.

Thanks for putting this together, Arun.

Karthik

On Tue, Nov 18, 2014 at 11:07 AM, Jian He j...@hortonworks.com wrote:


+1,

Built from source.
Deployed a single node cluster.
Ran sample MapReduce jobs while restarting RM successfully.

Jian

On Tue, Nov 18, 2014 at 8:47 AM, Eric Payne erichadoo...@yahoo.com.invalid



wrote:


+1 . Thanks Arun, for producing this release. I downloaded and built
the source. I started local cluster and ran wordcount, sleep, and

streaming

jobs.

  - I ran a distributed shell job which tested preserving containers

across

AM restart by setting the -keep_containers_across_application_attempts

flag

and killing the first AM once the containers start. I checked the results
by looking in the timeline server and comparing the start times of the
non-AM containers against the start times of the later AM container. -
I enabled the preemption feature and verified containers were preempted

and

queues were adjusted to guaranteed levels.
- I ran unit tests for hadoop-yarn-server-resourcemanage. All passed with
the exception of TestContainerResourceUsage. - I ran unit tests for
hadoop-hdfs. All passed with the exception of
TestBPOfferService#testBasicFunctionality (HDFS-3930) Thank you,-Eric

Payne





   From: Arun C Murthy a...@hortonworks.com
  To: common-dev@hadoop.apache.org common-dev@hadoop.apache.org; 
hdfs-...@hadoop.apache.org hdfs-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org yarn-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org mapreduce-...@hadoop.apache.org
  Sent: Thursday, November 13, 2014 5:08 PM
  Subject: [VOTE] Release Apache Hadoop 2.6.0

Folks,

I've created another release candidate (rc1) for hadoop-2.6.0 based on

the

feedback.

The RC is available at:
http://people.apache.org/~acmurthy/hadoop-2.6.0-rc1
The RC tag in git is: release-2.6.0-rc1

The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/orgapachehadoop-1013.

Please try the release and vote; the vote will run for the usual 5 days.

thanks,
Arun


--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity

to

which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified

that

any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender

immediately

and delete it from your system. Thank You.






--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.

[jira] [Created] (HADOOP-11284) Fix variable name mismatch in hadoop-functions.sh

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-11284:
-

 Summary: Fix variable name mismatch in hadoop-functions.sh
 Key: HADOOP-11284
 URL: https://issues.apache.org/jira/browse/HADOOP-11284
 Project: Hadoop Common
  Issue Type: Bug
  Components: scripts
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Minor


Some functions use variables not given as argument but defined in outside of 
function. The variables are used as name of pid file. Though 
hadoop-functions.sh works by chance now, it should be fixed to avoid future bug.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Some information required related to Hadoop documentation developement

[Masatake Iwasaki]

Hi Naga,


   1.  what and how are these *.vm files(format) used in Hadoop [Hdfs and 
Yarn] documentation ?
   2.  Does vm files stand for velocity macro?

Velocity macro is used for variable substitution as far as I know.
For example::

  ---
  YARN Timeline Server
  ---
  ---
  ${maven.build.timestamp}

Though many of dosc uses APT format ( 
http://maven.apache.org/doxia/references/apt-format.html ),
you can also use markdown.
See 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown
for example.


  5.  Are these documentation pages built if  -Pdocs  parameters are provided during 
mvn install ? If so which Pom is responsible for building the html files ? Basically how 
to modify and test the generated html files?

Run mvn site to build site docs.
Running mvn site site:stage -DstagingDirectory=/var/www/html/hadoop-site
puts built docs to specifiled path.


  6.  From earlier patches  (ex : yarn 1696) i figured out that to add new page(in yarn 
documentation) we can add a *.vm file in 
hadoop-yarn-project\hadoop-yarn\hadoop-yarn-site\src\site\apt and make entry in 
hadoop-project\src\site\site.xml  for the new page? Is that sufficient or anything else 
needs to be taken care?

I think it is sufficient.


Masatake Iwasaki


(11/6/14, 9:51), Naganarasimha G R (Naga) wrote: Hi All,
 I wanted to know the following related to Hadoop Documentation

1.  what and how are these *.vm files(format) used in Hadoop [Hdfs and 
Yarn] documentation ?
2.  Does vm files stand for velocity macro?
3.  Do we use some kind of editor to create/update these files ?
4.  Are there any guidelines for writing Hadoop/YARN documentation ?
5.  Are these documentation pages built if  -Pdocs  parameters are provided during 
mvn install ? If so which Pom is responsible for building the html files ? Basically how 
to modify and test the generated html files?
6.  From earlier patches  (ex : yarn 1696) i figured out that to add new page(in yarn 
documentation) we can add a *.vm file in 
hadoop-yarn-project\hadoop-yarn\hadoop-yarn-site\src\site\apt and make entry in 
hadoop-project\src\site\site.xml  for the new page? Is that sufficient or anything else 
needs to be taken care?


 Regards,

 Naga



 Huawei Technologies Co., Ltd.
 Phone:
 Fax:
 Mobile:  +91 9980040283
 Email: naganarasimh...@huawei.commailto:naganarasimh...@huawei.com
 Huawei Technologies Co., Ltd.
 Bantian, Longgang District,Shenzhen 518129, P.R.China
 http://www.huawei.com

[jira] [Created] (HADOOP-11266) Remove no longer supported activation properties for packaging from pom

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-11266:
-

 Summary: Remove no longer supported activation properties for 
packaging from pom
 Key: HADOOP-11266
 URL: https://issues.apache.org/jira/browse/HADOOP-11266
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Trivial


Packaging rpm and deb are no longer supported.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11256) Some site docs have inconsistent appearance

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-11256:
-

 Summary: Some site docs have inconsistent appearance
 Key: HADOOP-11256
 URL: https://issues.apache.org/jira/browse/HADOOP-11256
 Project: Hadoop Common
  Issue Type: Bug
  Components: documentation
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Minor


The site docs of hadoop-gridmix, hadoop-rumen and hadoop-mapreduce-client-hs 
use default site.css because they do not have it in resources.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11242) Record the time of calling in tracing span of IPC server

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-11242:
-

 Summary: Record the time of calling in tracing span of IPC server
 Key: HADOOP-11242
 URL: https://issues.apache.org/jira/browse/HADOOP-11242
 Project: Hadoop Common
  Issue Type: Improvement
  Components: ipc
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Minor


Current tracing span starts when the Call is put into callQueue. Recording the 
time of calling is useful to debug.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11150) hadoop command should accept CLASSNAME without package name

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-11150:
-

 Summary: hadoop command should accept CLASSNAME without package 
name
 Key: HADOOP-11150
 URL: https://issues.apache.org/jira/browse/HADOOP-11150
 Project: Hadoop Common
  Issue Type: Improvement
  Components: scripts
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Minor


hadoop_validate_classname checks whether the classname contains .. It is 
possible that classname without package is used in some examples or tutorials.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11146) Update document of GridMix

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-11146:
-

 Summary: Update document of GridMix
 Key: HADOOP-11146
 URL: https://issues.apache.org/jira/browse/HADOOP-11146
 Project: Hadoop Common
  Issue Type: Improvement
  Components: documentation
Reporter: Masatake Iwasaki
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11147) Update document of Rumen

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-11147:
-

 Summary: Update document of Rumen
 Key: HADOOP-11147
 URL: https://issues.apache.org/jira/browse/HADOOP-11147
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-10976) moving the source code of hadoop-tools docs to the directry under hadoop-tools

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-10976:
-

 Summary: moving the source code of hadoop-tools docs to the 
directry under hadoop-tools
 Key: HADOOP-10976
 URL: https://issues.apache.org/jira/browse/HADOOP-10976
 Project: Hadoop Common
  Issue Type: Improvement
  Components: documentation
Reporter: Masatake Iwasaki
Priority: Minor


Some of the doc files of hadoop-tools are placed in the mapreduce project. It 
should be moved for the ease of maintenance.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: [VOTE] Release Apache Hadoop 2.5.0 RC2

[Masatake Iwasaki]

+1 (non-binding)

+ verified MD5 for tarball and source tarball
+ built from source tarball
+ ran example jobs such as
  nutchindexing, wordcount, dfsioe, hivebench, kmeans, pagerank, bayes, sort, 
terasort
  with HiBench on the cluster with 3 slave nodes.


(8/6/14, 13:59), Karthik Kambatla wrote:

Hi folks,

I have put together a release candidate (rc2) for Hadoop 2.5.0.

The RC is available at: http://people.apache.org/~kasha/hadoop-2.5.0-RC2/
The RC tag in svn is here:
https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.5.0-rc2/
The maven artifacts are staged at:
https://repository.apache.org/content/repositories/orgapachehadoop-1009/

You can find my public key at:
http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS

Please try the release and vote. The vote will run for the now usual 5
days.

Thanks

Re: [VOTE] Release Apache Hadoop 2.3.0

[Masatake Iwasaki]

+1 (non-binding)

I downloaded the source tarball,
built it,
started pseudo distrubuted cluster,
run wordcount example job.


 The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.3.0-rc0

Though the hadoop-2.3.0.tar.gz downloaded from above URL
contains site documentation in share/hadoop dir,
building from the source tar ball results in
no inclusion of site docs in binary tarball.


Regards,
Masatake Iwasaki


(2014/02/11 6:49), Arun C Murthy wrote: Folks,

 I've created a release candidate (rc0) for hadoop-2.3.0 that I would like to 
get released.

 The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.3.0-rc0
 The RC tag in svn is here: 
https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.3.0-rc0

 The maven artifacts are available via repository.apache.org.

 Please try the release and vote; the vote will run for the usual 7 days.

 thanks,
 Arun

 PS: Thanks to Andrew, Vinod  Alejandro for all their help in various release 
activities.

Re: Maintaining documentation

[Masatake Iwasaki]

 I added you as a contributor on Hadoop Common, let me know if you need
 anything else.

Thanks Andrew!

(2014/01/31 8:34), Andrew Wang wrote:

I added you as a contributor on Hadoop Common, let me know if you need
anything else.

I've hit that annoying dialog box before too, but what works for me is
using a private browsing window. Todd posited that the error is due to
stale cookies after a JIRA upgrade, and this fixes it.


On Thu, Jan 30, 2014 at 8:43 AM, Masatake Iwasaki 
iwasak...@oss.nttdata.co.jp wrote:


I've added your username to the wiki


I appreciate it Steve!



(2014/01/30 17:44), Steve Loughran wrote:


On 30 January 2014 01:10, Masatake Iwasaki iwasak...@oss.nttdata.co.jp
wrote:

  Hi JIRA and Wiki admins,


I would like permission to assign JIRA issues to myself
and edit Hadoop Wiki pages.
I have already fixed some on HADOOP-10086 and
I think there is further work.

My username is iwasakims on JIRA
and MasatakeIwasaki on Hadoop Wiki.

Regards,
Masatake Iwasaki


  I've added your username to the wiki


I can't add you you JIRA as every time I try to edit the roles page on
chrome, firefox or safari, a dialog pops up telling me it can't contact
the
JIRA server. Hopefully someone else will be more successful

Re: Maintaining documentation

[Masatake Iwasaki]

 I've added your username to the wiki

I appreciate it Steve!


(2014/01/30 17:44), Steve Loughran wrote:

On 30 January 2014 01:10, Masatake Iwasaki iwasak...@oss.nttdata.co.jpwrote:


Hi JIRA and Wiki admins,

I would like permission to assign JIRA issues to myself
and edit Hadoop Wiki pages.
I have already fixed some on HADOOP-10086 and
I think there is further work.

My username is iwasakims on JIRA
and MasatakeIwasaki on Hadoop Wiki.

Regards,
Masatake Iwasaki



I've added your username to the wiki

I can't add you you JIRA as every time I try to edit the roles page on
chrome, firefox or safari, a dialog pops up telling me it can't contact the
JIRA server. Hopefully someone else will be more successful

Re: Maintaining documentation

[Masatake Iwasaki]

Hi JIRA and Wiki admins,

I would like permission to assign JIRA issues to myself
and edit Hadoop Wiki pages.
I have already fixed some on HADOOP-10086 and
I think there is further work.

My username is iwasakims on JIRA
and MasatakeIwasaki on Hadoop Wiki.

Regards,
Masatake Iwasaki



(2014/01/28 15:47), Arpit Agarwal wrote:

Done, thanks guys for finding and fixing doc issues!


On Mon, Jan 27, 2014 at 10:27 PM, Akira AJISAKA
ajisa...@oss.nttdata.co.jpwrote:


Thank you for reviewing and committing, Arpit!



HADOOP-6350 - not sure of the status. Is the attached patch ready for
review?


The patch is not ready. I need to add the new metrics for the latest trunk.
I'll create a new patch in this week.

In addition, could you please review HADOOP-9830?
This issue is to fix a typo at the top page and I think it's easy to
review.

Akira


(2014/01/28 6:44), Arpit Agarwal wrote:


I committed the following patches today:
HADOOP-10086
   HADOOP-9982
   HADOOP-10212
HDFS-5297

Remaining from your initial list:
HADOOP-10139 - needs to be rebased
HDFS-5492 - I will review it this week.
HADOOP-6350 - not sure of the status. Is the attached patch ready for
review?

I will look at the other common/HDFS patches as I get time after this
week,
unless another committer wants to take a look.

Arpit

On Wed, Jan 22, 2014 at 2:40 AM, Akira AJISAKA
ajisa...@oss.nttdata.co.jpwrote:

  Akira, could you please resend a list of your patches that need



reviewing



if you have it handy.





Here is the list of all the documentation patches I have.


https://issues.apache.org/jira/browse/HADOOP-6350?jql=
project%20in%20(HADOOP%2C%20YARN%2C%20HDFS%2C%20MAPREDUCE)%20AND%
20resolution%20%3D%20Unresolved%20AND%20component%20%3D%
20documentation%20AND%20status%20%3D%20%22Patch%20Available%22%20AND%
20assignee%20in%20(ajisakaa)%20ORDER%20BY%20priority%20DESC

In the above list, the JIRAs I think high priority are as follows:

   * HADOOP-6350
   * HADOOP-9982
   * HADOOP-10139
   * HADOOP-10212
   * YARN-1531
   * HDFS-5297
   * HDFS-5492
   * MAPREDUCE-4282

I will appreciate your reviews.

Thanks,
Akira


(2014/01/22 13:17), Masatake Iwasaki wrote:

 I will review

HADOOP-10086https://issues.apache.org/jira/browse/HADOOP-10086this
week unless another committer gets to it first.

I appreciate it Arpit!


Thanks for your efforts to clean up the documentation. Unfortunately
some
fixes can get lost in the noise and this is true not just for
documentation
fixes, so thanks for bringing these to our attention.

I will do what I can do to improve this. Such as helping review process.


Regards,
Masatake Iwasaki


(2014/01/22 3:22), Arpit Agarwal wrote:

  I will review

HADOOP-10086https://issues.apache.org/jira/browse/HADOOP-10086this
week unless another committer gets to it first.

Akira, could you please resend a list of your patches that need
reviewing
if you have it handy.

Thanks for your efforts to clean up the documentation. Unfortunately
some
fixes can get lost in the noise and this is true not just for
documentation
fixes, so thanks for bringing these to our attention.

Regards,
Arpit




On Mon, Jan 20, 2014 at 11:47 PM, Akira AJISAKA
ajisa...@oss.nttdata.co.jpwrote:

   Hi All,



I wrote several patches for documentation
but most of them are not reviewed.

I want to improve not only the new features but also
the documents. I also think the documents are not
well maintained mainly because :

* Deprecated commands and parameters still exist.
* Undocumented command options exist.
* There're a lot of dead links.

I'll be happy when the above problems are fixed for 2.4 release.
I'm waiting for your advices and reviews!

Thanks Masatake for this proposal.

Regards,
Akira

(2014/01/21 9:03), Masatake Iwasaki wrote:

  Hi developers,


I wrote a patch for Hadoop's site documentation and would like
reviews.
https://issues.apache.org/jira/browse/HADOOP-10086

Though I would like to fix documentation further,
there seems to be many open JIRA issues concerning documents.
I think the documents is not well maintained now
because the documentation work does not tend to get
enough attention of reviewers and commiters.

In order to impove the situation,
should I write Wiki pages rather than site documentation
or refactor whole documentation structure
in the way like HBase book (http://hbase.apache.org/book/book.html)?

I will appreciate your advices.

Regards,
Masatake Iwasaki

Re: Maintaining documentations

[Masatake Iwasaki]

 I will review 
HADOOP-10086https://issues.apache.org/jira/browse/HADOOP-10086this
 week unless another committer gets to it first.

I appreciate it Arpit!


 Thanks for your efforts to clean up the documentation. Unfortunately some
 fixes can get lost in the noise and this is true not just for documentation
 fixes, so thanks for bringing these to our attention.

I will do what I can do to improve this. Such as helping review process.


Regards,
Masatake Iwasaki


(2014/01/22 3:22), Arpit Agarwal wrote:

I will review 
HADOOP-10086https://issues.apache.org/jira/browse/HADOOP-10086this
week unless another committer gets to it first.

Akira, could you please resend a list of your patches that need reviewing
if you have it handy.

Thanks for your efforts to clean up the documentation. Unfortunately some
fixes can get lost in the noise and this is true not just for documentation
fixes, so thanks for bringing these to our attention.

Regards,
Arpit




On Mon, Jan 20, 2014 at 11:47 PM, Akira AJISAKA
ajisa...@oss.nttdata.co.jpwrote:


Hi All,

I wrote several patches for documentation
but most of them are not reviewed.

I want to improve not only the new features but also
the documents. I also think the documents are not
well maintained mainly because :

  * Deprecated commands and parameters still exist.
  * Undocumented command options exist.
  * There're a lot of dead links.

I'll be happy when the above problems are fixed for 2.4 release.
I'm waiting for your advices and reviews!

Thanks Masatake for this proposal.

Regards,
Akira

(2014/01/21 9:03), Masatake Iwasaki wrote:

Hi developers,

I wrote a patch for Hadoop's site documentation and would like reviews.
https://issues.apache.org/jira/browse/HADOOP-10086

Though I would like to fix documentation further,
there seems to be many open JIRA issues concerning documents.
I think the documents is not well maintained now
because the documentation work does not tend to get
enough attention of reviewers and commiters.

In order to impove the situation,
should I write Wiki pages rather than site documentation
or refactor whole documentation structure
in the way like HBase book (http://hbase.apache.org/book/book.html)?

I will appreciate your advices.

Regards,
Masatake Iwasaki

Maintaining documentations

[Masatake Iwasaki]

Hi developers,

I wrote a patch for Hadoop's site documentation and would like reviews.
https://issues.apache.org/jira/browse/HADOOP-10086

Though I would like to fix documentation further,
there seems to be many open JIRA issues concerning documents.
I think the documents is not well maintained now
because the documentation work does not tend to get
enough attention of reviewers and commiters.

In order to impove the situation,
should I write Wiki pages rather than site documentation
or refactor whole documentation structure
in the way like HBase book (http://hbase.apache.org/book/book.html)?

I will appreciate your advices.

Regards,
Masatake Iwasaki

[jira] [Created] (HADOOP-10086) User document for authentication in secure cluster

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-10086:
-

 Summary: User document for authentication in secure cluster
 Key: HADOOP-10086
 URL: https://issues.apache.org/jira/browse/HADOOP-10086
 Project: Hadoop Common
  Issue Type: Improvement
  Components: documentation
Reporter: Masatake Iwasaki
Priority: Minor


There are no independent section for basic security features such as 
authentication and group mapping in the user documentation, though there are 
sections for Service Level Authorization and HTTP Authentication.
Creating independent section for authentication and moving contents about 
secure cluster currently residing in Cluster Setup section could be good 
starting point.




--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (HADOOP-9979) HADOOP_IDENT_STRING should not be changed in hadoop-env.sh for hadoop daemons running as services

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-9979:


 Summary: HADOOP_IDENT_STRING should not be changed in 
hadoop-env.sh for hadoop daemons running as services
 Key: HADOOP-9979
 URL: https://issues.apache.org/jira/browse/HADOOP-9979
 Project: Hadoop Common
  Issue Type: Improvement
  Components: conf
Reporter: Masatake Iwasaki
Priority: Trivial


Because configurations for services are set in the files such as /etc/default/* 
for Linux cases. In order to avoid this to be overrided by hadoop-env.sh, there 
should be some comments for caution.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira