[jira] [Created] (HADOOP-14240) Configuration#get return value optimization

2017-03-24 Thread Jonathan Eagles (JIRA) 
Jonathan Eagles created HADOOP-14240:


 Summary: Configuration#get return value optimization
 Key: HADOOP-14240
 URL: https://issues.apache.org/jira/browse/HADOOP-14240
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Jonathan Eagles
Assignee: Jonathan Eagles


The string array return value can be more efficiently determined and some 
general redundancies can be removed to improve the speed for Configuration.get.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14239) S3A Retry Multiple S3 Key Deletion

2017-03-24 Thread Kazuyuki Tanimura (JIRA) 
Kazuyuki Tanimura created HADOOP-14239:
--

 Summary: S3A Retry Multiple S3 Key Deletion
 Key: HADOOP-14239
 URL: https://issues.apache.org/jira/browse/HADOOP-14239
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/s3
Affects Versions: 3.0.0-alpha2, 3.0.0-alpha1, 2.8.0, 2.8.1
 Environment: EC2, AWS
Reporter: Kazuyuki Tanimura


When fs.s3a.multiobjectdelete.enable == true, It tries to delete multiple S3 
keys at once.

Although this is a great feature, it becomes problematic when AWS fails 
deleting some S3 keys out of the deletion list. The aws-java-sdk internally 
retries to delete them, but it does not help because it simply retries the same 
list of S3 keys including the successfully deleted ones. In that case, all 
successive retries fail deleting previously deleted keys since they do not 
exist any more. Eventually it throws an Exception and leads to a job failure 
entirely.

Luckily, the AWS API reports which keys it failed to delete. We should retry 
only for the keys that failed to be deleted from S3A



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14238) [Umbrella] Rechecking Guava's object is not exposed to user-facing API

2017-03-24 Thread Tsuyoshi Ozawa (JIRA) 
Tsuyoshi Ozawa created HADOOP-14238:
---

 Summary: [Umbrella] Rechecking Guava's object is not exposed to 
user-facing API
 Key: HADOOP-14238
 URL: https://issues.apache.org/jira/browse/HADOOP-14238
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Tsuyoshi Ozawa


This is reported by [~hitesh] on HADOOP-10101.
At least, AMRMClient#waitFor takes Guava's Supplier instance as an instance.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [RESULT] [VOTE] Release Apache Hadoop 2.8.0 (RC3)

2017-03-24 Thread Junping Du 
Thanks Allen, I already add it back in JIRA's 2.8.0 release date.
However, I met another issue in uploading our 2.8.0 release bits to SVN: 
hadoop-2.8.0.tar.gz get uploaded failed with following exception:

Adding  (bin)  hadoop-2.8.0/hadoop-2.8.0.tar.gz
Transmitting file data .svn: E175002: Commit failed (details follow):
svn: E175002: PUT request on 
'/repos/dist/!svn/txr/18902-g1s/release/hadoop/common/hadoop-2.8.0/hadoop-2.8.0.tar.gz'
 failed

Other (smaller) files get uploaded successfully. Once I suspected it could due 
to no space, so I removed some old release bits (like: 2.5.2) but still no 
luck. So may be the size of hadoop-2.8.0.tar.gz become a problem here? 
I already reported the issue in 
https://issues.apache.org/jira/browse/INFRA-13749, but will appreciate if 
someone got idea on how to get through this.

Thanks,

Junping

From: Allen Wittenauer 
Sent: Thursday, March 23, 2017 5:27 PM
To: Junping Du
Cc: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Subject: Re: [RESULT] [VOTE] Release Apache Hadoop 2.8.0 (RC3)

Just a heads up.  Looks like some removed the Finish Date off of 2.8.0 in JIRA. 
 It needs to be put back to match what is in the artifacts that we voted on.

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14237) S3A Support Shared Instance Profile Credentials Across All Instances

2017-03-24 Thread Kazuyuki Tanimura (JIRA) 
Kazuyuki Tanimura created HADOOP-14237:
--

 Summary: S3A Support Shared Instance Profile Credentials Across 
All Instances
 Key: HADOOP-14237
 URL: https://issues.apache.org/jira/browse/HADOOP-14237
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/s3
Affects Versions: 3.0.0-alpha2, 3.0.0-alpha1, 2.8.0, 2.8.1
 Environment: EC2, AWS
Reporter: Kazuyuki Tanimura


When I run a large Hadoop cluster on EC2 instances with IAM Role, it fails 
getting the instance profile credentials, eventually all jobs on the cluster 
fail. Since a number of S3A clients (all mappers and reducers) try to get the 
credentials, the AWS credential endpoint starts responding 5xx and 4xx error 
codes.

SharedInstanceProfileCredentialsProvider.java is sort of trying to solve it, 
but it still does not share the credentials with other EC2 instances / 
processes.

This issue prevents users from creating Hadoop clusters on EC2



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14236) S3Guard: S3AFileSystem::rename() should move non-listed sub-directory entries in metadata store

2017-03-24 Thread Mingliang Liu (JIRA) 
Mingliang Liu created HADOOP-14236:
--

 Summary: S3Guard: S3AFileSystem::rename() should move non-listed 
sub-directory entries in metadata store
 Key: HADOOP-14236
 URL: https://issues.apache.org/jira/browse/HADOOP-14236
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs/s3
Reporter: Mingliang Liu
Assignee: Mingliang Liu


After running integration test {{ITestS3AFileSystemContract}}, I found the 
following items are not cleaned up in DynamoDB:
{code}
parent=/mliu-s3guard/user/mliu/s3afilesystemcontract/testRenameDirectoryAsExisting/dir,
 child=subdir
parent=/mliu-s3guard/user/mliu/s3afilesystemcontract/testRenameDirectoryAsExistingNew/newdir/subdir,
 child=file2
{code}
At first I thought it’s similar to [HADOOP-14226] or [HADOOP-14227], and we 
need to be careful when cleaning up test data.

Then I found it’s a bug in the code of integrating S3Guard with S3AFileSystem: 
for rename we miss sub-directory items to put (dest) and delete (src). The 
reason is that in S3A, we delete those fake directory objects if they are not 
necessary, e.g. non-empty. So when we list the objects to rename, the object 
summaries will only return _file_ objects. This has two consequences after 
rename:
#  there will be left items for src path in metadata store - left-overs will 
confuse {{get(Path)}} which should return null
# we are not persisting the whole subtree for dest path to metadata store - 
this will break the DynamoDBMetadataStore invariant: _if a path exists, all its 
ancestors will also exist in the table_.

Existing tests are not complaining about this though. If this is a real bug, 
let’s address it here.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14235) S3A Path does not understand colon (:) when globbing

2017-03-24 Thread Kazuyuki Tanimura (JIRA) 
Kazuyuki Tanimura created HADOOP-14235:
--

 Summary: S3A Path does not understand colon (:) when globbing
 Key: HADOOP-14235
 URL: https://issues.apache.org/jira/browse/HADOOP-14235
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/s3
Affects Versions: 3.0.0-alpha2, 3.0.0-alpha1, 2.8.0, 2.8.1
Reporter: Kazuyuki Tanimura


S3 paths, colons (:) are valid character in S3 paths. However, the Java URI 
class, which is used in the Path class, does not allow it.

This becomes a problem particularly when we are globbing S3 paths. The globber 
thinks paths with colons are invalid paths and throws URISyntaxException.

The reason is we are sharing Globber.java with all other Fs. Some of the rules 
for regular Fs are not applicable to S3 just like this colon as an example.

Same issue is reported here https://issues.apache.org/jira/browse/SPARK-20061

The good news is I have a one line fix that I am about to send a pull request.

However, for a right fix, we should separate the S3 globber from the 
Globber.java as proposed at https://issues.apache.org/jira/browse/HADOOP-13371



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14234) Improve ADLS FileSystem tests with JUnit4

2017-03-24 Thread John Zhuge (JIRA) 
John Zhuge created HADOOP-14234:
---

 Summary: Improve ADLS FileSystem tests with JUnit4
 Key: HADOOP-14234
 URL: https://issues.apache.org/jira/browse/HADOOP-14234
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs/adl, test
Affects Versions: 2.8.0
Reporter: John Zhuge
Priority: Minor


HADOOP-14180 switches FileSystem contract tests to JUnit4 and makes various 
enhancements. Improve ADLS FileSystem contract tests based on that.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le

2017-03-24 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/267/

[Mar 23, 2017 12:54:54 PM] (stevel) HADOOP-14120 needless 
S3AFileSystem.setOptionalPutRequestParameters in
[Mar 23, 2017 4:28:10 PM] (raviprak) HADOOP-14213. Move Configuration runtime 
check for hadoop-site.xml to
[Mar 23, 2017 6:33:29 PM] (liuml07) HADOOP-14135. Remove URI parameter in 
AWSCredentialProvider
[Mar 23, 2017 7:13:23 PM] (cdouglas) HADOOP-14196. Azure Data Lake doc is 
missing required config entry.
[Mar 23, 2017 7:17:27 PM] (cdouglas) HADOOP-14197. Fix ADLS doc for credential 
provider. Contributed by John
[Mar 23, 2017 8:34:03 PM] (liuml07) HADOOP-14214. 
DomainSocketWatcher::add()/delete() should not self
[Mar 23, 2017 8:57:08 PM] (jzhuge) HADOOP-14195. 
CredentialProviderFactory$getProviders is not thread-safe.
[Mar 24, 2017 6:31:27 AM] (jzhuge) HDFS-10649. Remove unused 
PermissionStatus#applyUMask. Contributed by




-1 overall


The following subsystems voted -1:
compile unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc javac


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.security.TestShellBasedUnixGroupsMapping 
   hadoop.hdfs.TestEncryptedTransfer 
   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer 
   hadoop.hdfs.server.mover.TestMover 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.yarn.server.timeline.TestRollingLevelDB 
   hadoop.yarn.server.timeline.TestTimelineDataManager 
   hadoop.yarn.server.timeline.TestLeveldbTimelineStore 
   hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore 
   hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore 
   
hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer 
   hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore 
   
hadoop.yarn.server.resourcemanager.TestOpportunisticContainerAllocatorAMService 
   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization 
   hadoop.yarn.server.TestContainerManagerSecurity 
   hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore 
   hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient 
   hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore 
   hadoop.yarn.applications.distributedshell.TestDistributedShell 
   hadoop.mapred.TestShuffleHandler 
   hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService 

Timed out junit tests :

   org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean 
   org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache 
  

   compile:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/267/artifact/out/patch-compile-root.txt
  [136K]

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/267/artifact/out/patch-compile-root.txt
  [136K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/267/artifact/out/patch-compile-root.txt
  [136K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/267/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
  [144K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/267/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [244K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/267/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [16K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/267/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt
  [52K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/267/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [68K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/267/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [324K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/267/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timeline-pluginstorage.txt
  [28K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/267/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/267/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-ui.txt
  [8.0K]

Re: HADOOP-14163 proposal for new hadoop.apache.org

2017-03-24 Thread Andrew Wang 
Thanks again for working on this Marton!

Based on my read of the blog post you linked, we should have the git branch
ready before asking infra to switch it over.

I can do a more detailed review on the JIRA once you rev, and can help with
the INFRA ticket once it's ready. We'll also have to update BUILDING.txt
and the wiki instructions as part of this.

Best,
Andrew

On Fri, Mar 24, 2017 at 3:06 AM, Marton Elek  wrote:

>
>
> Thank you all of the feedbacks, I fixed all of them (except one, see the
> comment below) and updated the http://hadoop.anzix.net preview site.
>
> So the next steps:
>
> 0. Let me know if you have any comment about the latest version
>
> 1. I wait for the 2.8.0 announcement, and migrate the new announcement as
> well. (wouldn't like to complicate the 2.8.0 with the site change)
>
> 2. I like the suggestion of Owen to move the site to a specific git
> branch. I wouldn't like to pending on it if it's too much time, but if any
> of the commiters could pick it up, I would wait for it.
>
> I tested it, and seems to be easy:
>
> git svn clone https://svn.apache.org/repos/asf/hadoop/common/site/main
> cd main
> git remote add elek g...@github.com:elek/hadoop.git
> git push elek master:asf-site
>
> According to the blog entry, an INFRA issue should be opened (I guess by a
> commiter or maybe a pmc member):
>
> https://blogs.apache.org/infra/entry/git_based_websites_available
>
> 3. After that I can submit the new site as a regular patch against the
> asf-site branch.
>
> 4. If it's merged, I can update the release wiki pages
>
> Marton
>
> ps:
>
> The only suggested item which is not implemented is the short version
> names in the documentation menu (2.7 instead of 2.7.3).
>
> I think there are two forces: usability of the site and the simplicity of
> the site generation. Ideally a new release could be added to the site as
> easy as possible (that was one of the motivation of the migration).
>
> While a new tag could be added to the header of the markdown files (eg:
> versionLine: 3.0), it requires multiple files update during a new release.
> And if something would be missed, there could be displayed multiple "2.7"
> menu item (one for 2.7.3 and for 2.7.4). So the current method is not so
> nice, but much more bug-safe.
>
> I prefer to keep the current/content in this step (if possible) and if the
> site is migrated we can submit new patches (hopefully against a git branch)
> in the normal way and further improve the site.
>
>
> 
> From: Owen O'Malley 
> Sent: Monday, March 13, 2017 6:15 PM
> To: Marton Elek
> Cc: common-dev@hadoop.apache.org
> Subject: Re: HADOOP-14163 proposal for new hadoop.apache.org
>
> Thanks for addressing this. Getting rid of Hadoop's use of forrest is a
> good thing.
>
> In terms of content, the documentation links should be sorted by number
> with only the latest from each minor release line (eg. 3.0, 2.7, 2.6).
>
> The download page points to the mirrors for checksums and signatures. It
> should use the direct links, such as
>
> https://dist.apache.org/repos/dist/release/hadoop/common/
> hadoop-2.7.3/hadoop-2.7.3-src.tar.gz.asc
> https://dist.apache.org/repos/dist/release/hadoop/common/
> hadoop-2.7.3/hadoop-2.7.3-src.tar.gz.mds
>
> Speaking of which, Hadoop's dist directory is huge and should be heavily
> pruned. We should probably take it down to just hadoop-2.6.5, hadoop-2.7.3,
> and hadoop-3.0.0-alpha2.
>
> You might also want to move us to git-pubsub so that we can use a branch in
> our source code git repository to publish the html. Typically this uses the
> asf-site branch.
>
> .. Owen
>
> On Mon, Mar 13, 2017 at 7:28 AM, Marton Elek 
> wrote:
>
> >
> > Hi,
> >
> > In the previous thread the current forrest based hadoop site is
> identified
> > as one of the pain points of the release process.
> >
> > I created a new version of the site with exactly the same content.
> >
> >  As it uses newer site generator (hugo), now:
> >
> > 1. It’s enough to create one new markdown file per release, and all the
> > documentation/download links will be automatically added.
> > 2. It requires only one single binary to render.
> >
> >
> > A preview version is temporary hosted at
> >
> >  http://hadoop.anzix.net/
> >
> > to make it easier to review.
> >
> >
> > For more details, you can check my comments on the issue
> > https://issues.apache.org/jira/browse/HADOOP-14163
> >
> > I would be thankful to get any feedback/review.
> >
> > Cheers,
> > Marton
> >
> >
> >
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

[jira] [Created] (HADOOP-14233) Don't pre-construct PreCondition.check failure message in Configuration#set

2017-03-24 Thread Jonathan Eagles (JIRA) 
Jonathan Eagles created HADOOP-14233:


 Summary: Don't pre-construct PreCondition.check failure message in 
Configuration#set
 Key: HADOOP-14233
 URL: https://issues.apache.org/jira/browse/HADOOP-14233
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Jonathan Eagles






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Reopened] (HADOOP-14232) RumenToSLS: rackName may contains slashes

2017-03-24 Thread Julien Vaudour (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-14232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julien Vaudour reopened HADOOP-14232:
-

> RumenToSLS: rackName may contains slashes
> -
>
> Key: HADOOP-14232
> URL: https://issues.apache.org/jira/browse/HADOOP-14232
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 2.6.0
>Reporter: Julien Vaudour
>Priority: Minor
>
> Rack name may contain slashes, so hostName will contain several slashes.
> Separation between rack name and hostName is the last slash.
> For example: /platform1/pod1/rack1/node1



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-14232) RumenToSLS: rackName may contains slashes

2017-03-24 Thread Julien Vaudour (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-14232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julien Vaudour resolved HADOOP-14232.
-
Resolution: Won't Fix

> RumenToSLS: rackName may contains slashes
> -
>
> Key: HADOOP-14232
> URL: https://issues.apache.org/jira/browse/HADOOP-14232
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 2.6.0
>Reporter: Julien Vaudour
>Priority: Minor
>
> Rack name may contain slashes, so hostName will contain several slashes.
> Separation between rack name and hostName is the last slash.
> For example: /platform1/pod1/rack1/node1



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14232) RumenToSLS: rackName may contains slashes

2017-03-24 Thread Julien Vaudour (JIRA) 
Julien Vaudour created HADOOP-14232:
---

 Summary: RumenToSLS: rackName may contains slashes
 Key: HADOOP-14232
 URL: https://issues.apache.org/jira/browse/HADOOP-14232
 Project: Hadoop Common
  Issue Type: Bug
  Components: tools
Affects Versions: 2.6.0
Reporter: Julien Vaudour
Priority: Minor


Rack name may contain slashes, so hostName will contain several slashes.
Separation between rack name and hostName is the last slash.
For example: /platform1/pod1/rack1/node1



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14231) Using parentheses is not allowed in auth_to_local regex

2017-03-24 Thread Andras Bokor (JIRA) 
Andras Bokor created HADOOP-14231:
-

 Summary: Using parentheses is not allowed in auth_to_local regex
 Key: HADOOP-14231
 URL: https://issues.apache.org/jira/browse/HADOOP-14231
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Andras Bokor
Assignee: Andras Bokor
Priority: Minor


I tried to set the following  property for auth_to_local property:
{code}"RULE:[2:$1]((n|d)n)s/.*/hdfs//{code}
but I got the following exception:
{code}Exception in thread "main" java.util.regex.PatternSyntaxException: 
Unclosed group near index 9
(nn|dn|jn{code}

I found that this occurs because {{ruleParser}} in 
{{org.apache.hadoop.security.authentication.util.KerberosName}} excludes 
closing parentheses.

I do not really see the value of excluding parentheses (do I miss something?) 
so I would remove this restriction to be able to use more regex functionalities.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2017-03-24 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/355/

[Mar 23, 2017 12:54:54 PM] (stevel) HADOOP-14120 needless 
S3AFileSystem.setOptionalPutRequestParameters in
[Mar 23, 2017 4:28:10 PM] (raviprak) HADOOP-14213. Move Configuration runtime 
check for hadoop-site.xml to
[Mar 23, 2017 6:33:29 PM] (liuml07) HADOOP-14135. Remove URI parameter in 
AWSCredentialProvider
[Mar 23, 2017 7:13:23 PM] (cdouglas) HADOOP-14196. Azure Data Lake doc is 
missing required config entry.
[Mar 23, 2017 7:17:27 PM] (cdouglas) HADOOP-14197. Fix ADLS doc for credential 
provider. Contributed by John
[Mar 23, 2017 8:34:03 PM] (liuml07) HADOOP-14214. 
DomainSocketWatcher::add()/delete() should not self
[Mar 23, 2017 8:57:08 PM] (jzhuge) HADOOP-14195. 
CredentialProviderFactory$getProviders is not thread-safe.
[Mar 24, 2017 6:31:27 AM] (jzhuge) HDFS-10649. Remove unused 
PermissionStatus#applyUMask. Contributed by




-1 overall


The following subsystems voted -1:
asflicense unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.security.TestRaceWhenRelogin 
   hadoop.net.TestDNS 
   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization 
   hadoop.yarn.server.TestContainerManagerSecurity 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/355/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/355/artifact/out/diff-compile-javac-root.txt
  [184K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/355/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/355/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/355/artifact/out/diff-patch-shellcheck.txt
  [24K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/355/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/355/artifact/out/whitespace-eol.txt
  [11M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/355/artifact/out/whitespace-tabs.txt
  [1.3M]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/355/artifact/out/diff-javadoc-javadoc-root.txt
  [2.2M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/355/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
  [136K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/355/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [324K]

   asflicense:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/355/artifact/out/patch-asflicense-problems.txt
  [4.0K]

Powered by Apache Yetus 0.5.0-SNAPSHOT   http://yetus.apache.org



-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14229) hadoop.security.auth_to_local example is incorrect in the documentation

2017-03-24 Thread Andras Bokor (JIRA) 
Andras Bokor created HADOOP-14229:
-

 Summary: hadoop.security.auth_to_local example is incorrect in the 
documentation
 Key: HADOOP-14229
 URL: https://issues.apache.org/jira/browse/HADOOP-14229
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Andras Bokor
Assignee: Andras Bokor
Priority: Trivial


Let's see jhs as example:
{code}RULE:[2:$1@$0](jhs/.*@.*REALM.TLD)s/.*/mapred/{code}
That means principal has 2 components (jhs/myhost@REALM).
The second column converts this to jhs@REALM. So the regex will not match on 
this since regex expects / in the principal.
I am not sure what was the original intend here so I suggest two solutions:
{code}RULE:[2:$1/$2@$0](jhs/.*@REALM.TLD)s/.*/mapred/{code}
or
{code}RULE:[2:$1@$0](j...@realm.tld)s/.*/mapred/{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: HADOOP-14163 proposal for new hadoop.apache.org

2017-03-24 Thread Marton Elek 


Thank you all of the feedbacks, I fixed all of them (except one, see the 
comment below) and updated the http://hadoop.anzix.net preview site.

So the next steps:

0. Let me know if you have any comment about the latest version

1. I wait for the 2.8.0 announcement, and migrate the new announcement as well. 
(wouldn't like to complicate the 2.8.0 with the site change)

2. I like the suggestion of Owen to move the site to a specific git branch. I 
wouldn't like to pending on it if it's too much time, but if any of the 
commiters could pick it up, I would wait for it. 

I tested it, and seems to be easy: 

git svn clone https://svn.apache.org/repos/asf/hadoop/common/site/main
cd main
git remote add elek g...@github.com:elek/hadoop.git
git push elek master:asf-site

According to the blog entry, an INFRA issue should be opened (I guess by a 
commiter or maybe a pmc member):

https://blogs.apache.org/infra/entry/git_based_websites_available

3. After that I can submit the new site as a regular patch against the asf-site 
branch.

4. If it's merged, I can update the release wiki pages

Marton

ps:

The only suggested item which is not implemented is the short version names in 
the documentation menu (2.7 instead of 2.7.3).

I think there are two forces: usability of the site and the simplicity of the 
site generation. Ideally a new release could be added to the site as easy as 
possible (that was one of the motivation of the migration).

While a new tag could be added to the header of the markdown files (eg: 
versionLine: 3.0), it requires multiple files update during a new release. And 
if something would be missed, there could be displayed multiple "2.7" menu item 
(one for 2.7.3 and for 2.7.4). So the current method is not so nice, but much 
more bug-safe.

I prefer to keep the current/content in this step (if possible) and if the site 
is migrated we can submit new patches (hopefully against a git branch) in the 
normal way and further improve the site.



From: Owen O'Malley 
Sent: Monday, March 13, 2017 6:15 PM
To: Marton Elek
Cc: common-dev@hadoop.apache.org
Subject: Re: HADOOP-14163 proposal for new hadoop.apache.org

Thanks for addressing this. Getting rid of Hadoop's use of forrest is a
good thing.

In terms of content, the documentation links should be sorted by number
with only the latest from each minor release line (eg. 3.0, 2.7, 2.6).

The download page points to the mirrors for checksums and signatures. It
should use the direct links, such as

https://dist.apache.org/repos/dist/release/hadoop/common/hadoop-2.7.3/hadoop-2.7.3-src.tar.gz.asc
https://dist.apache.org/repos/dist/release/hadoop/common/hadoop-2.7.3/hadoop-2.7.3-src.tar.gz.mds

Speaking of which, Hadoop's dist directory is huge and should be heavily
pruned. We should probably take it down to just hadoop-2.6.5, hadoop-2.7.3,
and hadoop-3.0.0-alpha2.

You might also want to move us to git-pubsub so that we can use a branch in
our source code git repository to publish the html. Typically this uses the
asf-site branch.

.. Owen

On Mon, Mar 13, 2017 at 7:28 AM, Marton Elek  wrote:

>
> Hi,
>
> In the previous thread the current forrest based hadoop site is identified
> as one of the pain points of the release process.
>
> I created a new version of the site with exactly the same content.
>
>  As it uses newer site generator (hugo), now:
>
> 1. It’s enough to create one new markdown file per release, and all the
> documentation/download links will be automatically added.
> 2. It requires only one single binary to render.
>
>
> A preview version is temporary hosted at
>
>  http://hadoop.anzix.net/
>
> to make it easier to review.
>
>
> For more details, you can check my comments on the issue
> https://issues.apache.org/jira/browse/HADOOP-14163
>
> I would be thankful to get any feedback/review.
>
> Cheers,
> Marton
>
>
>

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14228) Fix getNumMapTasks() documentation in JobConf

2017-03-24 Thread JIRA 
Joe Mészáros created HADOOP-14228:
-

 Summary: Fix getNumMapTasks() documentation in JobConf
 Key: HADOOP-14228
 URL: https://issues.apache.org/jira/browse/HADOOP-14228
 Project: Hadoop Common
  Issue Type: Bug
  Components: contrib/cloud, documentation
Reporter: Joe Mészáros
Priority: Minor


The original description of the getNumMapTasks() method in JobConf was invalid, 
because it referenced to the number of reducer tasks instead of the map tasks.

 from: Get configured the number of reduce tasks for this job.
  to: Get the configured number of map tasks for this job.

It was maybe the result of a tricky copy-paste ;-)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org