[jira] [Created] (HADOOP-15598) DataChecksum calculate checksum is contented on hashtable lock
Prasanth Jayachandran created HADOOP-15598: -- Summary: DataChecksum calculate checksum is contented on hashtable lock Key: HADOOP-15598 URL: https://issues.apache.org/jira/browse/HADOOP-15598 Project: Hadoop Common Issue Type: Bug Components: common Affects Versions: 3.2.0 Reporter: Prasanth Jayachandran Attachments: Screen Shot 2018-07-11 at 1.45.06 AM.png When profiling a multi-threaded hive streaming ingest, observed lock contention on java.util.Properties getProperty() to check if os is "sparc". java.util.Properties internally uses HashTable. HashTable.get() is synchronized method. In the test application, on a 30s profile with 64 threads ~40% CPU time is spent on getProperty() contention. See attached snapshot. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[VOTE] Release Apache Hadoop 2.7.7
Hi I've got RC0 of Hadoop 2.7.7 up for people to download and play with http://people.apache.org/~stevel/hadoop-2.7.7/RC0/ Nexus artifacts 2.7.7 are up in staging https://repository.apache.org/content/repositories/orgapachehadoop-1135 Git (signed) tag release-2.7.7-RC0, checksum c1aad84bd27cd79c3d1a7dd58202a8c3ee1ed3ac My current GPG key is 38237EE425050285077DB57AD22CF846DBB162A0 you can download this direct (and it MUST be direct) from the ASF HTTPS site https://dist.apache.org/repos/dist/release/hadoop/common/KEYS Please try the release and vote. The vote will run for 5 days. Thanks -Steve (I should add: this is my first ever attempt at a Hadoop release, please be (a) rigorous and (b) forgiving. Credit to Wei-Chiu Chuang for his advice). - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15232) AWSCredentialProviderList to throw custom NoCredentialsException; retry logic to handle
[
https://issues.apache.org/jira/browse/HADOOP-15232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-15232.
-
Resolution: Duplicate
> AWSCredentialProviderList to throw custom NoCredentialsException; retry logic
> to handle
> ---
>
> Key: HADOOP-15232
> URL: https://issues.apache.org/jira/browse/HADOOP-15232
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Priority: Minor
>
> When no credentials are found in the list, an {{AmazonClientException}} is
> raised with a message, which may then be retried the S3a retry code. If a
> specific exception were raised, the retry handler would know to skip
> everything here.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14592) ITestS3ATemporaryCredentials to cover all ddb metastore ops with session credentials
[
https://issues.apache.org/jira/browse/HADOOP-14592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-14592.
-
Resolution: Duplicate
HADOOP-15572 does this only more strictly
> ITestS3ATemporaryCredentials to cover all ddb metastore ops with session
> credentials
>
>
> Key: HADOOP-14592
> URL: https://issues.apache.org/jira/browse/HADOOP-14592
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Minor
>
> {{ITestS3ATemporaryCredentials}} tests a couple of operations with temp
> credentials, but for completeness it should perform all operations which will
> access dynamo DB, so verifying that temporary credentials work everywhere.
> I know this is implicit from anyone running the tests in an EC2 container and
> picking up the IAM details, so I'm not expecting the tests to fail —but they
> should be there
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.7
Thanks for working on this, Steve!! +1 non-binding - Built from source on 1.8.0_111-b14, mac - Verified checksum - Built non secure local cluster - Run yarn jobs like: pi,word count, DFSIO (read/write) - Performed basic HDFS operations - Basic sanity for NameNode and RM UI. Best, Ajay On 7/11/18, 8:05 AM, "Steve Loughran" wrote: Hi I've got RC0 of Hadoop 2.7.7 up for people to download and play with http://people.apache.org/~stevel/hadoop-2.7.7/RC0/ Nexus artifacts 2.7.7 are up in staging https://repository.apache.org/content/repositories/orgapachehadoop-1135 Git (signed) tag release-2.7.7-RC0, checksum c1aad84bd27cd79c3d1a7dd58202a8c3ee1ed3ac My current GPG key is 38237EE425050285077DB57AD22CF846DBB162A0 you can download this direct (and it MUST be direct) from the ASF HTTPS site https://dist.apache.org/repos/dist/release/hadoop/common/KEYS Please try the release and vote. The vote will run for 5 days. Thanks -Steve (I should add: this is my first ever attempt at a Hadoop release, please be (a) rigorous and (b) forgiving. Credit to Wei-Chiu Chuang for his advice). - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15599) Improve TrashPolicyDefault error message
Wei-Chiu Chuang created HADOOP-15599:
Summary: Improve TrashPolicyDefault error message
Key: HADOOP-15599
URL: https://issues.apache.org/jira/browse/HADOOP-15599
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Wei-Chiu Chuang
Found the following warning message:
{noformat}
2018-07-09 06:00:00,486 WARN org.apache.hadoop.fs.TrashPolicyDefault:
Unexpected item in trash: /user/hue/.Trash/hue. Ignoring.{noformat}
However, it's not clear what file in the trash directory that was unexpected.
Relevant code:
{code:java|title=TrashPolicyDefault#deleteCheckpoint}
try {
time = getTimeFromCheckpoint(name);
} catch (ParseException e) {
LOG.warn("Unexpected item in trash: "+dir+". Ignoring.");
continue;
}
{code}
File this Jira to get the file name recorded.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15600) Set default proxy user settings to non-routable IP addresses and default users group
Eric Yang created HADOOP-15600:
--
Summary: Set default proxy user settings to non-routable IP
addresses and default users group
Key: HADOOP-15600
URL: https://issues.apache.org/jira/browse/HADOOP-15600
Project: Hadoop Common
Issue Type: Sub-task
Components: security
Reporter: Eric Yang
The default setting to restrict the cluster nodes to communicate with peer
nodes are controlled by: hadoop.proxyuser.[hdfs.yarn].hosts, and
hadoop.proxyuser.[hdfs|yarn].groups. These settings are default to be opened
which allows any hosts to impersonate any user.
The proposal is to default settings to:
{code}
hadoop.proxyuser.hdfs.hosts
127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,169.254.0.0/16
hadoop.proxyuser.hdfs.groups
users
hadoop.proxyuser.yarn.hosts
127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,169.254.0.0/16
hadoop.proxyuser.yarn.groups
users
{code}
This will allow the cluster to default to a closed network and default "users"
group to reduce risks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Review request for HDFS-13654
Hi everyone, I recently submitted a patch for HttpFS security. Would anyone mind reviewing it? https://issues.apache.org/jira/browse/HDFS-13654 Thanks, -Takanobu - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15601) Change yarn.admin.acl setting to be more restricted
Eric Yang created HADOOP-15601: -- Summary: Change yarn.admin.acl setting to be more restricted Key: HADOOP-15601 URL: https://issues.apache.org/jira/browse/HADOOP-15601 Project: Hadoop Common Issue Type: Sub-task Components: security Reporter: Eric Yang Yarn.admin.acl is default to *, which means everyone is yarn administrator by default. It is probably better to default yarn.admin.acl to the user who runs yarn framework to prevent attacks. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
