Next week's Hadoop storage community call: Consistent Read from Standby

[Wei-Chiu Chuang]

Hi!

Consistent Read from Standby is one of the major features that's going to
land in the upcoming Hadoop 3.3.0 release.

I'm happy to announce that Konstantin graciously agreed to talk about the
production experience with this feature.

Please note that we will start this call at 11am pacific time, instead of
the 10am like before. I asked Konstantin to allocate an hour of time slot
to leave time for Q

Date/time:
March 18 11am pacific time, 6pm GMT

Please join via Zoom:
https://cloudera.zoom.us/j/880548968

Past meeting minutes:
https://docs.google.com/document/d/1jXM5Ujvf-zhcyw_5kiQVx6g-HeKe-YGnFS_1-qFXomI/edit

[jira] [Created] (HADOOP-16924) Update guava version to 28.1-jre

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HADOOP-16924:


 Summary: Update guava version to 28.1-jre
 Key: HADOOP-16924
 URL: https://issues.apache.org/jira/browse/HADOOP-16924
 Project: Hadoop Common
  Issue Type: Task
Affects Versions: 3.4.0
Reporter: Wei-Chiu Chuang






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-16776) backport HADOOP-16775: distcp copies to s3 are randomly corrupted

[Steve Loughran (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HADOOP-16776.
-
Resolution: Fixed

> backport HADOOP-16775: distcp copies to s3 are randomly corrupted
> -
>
> Key: HADOOP-16776
> URL: https://issues.apache.org/jira/browse/HADOOP-16776
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: tools/distcp
>Affects Versions: 2.8.0, 3.0.0, 2.10.0
>Reporter: Amir Shenavandeh
>Assignee: Amir Shenavandeh
>Priority: Blocker
>  Labels: DistCp
> Fix For: 3.1.4, 2.10.1
>
> Attachments: HADOOP-16776-branch-2.8-001.patch, 
> HADOOP-16776-branch-2.8-002.patch
>
>
> This is to back port HADOOP-16775 to hadoop 2.8 branch.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Reopened] (HADOOP-16776) backport HADOOP-16775: distcp copies to s3 are randomly corrupted

[Steve Loughran (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran reopened HADOOP-16776:
-
  Assignee: Amir Shenavandeh

> backport HADOOP-16775: distcp copies to s3 are randomly corrupted
> -
>
> Key: HADOOP-16776
> URL: https://issues.apache.org/jira/browse/HADOOP-16776
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: tools/distcp
>Affects Versions: 2.8.0, 3.0.0, 2.10.0
>Reporter: Amir Shenavandeh
>Assignee: Amir Shenavandeh
>Priority: Blocker
>  Labels: DistCp
> Fix For: 3.1.4, 2.10.1
>
> Attachments: HADOOP-16776-branch-2.8-001.patch, 
> HADOOP-16776-branch-2.8-002.patch
>
>
> This is to back port HADOOP-16775 to hadoop 2.8 branch.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-16885) Encryption zone file copy failure leaks temp file ._COPYING_ and wrapped stream

[Wei-Chiu Chuang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang resolved HADOOP-16885.
--
Resolution: Fixed

> Encryption zone file copy failure leaks temp file ._COPYING_ and wrapped 
> stream
> ---
>
> Key: HADOOP-16885
> URL: https://issues.apache.org/jira/browse/HADOOP-16885
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
>Priority: Major
> Fix For: 3.3.0
>
>
> Copy file into encryption on  trunk with HADOOP-16490 caused a leaking temp 
> file _COPYING_ left and potential wrapped stream unclosed. This ticked is 
> opened to track the fix for it. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Apache Hadoop Ozone 0.5.0-beta RC1

[Arpit Agarwal]

HDDS-3116 is now fixed in the ozone-0.5.0 branch.

Folks - any more potential blockers before Dinesh spins RC2? I don’t see 
anything in Jira at the moment:

https://issues.apache.org/jira/issues/?jql=project%20in%20(%22HDDS%22)%20and%20%22Target%20Version%2Fs%22%20in%20(0.5.0)%20and%20resolution%20in%20(Unresolved)%20and%20priority%20in%20(Blocker)
 


Thanks,
Arpit


> On Mar 9, 2020, at 6:15 PM, Shashikant Banerjee 
>  wrote:
> 
> I think https://issues.apache.org/jira/browse/HDDS-3116 
>  is a blocker for
> the release. Because of this, datanodes fail to communicate with SCM and
> marked dead and don't seem to recover.
> This has been observed in multiple test setups.
> 
> Thanks
> Shashi
> 
> On Mon, Mar 9, 2020 at 9:20 PM Attila Doroszlai  >
> wrote:
> 
>> +1
>> 
>> * Verified GPG signature and SHA512 checksum
>> * Compiled sources
>> * Ran ozone smoke test against both binary and locally compiled versions
>> 
>> Thanks Dinesh for RC1.
>> 
>> -Attila
>> 
>> On Sun, Mar 8, 2020 at 2:34 AM Arpit Agarwal
>>  wrote:
>>> 
>>> +1 (binding)
>>> Verified mds, sha512
>>> Verified signatures
>>> Built from source
>>> Deployed to 3 node cluster
>>> Tried a few ozone shell and filesystem commands
>>> Ran freon load generator
>>> Thanks Dinesh for putting the RC1 together.
>>> 
>>> 
>>> 
 On Mar 6, 2020, at 4:46 PM, Dinesh Chitlangia 
>> wrote:
 
 Hi Folks,
 
 We have put together RC1 for Apache Hadoop Ozone 0.5.0-beta.
 
 The RC artifacts are at:
 https://home.apache.org/~dineshc/ozone-0.5.0-rc1/
 
 The public key used for signing the artifacts can be found at:
 https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
 
 The maven artifacts are staged at:
 
>> https://repository.apache.org/content/repositories/orgapachehadoop-1260
 
 The RC tag in git is at:
 https://github.com/apache/hadoop-ozone/tree/ozone-0.5.0-beta-RC1
 
 This release contains 800+ fixes/improvements [1].
 Thanks to everyone who put in the effort to make this happen.
 
 *The vote will run for 7 days, ending on March 13th 2020 at 11:59 pm
>> PST.*
 
 Note: This release is beta quality, it’s not recommended to use in
 production but we believe that it’s stable enough to try out the
>> feature
 set and collect feedback.
 
 
 [1] https://s.apache.org/ozone-0.5.0-fixed-issues
 
 Thanks,
 Dinesh Chitlangia
>>> 
>> 
>> -
>> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org 
>> 
>> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org 
>> 

[jira] [Created] (HADOOP-16923) mkdir on s3a should not be sensitive to trailing '/'

[Zoltan Haindrich (Jira)]

Zoltan Haindrich created HADOOP-16923:
-

 Summary: mkdir on s3a should not be sensitive to trailing '/'
 Key: HADOOP-16923
 URL: https://issues.apache.org/jira/browse/HADOOP-16923
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/s3
Reporter: Zoltan Haindrich


I would have expected to create the directory for both calls:

{code}
[hive@hiveserver2-0 lib]$ hdfs dfs -mkdir 
s3a://qe-s3-bucket-mst-xfpn-dwx-external/custom-jars2/
/usr/bin/hdfs: line 4: /usr/lib/bigtop-utils/bigtop-detect-javahome: No such 
file or directory
log4j:WARN No appenders could be found for logger 
(org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
info.
mkdir: get on s3a://qe-s3-bucket-mst-xfpn-dwx-external/custom-jars2/: 
com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: One or more 
parameter values were invalid: An AttributeValue may not contain an empty 
string (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: 
ValidationException; Request ID: 
1AE0KA2Q5ADI47R75N8BDJE973VV4KQNSO5AEMVJF66Q9ASUAAJG): One or more parameter 
values were invalid: An AttributeValue may not contain an empty string 
(Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException; 
Request ID: 1AE0KA2Q5ADI47R75N8BDJE973VV4KQNSO5AEMVJF66Q9ASUAAJG)
[hive@hiveserver2-0 lib]$ hdfs dfs -mkdir 
s3a://qe-s3-bucket-mst-xfpn-dwx-external/custom-jars2
/usr/bin/hdfs: line 4: /usr/lib/bigtop-utils/bigtop-detect-javahome: No such 
file or directory
log4j:WARN No appenders could be found for logger 
(org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
info.
[hive@hiveserver2-0 lib]$ 
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1437/

[Mar 12, 2020 9:33:12 AM] (sunilg) YARN-10191. FS-CS converter: call 
System.exit function call for every
[Mar 12, 2020 11:18:37 AM] (snemeth) YARN-10193. FS-CS converter: fix incorrect 
capacity conversion.
[Mar 12, 2020 11:29:03 AM] (snemeth) YARN-9997. Code cleanup in 
ZKConfigurationStore. Contributed by Andras
[Mar 12, 2020 1:29:17 PM] (surendralilhore) HDFS-14442. Disagreement between 
HAUtil.getAddressOfActive and
[Mar 12, 2020 2:13:55 PM] (github) HADOOP-15430. hadoop fs -mkdir -p 
path-ending-with-slash/ fails with
[Mar 13, 2020 12:22:11 AM] (inigoiri) HDFS-14612. SlowDiskReport won't update 
when SlowDisks is always empty




-1 overall


The following subsystems voted -1:
asflicense findbugs pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml
 

FindBugs :

   module:hadoop-cloud-storage-project/hadoop-cos 
   Redundant nullcheck of dir, which is known to be non-null in 
org.apache.hadoop.fs.cosn.BufferPool.createDir(String) Redundant null check at 
BufferPool.java:is known to be non-null in 
org.apache.hadoop.fs.cosn.BufferPool.createDir(String) Redundant null check at 
BufferPool.java:[line 66] 
   org.apache.hadoop.fs.cosn.CosNInputStream$ReadBuffer.getBuffer() may 
expose internal representation by returning CosNInputStream$ReadBuffer.buffer 
At CosNInputStream.java:by returning CosNInputStream$ReadBuffer.buffer At 
CosNInputStream.java:[line 87] 
   Found reliance on default encoding in 
org.apache.hadoop.fs.cosn.CosNativeFileSystemStore.storeFile(String, File, 
byte[]):in org.apache.hadoop.fs.cosn.CosNativeFileSystemStore.storeFile(String, 
File, byte[]): new String(byte[]) At CosNativeFileSystemStore.java:[line 199] 
   Found reliance on default encoding in 
org.apache.hadoop.fs.cosn.CosNativeFileSystemStore.storeFileWithRetry(String, 
InputStream, byte[], long):in 
org.apache.hadoop.fs.cosn.CosNativeFileSystemStore.storeFileWithRetry(String, 
InputStream, byte[], long): new String(byte[]) At 
CosNativeFileSystemStore.java:[line 178] 
   org.apache.hadoop.fs.cosn.CosNativeFileSystemStore.uploadPart(File, 
String, String, int) may fail to clean up java.io.InputStream Obligation to 
clean up resource created at CosNativeFileSystemStore.java:fail to clean up 
java.io.InputStream Obligation to clean up resource created at 
CosNativeFileSystemStore.java:[line 252] is not discharged 

Failed junit tests :

   hadoop.hdfs.server.balancer.TestBalancer 
   hadoop.hdfs.server.namenode.TestNameNodeMXBean 
   hadoop.yarn.applications.distributedshell.TestDistributedShell 
   hadoop.yarn.sls.TestSLSStreamAMSynth 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1437/artifact/out/diff-compile-cc-root.txt
  [8.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1437/artifact/out/diff-compile-javac-root.txt
  [424K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1437/artifact/out/diff-checkstyle-root.txt
  [16M]

   pathlen:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1437/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1437/artifact/out/diff-patch-pylint.txt
  [24K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1437/artifact/out/diff-patch-shellcheck.txt
  [16K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1437/artifact/out/diff-patch-shelldocs.txt
  [44K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1437/artifact/out/whitespace-eol.txt
  [9.9M]
   

Apache Hadoop qbt Report: branch2.10+JDK7 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86/623/

[Mar 12, 2020 3:49:38 AM] (iwasakims) HDFS-15077. Fix intermittent failure of




-1 overall


The following subsystems voted -1:
asflicense findbugs hadolint pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/empty-configuration.xml
 
   hadoop-tools/hadoop-azure/src/config/checkstyle-suppressions.xml 
   hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/public/crossdomain.xml 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml
 

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client
 
   Boxed value is unboxed and then immediately reboxed in 
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result,
 byte[], byte[], KeyConverter, ValueConverter, boolean) At 
ColumnRWHelper.java:then immediately reboxed in 
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result,
 byte[], byte[], KeyConverter, ValueConverter, boolean) At 
ColumnRWHelper.java:[line 335] 

Failed junit tests :

   hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys 
   hadoop.hdfs.server.datanode.TestFsDatasetCache 
   hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   hadoop.registry.secure.TestSecureLogins 
   hadoop.yarn.client.api.impl.TestAMRMProxy 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86/623/artifact/out/diff-compile-cc-root-jdk1.7.0_95.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86/623/artifact/out/diff-compile-javac-root-jdk1.7.0_95.txt
  [328K]

   cc:

   
https://builds.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86/623/artifact/out/diff-compile-cc-root-jdk1.8.0_242.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86/623/artifact/out/diff-compile-javac-root-jdk1.8.0_242.txt
  [308K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86/623/artifact/out/diff-checkstyle-root.txt
  [16M]

   hadolint:

   
https://builds.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86/623/artifact/out/diff-patch-hadolint.txt
  [4.0K]

   pathlen:

   
https://builds.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86/623/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86/623/artifact/out/diff-patch-pylint.txt
  [24K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86/623/artifact/out/diff-patch-shellcheck.txt
  [56K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86/623/artifact/out/diff-patch-shelldocs.txt
  [8.0K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86/623/artifact/out/whitespace-eol.txt
  [12M]
   
https://builds.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86/623/artifact/out/whitespace-tabs.txt
  [1.3M]

   xml:

   
https://builds.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86/623/artifact/out/xml.txt
  [12K]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86/623/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-client-warnings.html
  [8.0K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86/623/artifact/out/diff-javadoc-javadoc-root-jdk1.7.0_95.txt
  [16K]
   
https://builds.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86/623/artifact/out/diff-javadoc-javadoc-root-jdk1.8.0_242.txt
  [1.1M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86/623/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [284K]
   
https://builds.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86/623/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86/623/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-registry.txt
  [12K]