[jira] [Resolved] (HADOOP-17999) No-op implementation of setWriteChecksum and setVerifyChecksum in ViewFileSystem

2021-11-16 Thread Konstantin Shvachko (Jira) 


 [ 
https://issues.apache.org/jira/browse/HADOOP-17999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Shvachko resolved HADOOP-17999.
--
Fix Version/s: 3.4.0
   2.10.2
   3.2.4
   3.3.3
 Hadoop Flags: Reviewed
   Resolution: Fixed

I just committed this to all supported branches. Thank you [~abhishekd].

> No-op implementation of setWriteChecksum and setVerifyChecksum in 
> ViewFileSystem
> 
>
> Key: HADOOP-17999
> URL: https://issues.apache.org/jira/browse/HADOOP-17999
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Abhishek Das
>Assignee: Abhishek Das
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0, 2.10.2, 3.2.4, 3.3.3
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Currently setVerifyChecksum and setWriteChecksum initializes all target file 
> systems which causes delay in hadoop shell copy commands such as get, put, 
> copyFromLocal etc.
> This also eventually causes OOM.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [DISCUSS] Hadoop 3.3.2 release?

2021-11-16 Thread Chao Sun 
Hi, I just created a release branch branch-3.3.2 and updated the version of
branch-3.3 to 3.3.3-SNAPSHOT. Will try to get the RC1 ready soon.

On Thu, Oct 28, 2021 at 9:58 AM Chao Sun  wrote:

> Hi, sorry for the delay! I plan to start the release process within a few
> days.
>
> On Thu, Oct 28, 2021 at 3:25 AM Hui Fei  wrote:
>
>> Thank you, Chao!
>> Do you need any help? I can assist you.
>>
>> Chao Sun  于2021年9月8日周三 上午1:06写道：
>>
>>> Hi all,
>>>
>>> It has been almost 3 months since the 3.3.1 release and branch-3.3 has
>>> accumulated quite a few commits (118 atm). In particular, Spark community
>>> recently found an issue which prevents one from using the shaded Hadoop
>>> client together with certain compression codecs such as lz4 and snappy
>>> codec. The details are recorded in HADOOP-17891 and SPARK-36669.
>>>
>>> Therefore, I'm wondering if anyone is also interested in a 3.3.2 release.
>>> If there is no objection, I'd like to volunteer myself for the work as
>>> well.
>>>
>>> Best Regards,
>>> Chao
>>>
>>

Re: Tracing the adding of Hadoop CI's endpoint

2021-11-16 Thread Akira Ajisaka 
(Added common-dev to CC and moved general to BCC because this is a
development topic)

Hi Gautham,

Yes, your assumption is correct. Every Jenkins job has a BUILD_URL
environment variable [1], and it is used in Yetus [2].

[1] https://wiki.jenkins.io/display/JENKINS/Building+a+software+project
[2]
https://github.com/apache/yetus/blob/477eeecec11d18b3f4e10bb25b1b5987970f2deb/precommit/src/main/shell/test-patch.sh#L714

Thanks,
Akira

On Tue, Nov 16, 2021 at 3:55 AM Gautham Banasandra 
wrote:

> Hi Hadoop devs,
>
> Any ideas on this? I'm guessing there's some environment variable that
> specifies the Hadoop CI's endpoint which Yetus reads. I'm unable to find it
> though. I would really appreciate any help regarding this.
>
> Thanks,
> --Gautham
>
> On Fri, 12 Nov 2021 at 21:38, Gautham Banasandra 
> wrote:
>
> > Hi Hadoop devs,
> >
> > At the end of the Yetus's pre-commit run, we see the following table -
> >
> > || Subsystem || Report/Notes ||
> >
> >
> 
> > |  cc  |
> https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3650/1/artifact/out/results-compile-cc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt
> > |
> > |  unit  |
> https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3650/1/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
> > |
> >
> > May I know how the Hadoop CI's endpoint gets specified to Yetus?
> >
> > Please see this JIRA to get some context on what I'm trying to do with
> > this - [HADOOP-18007] Use platform specific endpoints for CI report - ASF
> > JIRA (apache.org) 
> >
> > Thanks,
> > --Gautham
> >
>

[jira] [Resolved] (HADOOP-18002) abfs rename idempotency broken -remove recovery

2021-11-16 Thread Mukund Thakur (Jira) 


 [ 
https://issues.apache.org/jira/browse/HADOOP-18002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mukund Thakur resolved HADOOP-18002.

Resolution: Fixed

> abfs rename idempotency broken -remove recovery
> ---
>
> Key: HADOOP-18002
> URL: https://issues.apache.org/jira/browse/HADOOP-18002
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/azure
>Affects Versions: 3.3.2
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> rename idempotency logic of HADOOP-17105   is broken as modtimes aren't 
> uodated on rename.
> remove, with the changes from the PR of HADOOP-17981.
> also fix delete recovery test after HADOOP-17934



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64

2021-11-16 Thread Apache Jenkins Server 
For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/483/

No changes




-1 overall


The following subsystems voted -1:
asflicense hadolint mvnsite pathlen unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.fs.TestFileUtil 
   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA 
   hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys 
   hadoop.hdfs.TestMultipleNNPortQOP 
   
hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain 
   hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat 
   hadoop.hdfs.server.federation.router.TestRouterQuota 
   hadoop.hdfs.server.federation.resolver.order.TestLocalResolver 
   hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver 
   hadoop.yarn.server.nodemanager.amrmproxy.TestFederationInterceptor 
   
hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker
 
   hadoop.yarn.server.resourcemanager.TestClientRMService 
   hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter 
   hadoop.mapreduce.lib.input.TestLineRecordReader 
   hadoop.mapred.TestLineRecordReader 
   hadoop.yarn.sls.TestSLSRunner 
   hadoop.resourceestimator.service.TestResourceEstimatorService 
   hadoop.resourceestimator.solver.impl.TestLpSolver 
  

   cc:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/483/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/483/artifact/out/diff-compile-javac-root.txt
  [496K]

   checkstyle:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/483/artifact/out/diff-checkstyle-root.txt
  [14M]

   hadolint:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/483/artifact/out/diff-patch-hadolint.txt
  [4.0K]

   mvnsite:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/483/artifact/out/patch-mvnsite-root.txt
  [584K]

   pathlen:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/483/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/483/artifact/out/diff-patch-pylint.txt
  [48K]

   shellcheck:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/483/artifact/out/diff-patch-shellcheck.txt
  [56K]

   shelldocs:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/483/artifact/out/diff-patch-shelldocs.txt
  [48K]

   whitespace:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/483/artifact/out/whitespace-eol.txt
  [12M]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/483/artifact/out/whitespace-tabs.txt
  [1.3M]

   javadoc:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/483/artifact/out/patch-javadoc-root.txt
  [32K]

   unit:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/483/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
  [232K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/483/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [432K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/483/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt
  [12K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/483/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
  [40K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/483/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
  [20K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/483/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [60K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/483/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [128K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/483/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
  [104K]