date:20240429

[jira] [Resolved] (HADOOP-19151) Support configurable SASL mechanism

2024-04-29 Thread Tsz-wo Sze (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-19151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsz-wo Sze resolved HADOOP-19151.
-
Fix Version/s: 3.5.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

The pull request is now merged.

> Support configurable SASL mechanism
> ---
>
> Key: HADOOP-19151
> URL: https://issues.apache.org/jira/browse/HADOOP-19151
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: security
>Reporter: Tsz-wo Sze
>Assignee: Tsz-wo Sze
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.5.0
>
>
> Currently, the SASL mechanism is hard coded to DIGEST-MD5.  As mentioned in 
> HADOOP-14811, DIGEST-MD5 is known to be insecure; see 
> [rfc6331|https://datatracker.ietf.org/doc/html/rfc6331].
> In this JIRA, we will make the SASL mechanism configurable.  The default 
> mechanism will still be DIGEST-MD5 in order to maintain compatibility.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-19150) Test ITestAbfsRestOperationException#testAuthFailException is broken.

2024-04-29 Thread Mukund Thakur (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-19150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mukund Thakur resolved HADOOP-19150.

Fix Version/s: 3.4.1
   Resolution: Fixed

> Test ITestAbfsRestOperationException#testAuthFailException is broken. 
> --
>
> Key: HADOOP-19150
> URL: https://issues.apache.org/jira/browse/HADOOP-19150
> Project: Hadoop Common
>  Issue Type: Sub-task
>Reporter: Mukund Thakur
>Assignee: Anuj Modi
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.1
>
>
> {code:java}
> intercept(Exception.class,
> () -> {
>   fs.getFileStatus(new Path("/"));
> }); {code}
> Intercept shouldn't be used as there are assertions in catch statements. 
>  
> CC [~ste...@apache.org]  [~anujmodi2021] [~asrani_anmol] 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-19159) Fix hadoop-aws document for fs.s3a.committer.abort.pending.uploads

2024-04-29 Thread Steve Loughran (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-19159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HADOOP-19159.
-
Fix Version/s: 3.3.9
   3.5.0
   3.4.1
   Resolution: Fixed

> Fix hadoop-aws document for fs.s3a.committer.abort.pending.uploads
> --
>
> Key: HADOOP-19159
> URL: https://issues.apache.org/jira/browse/HADOOP-19159
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: documentation
>Reporter: Xi Chen
>Assignee: Xi Chen
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.3.9, 3.5.0, 3.4.1
>
>
> The description about `fs.s3a.committer.abort.pending.uploads` in the 
> _Concurrent Jobs writing to the same destination_ is not all correct.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Hadoop Windows Build

2024-04-29 Thread Ayush Saxena

Thanx Chris, that would be great

-Ayush

On Mon, 29 Apr 2024 at 19:07, Chris Thistlethwaite 
wrote:

> I'm following along on lists.a.o. I can cancel all the Windows jobs in
> queue, we have a groovy script for that.
>
> -Chris T.
> #asfinfra
>
> On 2024/04/28 17:35:21 Gautham Banasandra wrote:
> > Yeah, I just noticed that. May I know how I can abort all the jobs at
> once? I only saw that I
> > can cancel the jobs one-by-one.
> >
> > Thanks,
> > --Gautham
> >
> > On 2024/04/28 15:19:13 Ayush Saxena wrote:
> > > Thanx Gautham for chasing this.
> > >
> > > I think there are still some 119 in the build queue, if you see on the
> left
> > > here [1](Search for Build Queue). They are all stuck on "Waiting for
> next
> > > available executor on Windows"
> > >
> > > If you aborted all previously & they showed up now again, then
> something is
> > > still messed up with the configurations that the pipeline is getting
> > > triggered for the existing PR (not new), if you didn't abort earlier
> then
> > > maybe you need to abort all the ones in queue and free up the
> resources.
> > >
> > > One example of build waiting (as of now) for resource since past 7
> hours [2]
> > >
> > > Let me know if you are stuck, we can together get things figured out
> :-)
> > >
> > > -Ayush
> > >
> > >
> > > [1]
> > >
> https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/view/change-requests/builds
> > > [2]
> > >
> https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/view/change-requests/job/PR-6423/2/console
> > >
> > > On Sun, 28 Apr 2024 at 13:43, Gautham Banasandra 
> wrote:
> > >
> > > > Hi folks,
> > > >
> > > > I apologize for the inconvenience caused. I've now applied the
> mitigation
> > > > described in [3].
> > > >
> > > > Unfortunately, there are only 12 Windows nodes in the whole swarm of
> > > > Jenkins build nodes.
> > > > Thus, this caused a starvation of the Windows nodes for other
> projects.
> > > >
> > > > I had reached out to the infra team several months ago and requested
> them
> > > > to add more
> > > > Windows nodes, but it was turned down. I'm not sure if there's a way
> > > > around this, other than
> > > > getting more Windows nodes.
> > > >
> > > > Thanks,
> > > > --Gautham
> > > >
> > > > On 2024/04/28 04:53:32 Ayush Saxena wrote:
> > > > > Found this on dev@hadoop -> Moving to common-dev (the ML we use)
> > > > >
> > > > > I think there was some initiative to enable Windows Pre-Commit for
> every
> > > > PR
> > > > > and that seems to have gone wild, either the number of PRs raised
> are way
> > > > > more than the capacity the nodes can handle or something got
> > > > misconfigured
> > > > > in the job itself that the build is getting triggered for all the
> open PR
> > > > > not just new, which is leading to starvation of resources.
> > > > >
> > > > > To the best of my knowledge
> > > > > @Gautham Banasandra  / @Iñigo Goiri <
> > > > elgo...@gmail.com> are
> > > > > chasing the initiative, can you folks help check?
> > > > >
> > > > > There are concerns raised by the Infra team here [1] on dev@hadoop
> > > > >
> > > > > Most probably something messed up while configuring the
> > > > > hadoop-multibranch-windows job, it shows some 613 PR scheduled
> [2], I
> > > > think
> > > > > it scheduled for all open ones, something similar happened
> long-long ago
> > > > > when we were doing migrations, can fetch pointers from [3]
> > > > >
> > > > > [1]
> https://lists.apache.org/thread/7nsyd0vtpb87fhm0fpv8frh6dzk3b3tl
> > > > > [2]
> > > > >
> > > >
> https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/view/change-requests/builds
> > > > > [3]
> https://lists.apache.org/thread/8pxf2yon3r9g61zgv9cf120qnhrs8q23
> > > > >
> > > > > -Ayush
> > > > >
> > > > >
> > > > > On 2024/04/26 16:59:04 Wei-Chiu Chuang wrote:
> > > > > > I'm not familiar with Windows build. But you may have better luck
> > > > reaching
> > > > > > out to Apache Infra
> > > > > > https://infra.apache.org/contact.html
> > > > > >
> > > > > > mailing list, jira or even slack
> > > > > >
> > > > > > On Fri, Apr 26, 2024 at 9:42 AM Cesar Hernandez <
> cesargu...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hello,
> > > > > > > An option that can be implemented in the Hadoop pipeline [1]
> is to
> > > > set a
> > > > > > > timeout [2] on critical stages within the pipelines, for
> example in
> > > > > > > "Windows 10" stage .
> > > > > > > As for the issue the Ci build is logging [3] in the
> > > > hadoop-multibranch
> > > > > jobs
> > > > > > > reported by Chris, it seems the issue is around the Post
> (cleanup)
> > > > > pipeline
> > > > > > > process. My two cents is to use cleanWs() instead of
> deleteDir() as
> > > > > > > documented in: https://plugins.jenkins.io/ws-cleanup/
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > > >
> > > > >
> > > >
> https://github.com/apache/hadoop/blob/trunk/dev-support/jenkinsfile-windows-10
> > > > > > >
> > > > > > > [2]
> > > > > > >
> > > > > > >
> >

Re: Hadoop Windows Build

2024-04-29 Thread Chris Thistlethwaite

I'm following along on lists.a.o. I can cancel all the Windows jobs in queue, 
we have a groovy script for that.

-Chris T.
#asfinfra

On 2024/04/28 17:35:21 Gautham Banasandra wrote:
> Yeah, I just noticed that. May I know how I can abort all the jobs at once? I 
> only saw that I
> can cancel the jobs one-by-one.
> 
> Thanks,
> --Gautham
> 
> On 2024/04/28 15:19:13 Ayush Saxena wrote:
> > Thanx Gautham for chasing this.
> > 
> > I think there are still some 119 in the build queue, if you see on the left
> > here [1](Search for Build Queue). They are all stuck on "Waiting for next
> > available executor on Windows"
> > 
> > If you aborted all previously & they showed up now again, then something is
> > still messed up with the configurations that the pipeline is getting
> > triggered for the existing PR (not new), if you didn't abort earlier then
> > maybe you need to abort all the ones in queue and free up the resources.
> > 
> > One example of build waiting (as of now) for resource since past 7 hours [2]
> > 
> > Let me know if you are stuck, we can together get things figured out :-)
> > 
> > -Ayush
> > 
> > 
> > [1]
> > https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/view/change-requests/builds
> > [2]
> > https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/view/change-requests/job/PR-6423/2/console
> > 
> > On Sun, 28 Apr 2024 at 13:43, Gautham Banasandra  wrote:
> > 
> > > Hi folks,
> > >
> > > I apologize for the inconvenience caused. I've now applied the mitigation
> > > described in [3].
> > >
> > > Unfortunately, there are only 12 Windows nodes in the whole swarm of
> > > Jenkins build nodes.
> > > Thus, this caused a starvation of the Windows nodes for other projects.
> > >
> > > I had reached out to the infra team several months ago and requested them
> > > to add more
> > > Windows nodes, but it was turned down. I'm not sure if there's a way
> > > around this, other than
> > > getting more Windows nodes.
> > >
> > > Thanks,
> > > --Gautham
> > >
> > > On 2024/04/28 04:53:32 Ayush Saxena wrote:
> > > > Found this on dev@hadoop -> Moving to common-dev (the ML we use)
> > > >
> > > > I think there was some initiative to enable Windows Pre-Commit for every
> > > PR
> > > > and that seems to have gone wild, either the number of PRs raised are 
> > > > way
> > > > more than the capacity the nodes can handle or something got
> > > misconfigured
> > > > in the job itself that the build is getting triggered for all the open 
> > > > PR
> > > > not just new, which is leading to starvation of resources.
> > > >
> > > > To the best of my knowledge
> > > > @Gautham Banasandra  / @Iñigo Goiri <
> > > elgo...@gmail.com> are
> > > > chasing the initiative, can you folks help check?
> > > >
> > > > There are concerns raised by the Infra team here [1] on dev@hadoop
> > > >
> > > > Most probably something messed up while configuring the
> > > > hadoop-multibranch-windows job, it shows some 613 PR scheduled [2], I
> > > think
> > > > it scheduled for all open ones, something similar happened long-long ago
> > > > when we were doing migrations, can fetch pointers from [3]
> > > >
> > > > [1] https://lists.apache.org/thread/7nsyd0vtpb87fhm0fpv8frh6dzk3b3tl
> > > > [2]
> > > >
> > > https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/view/change-requests/builds
> > > > [3] https://lists.apache.org/thread/8pxf2yon3r9g61zgv9cf120qnhrs8q23
> > > >
> > > > -Ayush
> > > >
> > > >
> > > > On 2024/04/26 16:59:04 Wei-Chiu Chuang wrote:
> > > > > I'm not familiar with Windows build. But you may have better luck
> > > reaching
> > > > > out to Apache Infra
> > > > > https://infra.apache.org/contact.html
> > > > >
> > > > > mailing list, jira or even slack
> > > > >
> > > > > On Fri, Apr 26, 2024 at 9:42 AM Cesar Hernandez 
> > > > > wrote:
> > > > >
> > > > > > Hello,
> > > > > > An option that can be implemented in the Hadoop pipeline [1] is to
> > > set a
> > > > > > timeout [2] on critical stages within the pipelines, for example in
> > > > > > "Windows 10" stage .
> > > > > > As for the issue the Ci build is logging [3] in the
> > > hadoop-multibranch
> > > > jobs
> > > > > > reported by Chris, it seems the issue is around the Post (cleanup)
> > > > pipeline
> > > > > > process. My two cents is to use cleanWs() instead of deleteDir() as
> > > > > > documented in: https://plugins.jenkins.io/ws-cleanup/
> > > > > >
> > > > > > [1]
> > > > > >
> > > > > >
> > > >
> > > https://github.com/apache/hadoop/blob/trunk/dev-support/jenkinsfile-windows-10
> > > > > >
> > > > > > [2]
> > > > > >
> > > > > >
> > > >
> > > https://www.jenkins.io/doc/pipeline/steps/workflow-basic-steps/#timeout-enforce-time-limit
> > > > > >
> > > > > > [3]
> > > > > >
> > > > > > Still waiting to schedule task
> > > > > > Waiting for next available executor on ‘Windows
> > > > > > ’[Pipeline] //
> > > > > > node[Pipeline] stage
> > > > > > <
> > > > > >
> > > >
> > >

Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64

2024-04-29 Thread Apache Jenkins Server

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1377/

No changes




-1 overall


The following subsystems voted -1:
asflicense hadolint mvnsite pathlen unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.fs.TestFileUtil 
   hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate 
   
hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain 
   hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion 
   hadoop.hdfs.TestFileLengthOnClusterRestart 
   hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap 
   hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys 
   hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes 
   hadoop.hdfs.TestMultipleNNPortQOP 
   hadoop.hdfs.server.federation.router.TestRouterQuota 
   hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat 
   hadoop.hdfs.server.federation.resolver.order.TestLocalResolver 
   hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver 
   hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   hadoop.mapreduce.v2.app.TestRuntimeEstimators 
   hadoop.mapreduce.lib.input.TestLineRecordReader 
   hadoop.mapred.TestLineRecordReader 
   hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter 
   hadoop.resourceestimator.service.TestResourceEstimatorService 
   hadoop.resourceestimator.solver.impl.TestLpSolver 
   hadoop.yarn.sls.TestSLSRunner 
   
hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestNumaResourceAllocator
 
   
hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestNumaResourceHandlerImpl
 
   hadoop.yarn.server.resourcemanager.TestClientRMService 
   
hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker
 
  

   cc:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1377/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1377/artifact/out/diff-compile-javac-root.txt
  [488K]

   checkstyle:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1377/artifact/out/diff-checkstyle-root.txt
  [14M]

   hadolint:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1377/artifact/out/diff-patch-hadolint.txt
  [4.0K]

   mvnsite:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1377/artifact/out/patch-mvnsite-root.txt
  [568K]

   pathlen:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1377/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1377/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1377/artifact/out/diff-patch-shellcheck.txt
  [72K]

   whitespace:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1377/artifact/out/whitespace-eol.txt
  [12M]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1377/artifact/out/whitespace-tabs.txt
  [1.3M]

   javadoc:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1377/artifact/out/patch-javadoc-root.txt
  [36K]

   unit:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1377/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
  [220K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1377/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [460K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1377/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
  [36K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1377/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt
  [16K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1377/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt
  [44K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1377/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
  [104K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1377/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt
  [20K]

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64

2024-04-29 Thread Apache Jenkins Server

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1573/

[Apr 27, 2024, 12:30:21 PM] (github) HADOOP-19071. Update maven-surefire-plugin 
from 3.0.0 to 3.2.5. (#6664) Contributed by Shilun Fan.
[Apr 27, 2024, 12:36:11 PM] (github) HDFS-17367. Add PercentUsed for Different 
StorageTypes in JMX (#6735) Contributed by Hualong Zhang.

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-19151) Support configurable SASL mechanism

[jira] [Resolved] (HADOOP-19150) Test ITestAbfsRestOperationException#testAuthFailException is broken.

[jira] [Resolved] (HADOOP-19159) Fix hadoop-aws document for fs.s3a.committer.abort.pending.uploads

Re: Hadoop Windows Build

Re: Hadoop Windows Build

Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64

7 matches

Site Navigation

Mail list logo

Footer information