date:20220105

Re: Possibility of using ci-hadoop.a.o for Nutch integration tests

2022-01-05 Thread lewis john mcgibbney

Thank you for the information. I agree with you. I totally misunderstood
what ci-hadoop.a.o was.
Sorry for the noise.
lewismc

On Wed, Jan 5, 2022 at 21:56 Akira Ajisaka  wrote:

> (Adding builds@)
>
> Hi Lewis,
>
> Nutch is already using ci-builds.apache.org, so I think Nutch can
> continue using it. ci-hadoop.apache.org provides almost the same
> functionality as ci-builds.apache.org and there is no non-production
> Hadoop cluster running there. Therefore moving to ci-hadoop does not make
> sense.
>
> Short history: In the past there were some jenkins hosts that were labeled
> for Hadoop and its related projects. After the migration to cloudbees, the
> labeled hosts are moved under ci-hadoop.apache.org.
>
> Thanks,
> Akira
>
>
> On Thu, Jan 6, 2022 at 2:20 PM lewis john mcgibbney 
> wrote:
>
>> Thank you for the response and for directing the conversation to the
>> correct places.
>> I may have misunderstood what ci-hadoop.apache.org actually is. We are
>> looking for a non-production Hadoop cluster which we can use to simulate
>> Nutch jobs. I am not sure if this is what ci-hadoop.apache.org actually
>> is...
>> Instead it looks like lots of compute resources used to perform Jenkins
>> CI/CD tasks for Hadoop and associated projects rather than test things
>> on-top of Hadoop (and associated projects).
>> Any clarity on what ci-hadoop.apache.org actually is would be greatly
>> appreciated.
>>
>> Let me also clarify my language, rather than have the integration tests
>> run
>> on every PR, we could trigger the integration tests to be run by tagging a
>> Github bot i.e., "@nutchbot integration-test". Similar to what is done
>> with
>> Dependabot or conda-forge for anyon familiar with those mechanisms.
>>
>> Thanks for any advice or comments.
>> lewismc
>>
>> On Wed, Jan 5, 2022 at 9:05 PM Ayush Saxena  wrote:
>>
>> > Moved to Dev lists.
>> >
>> > Not sure about this though:
>> >  when a PR is submitted to Nutch project it will run some MR job in
>> Hadoop
>> > CI.
>> >
>> > Whatever that PR requires should run as part of Nutch Infra. Why in
>> Hadoop
>> > CI?
>> > Our CI is already loaded with our own workloads.
>> > If by any chance the above assertion gets a pass, then secondly we have
>> > very less number of people managing work related to CI and Infra. I
>> don’t
>> > think most of the people won’t have context or say in the Nutch project,
>> > neither bandwidth to fix stuff if it gets broken.
>> >
>> > Just my thoughts. Looped in the dev lists, if others have any feedback.
>> As
>> > for the process, this would require a consensus from the Hadoop PMC
>> >
>> > -Ayush
>> >
>> > > On 06-Jan-2022, at 7:02 AM, lewis john mcgibbney 
>> > wrote:
>> > >
>> > > Hi general@,
>> > >
>> > > Not sure if this is the correct mailing list. Please redirect me if
>> there
>> > > is a more suitable location. Thank you
>> > >
>> > > I am PMC over on the Nutch project (https://nutch.apache.org). I
>> would
>> > like
>> > > to investigate whether we can build an integration testing capability
>> for
>> > > the project. This would involve running a Nutch integration test suite
>> > > (collection of MR jobs) in a Hadoop CI environment. For example
>> whenever
>> > a
>> > > pull request is submitted to the Nutch project. This could easily be
>> > > automated through Jenkins.
>> > >
>> > > I’m not sure if this is something the Hadoop PMC would consider. Thank
>> > you
>> > > for the consideration.
>> > >
>> > > lewismc
>> > > --
>> > > http://home.apache.org/~lewismc/
>> > > http://people.apache.org/keys/committer/lewismc
>> >
>>
>>
>> --
>> http://home.apache.org/~lewismc/
>> http://people.apache.org/keys/committer/lewismc
>>
> --
http://home.apache.org/~lewismc/
http://people.apache.org/keys/committer/lewismc

Re: Next Mandarin Hadoop Online Meetup Jan 6th.

2022-01-05 Thread Wei-Chiu Chuang

Just a gentle reminder this is happening now.

On Mon, Jan 3, 2022 at 5:39 PM Wei-Chiu Chuang  wrote:

> Hello community,
>
> This week we'll going to have Tao Li (tomscut) speaking about  the
> experience of operating HDFS at BIGO. See you on Thursday!
>
> 题目：《HDFS在BIGO的实践》
> 概要：HDFS作为大数据底层存储服务，在BIGO的发展中起到了非常重要的作用。随着业务的发展和数据的爆发式增长，HDFS单个集群的瓶颈愈发凸显，我们借助Router，将多个HDFS集群整合成一个Namespace，增强集群的扩展能力；改造Router，使其支持Alluxio和自定义策略，并开启HDFS
> EC，实现热、温、冷数据分层存储。同时，通过对HDFS集群慢节点和慢盘的处理，提升了HDFS的读写性能。本次分享主要讲述BIGO对Router的实践经验，以及对慢节点和慢盘的处理经验。
> 关键词：Router，Slow Node，Slow Disk
> 演讲者：Tao Li (Apache id: tomscut)
>
> Date/Time: Jan 6 2PM Beijing Time.
>
> Zoom link: https://cloudera.zoom.us/j/97264903288
>
> One tap mobile
>
> +16465588656,,880548968# US (New York)
>
> +17207072699,,880548968# US
>
> Download Center 
>
> Dial by your location
>
> +1 646 558 8656 US (New York)
>
> +1 720 707 2699 US
>
> 877 853 5257 US Toll-free
>
> 888 475 4499 US Toll-free
>
> Meeting ID: 972 6490 3288
> Find your local number: https://zoom.us/u/acaGRDfMVl
>

Re: Possibility of using ci-hadoop.a.o for Nutch integration tests

2022-01-05 Thread Akira Ajisaka

(Adding builds@)

Hi Lewis,

Nutch is already using ci-builds.apache.org, so I think Nutch can continue
using it. ci-hadoop.apache.org provides almost the same functionality as
ci-builds.apache.org and there is no non-production Hadoop cluster running
there. Therefore moving to ci-hadoop does not make sense.

Short history: In the past there were some jenkins hosts that were labeled
for Hadoop and its related projects. After the migration to cloudbees, the
labeled hosts are moved under ci-hadoop.apache.org.

Thanks,
Akira


On Thu, Jan 6, 2022 at 2:20 PM lewis john mcgibbney 
wrote:

> Thank you for the response and for directing the conversation to the
> correct places.
> I may have misunderstood what ci-hadoop.apache.org actually is. We are
> looking for a non-production Hadoop cluster which we can use to simulate
> Nutch jobs. I am not sure if this is what ci-hadoop.apache.org actually
> is...
> Instead it looks like lots of compute resources used to perform Jenkins
> CI/CD tasks for Hadoop and associated projects rather than test things
> on-top of Hadoop (and associated projects).
> Any clarity on what ci-hadoop.apache.org actually is would be greatly
> appreciated.
>
> Let me also clarify my language, rather than have the integration tests run
> on every PR, we could trigger the integration tests to be run by tagging a
> Github bot i.e., "@nutchbot integration-test". Similar to what is done with
> Dependabot or conda-forge for anyon familiar with those mechanisms.
>
> Thanks for any advice or comments.
> lewismc
>
> On Wed, Jan 5, 2022 at 9:05 PM Ayush Saxena  wrote:
>
> > Moved to Dev lists.
> >
> > Not sure about this though:
> >  when a PR is submitted to Nutch project it will run some MR job in
> Hadoop
> > CI.
> >
> > Whatever that PR requires should run as part of Nutch Infra. Why in
> Hadoop
> > CI?
> > Our CI is already loaded with our own workloads.
> > If by any chance the above assertion gets a pass, then secondly we have
> > very less number of people managing work related to CI and Infra. I don’t
> > think most of the people won’t have context or say in the Nutch project,
> > neither bandwidth to fix stuff if it gets broken.
> >
> > Just my thoughts. Looped in the dev lists, if others have any feedback.
> As
> > for the process, this would require a consensus from the Hadoop PMC
> >
> > -Ayush
> >
> > > On 06-Jan-2022, at 7:02 AM, lewis john mcgibbney 
> > wrote:
> > >
> > > Hi general@,
> > >
> > > Not sure if this is the correct mailing list. Please redirect me if
> there
> > > is a more suitable location. Thank you
> > >
> > > I am PMC over on the Nutch project (https://nutch.apache.org). I would
> > like
> > > to investigate whether we can build an integration testing capability
> for
> > > the project. This would involve running a Nutch integration test suite
> > > (collection of MR jobs) in a Hadoop CI environment. For example
> whenever
> > a
> > > pull request is submitted to the Nutch project. This could easily be
> > > automated through Jenkins.
> > >
> > > I’m not sure if this is something the Hadoop PMC would consider. Thank
> > you
> > > for the consideration.
> > >
> > > lewismc
> > > --
> > > http://home.apache.org/~lewismc/
> > > http://people.apache.org/keys/committer/lewismc
> >
>
>
> --
> http://home.apache.org/~lewismc/
> http://people.apache.org/keys/committer/lewismc
>

Re: Possibility of using ci-hadoop.a.o for Nutch integration tests

2022-01-05 Thread lewis john mcgibbney

Thank you for the response and for directing the conversation to the
correct places.
I may have misunderstood what ci-hadoop.apache.org actually is. We are
looking for a non-production Hadoop cluster which we can use to simulate
Nutch jobs. I am not sure if this is what ci-hadoop.apache.org actually
is...
Instead it looks like lots of compute resources used to perform Jenkins
CI/CD tasks for Hadoop and associated projects rather than test things
on-top of Hadoop (and associated projects).
Any clarity on what ci-hadoop.apache.org actually is would be greatly
appreciated.

Let me also clarify my language, rather than have the integration tests run
on every PR, we could trigger the integration tests to be run by tagging a
Github bot i.e., "@nutchbot integration-test". Similar to what is done with
Dependabot or conda-forge for anyon familiar with those mechanisms.

Thanks for any advice or comments.
lewismc

On Wed, Jan 5, 2022 at 9:05 PM Ayush Saxena  wrote:

> Moved to Dev lists.
>
> Not sure about this though:
>  when a PR is submitted to Nutch project it will run some MR job in Hadoop
> CI.
>
> Whatever that PR requires should run as part of Nutch Infra. Why in Hadoop
> CI?
> Our CI is already loaded with our own workloads.
> If by any chance the above assertion gets a pass, then secondly we have
> very less number of people managing work related to CI and Infra. I don’t
> think most of the people won’t have context or say in the Nutch project,
> neither bandwidth to fix stuff if it gets broken.
>
> Just my thoughts. Looped in the dev lists, if others have any feedback. As
> for the process, this would require a consensus from the Hadoop PMC
>
> -Ayush
>
> > On 06-Jan-2022, at 7:02 AM, lewis john mcgibbney 
> wrote:
> >
> > Hi general@,
> >
> > Not sure if this is the correct mailing list. Please redirect me if there
> > is a more suitable location. Thank you
> >
> > I am PMC over on the Nutch project (https://nutch.apache.org). I would
> like
> > to investigate whether we can build an integration testing capability for
> > the project. This would involve running a Nutch integration test suite
> > (collection of MR jobs) in a Hadoop CI environment. For example whenever
> a
> > pull request is submitted to the Nutch project. This could easily be
> > automated through Jenkins.
> >
> > I’m not sure if this is something the Hadoop PMC would consider. Thank
> you
> > for the consideration.
> >
> > lewismc
> > --
> > http://home.apache.org/~lewismc/
> > http://people.apache.org/keys/committer/lewismc
>

-- 
http://home.apache.org/~lewismc/
http://people.apache.org/keys/committer/lewismc

Re: Possibility of using ci-hadoop.a.o for Nutch integration tests

2022-01-05 Thread Lewis John Mcgibbney

Thank you for the response and for directing the conversation to the
correct places.
I may have misunderstood what ci-hadoop.apache.org actually is. We are
looking for a non-production Hadoop cluster which we can use to simulate
Nutch jobs. I am not sure if this is what ci-hadoop.apache.org actually
is...
Instead it looks like lots of compute resources used to perform Jenkins
CI/CD tasks for Hadoop and associated projects rather than test things
on-top of Hadoop (and associated projects).
Any clarity on what ci-hadoop.apache.org actually is would be greatly
appreciated.
Thanks
lewismc

On Wed, Jan 5, 2022 at 9:05 PM Ayush Saxena  wrote:

> Moved to Dev lists.
>
> Not sure about this though:
>  when a PR is submitted to Nutch project it will run some MR job in Hadoop
> CI.
>
> Whatever that PR requires should run as part of Nutch Infra. Why in Hadoop
> CI?
> Our CI is already loaded with our own workloads.
> If by any chance the above assertion gets a pass, then secondly we have
> very less number of people managing work related to CI and Infra. I don’t
> think most of the people won’t have context or say in the Nutch project,
> neither bandwidth to fix stuff if it gets broken.
>
> Just my thoughts. Looped in the dev lists, if others have any feedback. As
> for the process, this would require a consensus from the Hadoop PMC
>
> -Ayush
>
> > On 06-Jan-2022, at 7:02 AM, lewis john mcgibbney 
> wrote:
> >
> > Hi general@,
> >
> > Not sure if this is the correct mailing list. Please redirect me if there
> > is a more suitable location. Thank you
> >
> > I am PMC over on the Nutch project (https://nutch.apache.org). I would
> like
> > to investigate whether we can build an integration testing capability for
> > the project. This would involve running a Nutch integration test suite
> > (collection of MR jobs) in a Hadoop CI environment. For example whenever
> a
> > pull request is submitted to the Nutch project. This could easily be
> > automated through Jenkins.
> >
> > I’m not sure if this is something the Hadoop PMC would consider. Thank
> you
> > for the consideration.
> >
> > lewismc
> > --
> > http://home.apache.org/~lewismc/
> > http://people.apache.org/keys/committer/lewismc
>

-- 

*Lewis*
Dr. Lewis J. McGibbney Ph.D, B.Sc
*Skype*: lewis.john.mcgibbney

Re: Possibility of using ci-hadoop.a.o for Nutch integration tests

2022-01-05 Thread Ayush Saxena

Moved to Dev lists.

Not sure about this though:
 when a PR is submitted to Nutch project it will run some MR job in Hadoop CI.

Whatever that PR requires should run as part of Nutch Infra. Why in Hadoop CI?
Our CI is already loaded with our own workloads.
If by any chance the above assertion gets a pass, then secondly we have very 
less number of people managing work related to CI and Infra. I don’t think most 
of the people won’t have context or say in the Nutch project, neither bandwidth 
to fix stuff if it gets broken.

Just my thoughts. Looped in the dev lists, if others have any feedback. As for 
the process, this would require a consensus from the Hadoop PMC

-Ayush

> On 06-Jan-2022, at 7:02 AM, lewis john mcgibbney  wrote:
> 
> Hi general@,
> 
> Not sure if this is the correct mailing list. Please redirect me if there
> is a more suitable location. Thank you
> 
> I am PMC over on the Nutch project (https://nutch.apache.org). I would like
> to investigate whether we can build an integration testing capability for
> the project. This would involve running a Nutch integration test suite
> (collection of MR jobs) in a Hadoop CI environment. For example whenever a
> pull request is submitted to the Nutch project. This could easily be
> automated through Jenkins.
> 
> I’m not sure if this is something the Hadoop PMC would consider. Thank you
> for the consideration.
> 
> lewismc
> -- 
> http://home.apache.org/~lewismc/
> http://people.apache.org/keys/committer/lewismc

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK11 on Linux/x86_64

2022-01-05 Thread Apache Jenkins Server

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/238/

[Jan 4, 2022 2:18:03 AM] (noreply) HADOOP-18061. Update the year to 2022. 
(#3845). Contributed by Ayush Saxena.
[Jan 4, 2022 2:20:01 AM] (noreply) HDFS-16393. RBF: Fix 
TestRouterRPCMultipleDestinationMountTableResolver. (#3849). Contributed by 
Ayush Saxena.
[Jan 4, 2022 2:25:13 AM] (noreply) HADOOP-18063. Remove unused import 
AbstractJavaKeyStoreProvider in Shell class. (#3846)
[Jan 4, 2022 4:25:16 AM] (noreply) HDFS-16409. Fix typo: 
testHasExeceptionsReturnsCorrectValue -> testHasExceptionsReturnsCorrectValue 
(#3835)
[Jan 4, 2022 4:59:54 PM] (noreply) HDFS-16407. Make hdfs_du tool cross platform 
(#3848)




-1 overall


The following subsystems voted -1:
blanks mvnsite pathlen spotbugs unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml
 

spotbugs :

   module:hadoop-hdfs-project/hadoop-hdfs 
   Redundant nullcheck of oldLock, which is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.DataStorage.isPreUpgradableLayout(Storage$StorageDirectory)
 Redundant null check at DataStorage.java:is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.DataStorage.isPreUpgradableLayout(Storage$StorageDirectory)
 Redundant null check at DataStorage.java:[line 695] 
   Redundant nullcheck of metaChannel, which is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MappableBlockLoader.verifyChecksum(long,
 FileInputStream, FileChannel, String) Redundant null check at 
MappableBlockLoader.java:is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MappableBlockLoader.verifyChecksum(long,
 FileInputStream, FileChannel, String) Redundant null check at 
MappableBlockLoader.java:[line 138] 
   Redundant nullcheck of blockChannel, which is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MemoryMappableBlockLoader.load(long,
 FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null 
check at MemoryMappableBlockLoader.java:is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MemoryMappableBlockLoader.load(long,
 FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null 
check at MemoryMappableBlockLoader.java:[line 75] 
   Redundant nullcheck of blockChannel, which is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.load(long,
 FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null 
check at NativePmemMappableBlockLoader.java:is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.load(long,
 FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null 
check at NativePmemMappableBlockLoader.java:[line 85] 
   Redundant nullcheck of metaChannel, which is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.verifyChecksumAndMapBlock(NativeIO$POSIX$PmemMappedRegion,
 long, FileInputStream, FileChannel, String) Redundant null check at 
NativePmemMappableBlockLoader.java:is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.verifyChecksumAndMapBlock(NativeIO$POSIX$PmemMappedRegion,
 long, FileInputStream, FileChannel, String) Redundant null check at 
NativePmemMappableBlockLoader.java:[line 130] 
   
org.apache.hadoop.hdfs.server.namenode.top.window.RollingWindowManager$UserCounts
 doesn't override java.util.ArrayList.equals(Object) At 
RollingWindowManager.java:At RollingWindowManager.java:[line 1] 

spotbugs :

   module:hadoop-yarn-project/hadoop-yarn 
   Redundant nullcheck of it, which is known to be non-null in

[jira] [Resolved] (HDFS-16371) Exclude slow disks when choosing volume

2022-01-05 Thread Takanobu Asanuma (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Takanobu Asanuma resolved HDFS-16371.
-
Fix Version/s: 3.4.0
   Resolution: Fixed

> Exclude slow disks when choosing volume
> ---
>
> Key: HDFS-16371
> URL: https://issues.apache.org/jira/browse/HDFS-16371
> Project: Hadoop HDFS
>  Issue Type: New Feature
>Reporter: tomscut
>Assignee: tomscut
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> Currently, the datanode can detect slow disks. See HDFS-11461.
> And after HDFS-16311, the slow disk information we collected is more accurate.
> So we can exclude these slow disks according to some rules when choosing 
> volume. This will prevents some slow disks from affecting the throughput of 
> the whole datanode.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64

2022-01-05 Thread Apache Jenkins Server

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/741/

[Jan 4, 2022 2:18:03 AM] (noreply) HADOOP-18061. Update the year to 2022. 
(#3845). Contributed by Ayush Saxena.
[Jan 4, 2022 2:20:01 AM] (noreply) HDFS-16393. RBF: Fix 
TestRouterRPCMultipleDestinationMountTableResolver. (#3849). Contributed by 
Ayush Saxena.
[Jan 4, 2022 2:25:13 AM] (noreply) HADOOP-18063. Remove unused import 
AbstractJavaKeyStoreProvider in Shell class. (#3846)
[Jan 4, 2022 4:25:16 AM] (noreply) HDFS-16409. Fix typo: 
testHasExeceptionsReturnsCorrectValue -> testHasExceptionsReturnsCorrectValue 
(#3835)
[Jan 4, 2022 4:59:54 PM] (noreply) HDFS-16407. Make hdfs_du tool cross platform 
(#3848)




-1 overall


The following subsystems voted -1:
blanks pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml
 

Failed junit tests :

   hadoop.ipc.TestIPC 
   hadoop.yarn.csi.client.TestCsiClient 
  

   cc:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/741/artifact/out/results-compile-cc-root.txt
 [96K]

   javac:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/741/artifact/out/results-compile-javac-root.txt
 [364K]

   blanks:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/741/artifact/out/blanks-eol.txt
 [13M]
  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/741/artifact/out/blanks-tabs.txt
 [2.0M]

   checkstyle:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/741/artifact/out/results-checkstyle-root.txt
 [14M]

   pathlen:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/741/artifact/out/results-pathlen.txt
 [16K]

   pylint:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/741/artifact/out/results-pylint.txt
 [20K]

   shellcheck:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/741/artifact/out/results-shellcheck.txt
 [28K]

   xml:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/741/artifact/out/xml.txt
 [24K]

   javadoc:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/741/artifact/out/results-javadoc-javadoc-root.txt
 [408K]

   unit:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/741/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
 [216K]
  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/741/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-csi.txt
 [20K]

Powered by Apache Yetus 0.14.0-SNAPSHOT   https://yetus.apache.org

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-16410) Insecure Xml parsing in OfflineEditsXmlLoader

2022-01-05 Thread Chao Sun (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chao Sun resolved HDFS-16410.
-
Fix Version/s: 3.4.0
   3.3.2
   Resolution: Fixed

> Insecure Xml parsing in OfflineEditsXmlLoader 
> --
>
> Key: HDFS-16410
> URL: https://issues.apache.org/jira/browse/HDFS-16410
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 3.3.1
>Reporter: Ashutosh Gupta
>Assignee: Ashutosh Gupta
>Priority: Minor
>  Labels: pull-request-available, security
> Fix For: 3.4.0, 3.3.2
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Insecure Xml parsing in OfflineEditsXmlLoader 
> [https://github.com/apache/hadoop/blob/03cfc852791c14fad39db4e5b14104a276c08e59/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineEditsViewer/OfflineEditsXmlLoader.java#L88]



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-16408) Ensure LeaseRecheckIntervalMs is greater than zero

2022-01-05 Thread Stephen O'Donnell (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephen O'Donnell resolved HDFS-16408.
--
Resolution: Fixed

> Ensure LeaseRecheckIntervalMs is greater than zero
> --
>
> Key: HDFS-16408
> URL: https://issues.apache.org/jira/browse/HDFS-16408
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Affects Versions: 3.1.3, 3.3.1
>Reporter: Jingxuan Fu
>Assignee: Jingxuan Fu
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0, 3.2.4, 3.3.3
>
>   Original Estimate: 1h
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> There is a problem with the try catch statement in the LeaseMonitor daemon 
> (in LeaseManager.java), when an unknown exception is caught, it simply prints 
> a warning message and continues with the next loop. 
> An extreme case is when the configuration item 
> 'dfs.namenode.lease-recheck-interval-ms' is accidentally set to a negative 
> number by the user, as the configuration item is read without checking its 
> range, 'fsnamesystem. getLeaseRecheckIntervalMs()' returns this value and is 
> used as an argument to Thread.sleep(). A negative argument will cause 
> Thread.sleep() to throw an IllegalArgumentException, which will be caught by 
> 'catch(Throwable e)' and a warning message will be printed. 
> This behavior is repeated for each subsequent loop. This means that a huge 
> amount of repetitive messages will be printed to the log file in a short 
> period of time, quickly consuming disk space and affecting the operation of 
> the system.
> As you can see, 178M log files are generated in one minute.
>  
> {code:java}
> ll logs/
> total 174456
> drwxrwxr-x  2 hadoop hadoop      4096 1月   3 15:13 ./
> drwxr-xr-x 11 hadoop hadoop      4096 1月   3 15:13 ../
> -rw-rw-r--  1 hadoop hadoop     36342 1月   3 15:14 
> hadoop-hadoop-datanode-ljq1.log
> -rw-rw-r--  1 hadoop hadoop      1243 1月   3 15:13 
> hadoop-hadoop-datanode-ljq1.out
> -rw-rw-r--  1 hadoop hadoop 178545466 1月   3 15:14 
> hadoop-hadoop-namenode-ljq1.log
> -rw-rw-r--  1 hadoop hadoop       692 1月   3 15:13 
> hadoop-hadoop-namenode-ljq1.out
> -rw-rw-r--  1 hadoop hadoop     33201 1月   3 15:14 
> hadoop-hadoop-secondarynamenode-ljq1.log
> -rw-rw-r--  1 hadoop hadoop      3764 1月   3 15:14 
> hadoop-hadoop-secondarynamenode-ljq1.out
> -rw-rw-r--  1 hadoop hadoop         0 1月   3 15:13 SecurityAuth-hadoop.audit
>  
> tail -n 15 logs/hadoop-hadoop-namenode-ljq1.log 
> 2022-01-03 15:14:46,032 WARN 
> org.apache.hadoop.hdfs.server.namenode.LeaseManager: Unexpected throwable: 
> java.lang.IllegalArgumentException: timeout value is negative
>         at java.base/java.lang.Thread.sleep(Native Method)
>         at 
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:534)
>         at java.base/java.lang.Thread.run(Thread.java:829)
> 2022-01-03 15:14:46,033 WARN 
> org.apache.hadoop.hdfs.server.namenode.LeaseManager: Unexpected throwable: 
> java.lang.IllegalArgumentException: timeout value is negative
>         at java.base/java.lang.Thread.sleep(Native Method)
>         at 
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:534)
>         at java.base/java.lang.Thread.run(Thread.java:829)
> 2022-01-03 15:14:46,033 WARN 
> org.apache.hadoop.hdfs.server.namenode.LeaseManager: Unexpected throwable: 
> java.lang.IllegalArgumentException: timeout value is negative
>         at java.base/java.lang.Thread.sleep(Native Method)
>         at 
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:534)
>         at java.base/java.lang.Thread.run(Thread.java:829)
> {code}
>  
> I think there are two potential solutions. 
> The first is to adjust the position of the try catch statement in the 
> LeaseMonitor daemon by moving 'catch(Throwable e)' to the outside of the loop 
> body. This can be done like the NameNodeResourceMonitor daemon, which ends 
> the thread when an unexpected exception is caught. 
> The second is to use Precondition.checkArgument() to scope the configuration 
> item 'dfs.namenode.lease-recheck-interval-ms' when it is read, to avoid the 
> wrong configuration item can affect the subsequent operation of the program.
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-16413) Reconfig dfs usage parameters for datanode

2022-01-05 Thread tomscut (Jira)

tomscut created HDFS-16413:
--

 Summary: Reconfig dfs usage parameters for datanode
 Key: HDFS-16413
 URL: https://issues.apache.org/jira/browse/HDFS-16413
 Project: Hadoop HDFS
  Issue Type: New Feature
Reporter: tomscut
Assignee: tomscut


Reconfig dfs usage parameters for datanode.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64

2022-01-05 Thread Apache Jenkins Server

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/533/

[Jan 4, 2022 2:36:27 AM] (Ayush Saxena) HADOOP-18061. Update the year to 2022. 
(#3845). Contributed by Ayush Saxena.




-1 overall


The following subsystems voted -1:
asflicense compile golang hadolint mvnsite pathlen unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.io.compress.snappy.TestSnappyCompressorDecompressor 
   hadoop.fs.TestFileUtil 
   hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys 
   
hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain 
   hadoop.hdfs.server.federation.resolver.order.TestLocalResolver 
   hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat 
   hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver 
   hadoop.hdfs.server.federation.router.TestRouterQuota 
   hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter 
   hadoop.mapreduce.lib.input.TestLineRecordReader 
   hadoop.mapred.TestLineRecordReader 
   hadoop.resourceestimator.service.TestResourceEstimatorService 
   hadoop.resourceestimator.solver.impl.TestLpSolver 
   hadoop.yarn.sls.TestSLSRunner 
   hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector 
   
hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker
 
   hadoop.yarn.server.resourcemanager.TestClientRMService 
  

   compile:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/533/artifact/out/patch-compile-root.txt
  [684K]

   cc:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/533/artifact/out/patch-compile-root.txt
  [684K]

   golang:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/533/artifact/out/patch-compile-root.txt
  [684K]

   javac:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/533/artifact/out/patch-compile-root.txt
  [684K]

   checkstyle:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/533/artifact/out/diff-checkstyle-root.txt
  [14M]

   hadolint:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/533/artifact/out/diff-patch-hadolint.txt
  [4.0K]

   mvnsite:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/533/artifact/out/patch-mvnsite-root.txt
  [608K]

   pathlen:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/533/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/533/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/533/artifact/out/diff-patch-shellcheck.txt
  [72K]

   whitespace:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/533/artifact/out/whitespace-eol.txt
  [12M]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/533/artifact/out/whitespace-tabs.txt
  [1.3M]

   javadoc:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/533/artifact/out/patch-javadoc-root.txt
  [76K]

   unit:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/533/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
  [244K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/533/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [424K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/533/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
  [36K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/533/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt
  [12K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/533/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
  [104K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/533/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt
  [20K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/533/artifact/out/patch-unit-hadoop-tools_hadoop-resourceestimator.txt
  [16K]

[jira] [Created] (HDFS-16412) Add metrics to support obtaining file size distribution

2022-01-05 Thread Xiangyi Zhu (Jira)

Xiangyi Zhu created HDFS-16412:
--

 Summary: Add metrics to support obtaining file size distribution
 Key: HDFS-16412
 URL: https://issues.apache.org/jira/browse/HDFS-16412
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: namenode
Affects Versions: 3.4.0
Reporter: Xiangyi Zhu
Assignee: Xiangyi Zhu


Use RangeMapRange "Map fileSizeRange" to store counters at 
different intervals. RangeMap key is a specific interval, and value is the 
counter corresponding to the interval.
**

*Counter update:*
When the file size changes or the file is deleted, the file size is obtained, 
and the counter in the corresponding interval is called to update the counter.
**

*Interval division:*
The default is to initialize the startup according to the following interval, 
or it can be initialized through the configuration file.
0MB
0-16MB
16-32MB
32-64MB
64-128MB
128-256MB
256-512MB
>512MB



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Possibility of using ci-hadoop.a.o for Nutch integration tests

Re: Next Mandarin Hadoop Online Meetup Jan 6th.

Re: Possibility of using ci-hadoop.a.o for Nutch integration tests

Re: Possibility of using ci-hadoop.a.o for Nutch integration tests

Re: Possibility of using ci-hadoop.a.o for Nutch integration tests

Re: Possibility of using ci-hadoop.a.o for Nutch integration tests

Apache Hadoop qbt Report: trunk+JDK11 on Linux/x86_64

[jira] [Resolved] (HDFS-16371) Exclude slow disks when choosing volume

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64

[jira] [Resolved] (HDFS-16410) Insecure Xml parsing in OfflineEditsXmlLoader

[jira] [Resolved] (HDFS-16408) Ensure LeaseRecheckIntervalMs is greater than zero

[jira] [Created] (HDFS-16413) Reconfig dfs usage parameters for datanode

Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64

[jira] [Created] (HDFS-16412) Add metrics to support obtaining file size distribution

14 matches

Site Navigation

Mail list logo

Footer information