Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1194/

[Jul 10, 2019 2:19:36 AM] (msingh) HDDS-1603. Handle Ratis Append Failure in 
Container State Machine.
[Jul 10, 2019 2:53:34 AM] (yqlin) HDFS-14632. Reduce useless 
#getNumLiveDataNodes call in SafeModeMonitor.
[Jul 10, 2019 11:15:55 AM] (msingh) HDDS-1748. Error message for 3 way commit 
failure is not verbose.
[Jul 10, 2019 11:22:51 AM] (elek) HDDS-1764. Fix hidden errors in acceptance 
tests
[Jul 10, 2019 1:31:28 PM] (elek) HDDS-1525. Mapreduce failure when using Hadoop 
2.7.5
[Jul 10, 2019 4:43:58 PM] (arp7) HDDS-1778. Fix existing blockade tests. (#1068)
[Jul 10, 2019 4:59:11 PM] (xkrogen) HDFS-14622. [Dynamometer] Update XML 
FsImage parsing logic to ignore
[Jul 10, 2019 6:03:58 PM] (aengineer) HDDS-1611. Evaluate ACL on volume bucket 
key and prefix to authorize
[Jul 10, 2019 6:11:52 PM] (inigoiri) HDFS-12703. Exceptions are fatal to 
decommissioning monitor. Contributed
[Jul 10, 2019 6:28:18 PM] (aengineer) HDDS-1611.[Addendum] Evaluate ACL on 
volume bucket key and prefix to
[Jul 10, 2019 7:57:02 PM] (stevel) HADOOP-16393. S3Guard init command uses 
global settings, not those of
[Jul 10, 2019 9:15:33 PM] (eyang) YARN-9660. Update support documentation for 
Docker on YARN.

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1783) Latency metric for applyTransaction in ContainerStateMachine

[Supratim Deka (JIRA)]

Supratim Deka created HDDS-1783:
---

 Summary: Latency metric for applyTransaction in 
ContainerStateMachine
 Key: HDDS-1783
 URL: https://issues.apache.org/jira/browse/HDDS-1783
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
  Components: Ozone Datanode
Reporter: Supratim Deka


applyTransaction is invoked from the Ratis pipeline and the 
ContainerStateMachine

uses a async executor to complete the task.

 

We require a latency metric to track the performance of log apply operations in 
the state machine. This will measure the end-to-end latency of apply which 
includes the queueing delay in the executor queues. Combined with the latency 
measurement in HddsDispatcher, this will be an indicator if the executors are 
overloaded.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1782) Add an option to MiniOzoneChaosCluster to read files multiple times.

[Mukul Kumar Singh (JIRA)]

Mukul Kumar Singh created HDDS-1782:
---

 Summary: Add an option to MiniOzoneChaosCluster to read files 
multiple times.
 Key: HDDS-1782
 URL: https://issues.apache.org/jira/browse/HDDS-1782
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
  Components: test
Affects Versions: 0.4.0
Reporter: Mukul Kumar Singh
Assignee: Mukul Kumar Singh


Right now MiniOzoneChaosCluster writes a file/ reads it and deletes it 
immediately. This jira proposes to add an option to read the file multiple time 
in MiniOzoneChaosCluster.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Incorrect NOTICE files in TLP releases

[Akira Ajisaka]

Hi Vinod,

This issue is now tracked by https://issues.apache.org/jira/browse/HADOOP-15958

Thanks,
Akira

On Fri, Jul 5, 2019 at 1:29 PM Vinod Kumar Vavilapalli
 wrote:
>
> A bit of an old email, but want to make sure this isn't missed.
>
> Has anyone looked into this concern?
>
> Ref https://issues.apache.org/jira/browse/ROL-2138 
> .
>
> Thanks
> +Vinod
>
> > Begin forwarded message:
> >
> > From: sebb 
> > Subject: Incorrect NOTICE files in TLP releases
> > Date: June 21, 2019 at 2:34:17 AM GMT+5:30
> > To: "bo...@apache.org Board" 
> > Reply-To: bo...@apache.org
> >
> > To whom it may concern:
> >
> > I had occasion to download the source for Roller, and happened to look
> > at the NOTICE file.
> > It does not conform to ASF policies, so I raised ROL-2138.
> >
> > One of the replies asked how to manage different N files for binary
> > and source releases, and pointed out that Hadoop and Karaf don't
> > appear to have multiple copies of the files.
> >
> > So I had a look at Hadoop and Karaf; their NOTICE files are also
> > non-standard, and it looks like Kafka does not have a NOTICE file in
> > the source bundle.
> >
> > I suspect these are not the only projects with non-conformant NOTICE files.
> > The LICENSE files are also likely to be incorrect (I have not checked).
> >
> > Sebb.
>

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1781) Add ContainerCache metrics in ContainerMetrics

[Supratim Deka (JIRA)]

Supratim Deka created HDDS-1781:
---

 Summary: Add ContainerCache metrics in ContainerMetrics
 Key: HDDS-1781
 URL: https://issues.apache.org/jira/browse/HDDS-1781
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
Reporter: Supratim Deka


ContainerCache cache handles to open Container DB instances. This LRU cache is 
configured with a limited capacity (1024 entries default).

Add metrics to track the performance of this cache(hits : misses) and also 
track the average latency to acquire a DB handle in case of cache miss.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-14644) That replication of block failed leads to decommission is blocked when the number of replicas of block is greater than the number of datanode

[Lisheng Sun (JIRA)]

Lisheng Sun created HDFS-14644:
--

 Summary: That replication of block failed leads to decommission is 
blocked when the number of replicas of block is greater than the number of 
datanode
 Key: HDFS-14644
 URL: https://issues.apache.org/jira/browse/HDFS-14644
 Project: Hadoop HDFS
  Issue Type: Improvement
Affects Versions: 2.7.7, 2.8.5, 3.0.3, 2.9.2, 3.1.1
Reporter: Lisheng Sun


2019-07-10,15:37:18,028 WARN 
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to 
place enough replicas, still in need of 5 to reach 10 
(unavailableStorages=[DISK, ARCHIVE], storagePolicy=BlockStoragePolicy\{HOT:7, 
storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, 
newBlock=false) All required storage types are unavailable: 
unavailableStorages=[DISK, ARCHIVE], storagePolicy=BlockStoragePolicy\{HOT:7, 
storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}
2019-07-10,15:37:18,028 WARN 
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to 
place enough replicas, still in need of 5 to reach 10 (unavailableStorages=[], 
storagePolicy=BlockStoragePolicy\{HOT:7, storageTypes=[DISK], 
creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=false) For more 
information, please enable DEBUG log level on 
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-1611) Evaluate ACL on volume bucket key and prefix to authorize access

[Anu Engineer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anu Engineer resolved HDDS-1611.

Resolution: Fixed

> Evaluate ACL on volume bucket key and prefix to authorize access 
> -
>
> Key: HDDS-1611
> URL: https://issues.apache.org/jira/browse/HDDS-1611
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>Reporter: Xiaoyu Yao
>Assignee: Ajay Kumar
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.5.0, 0.4.1
>
> Attachments: HDDS-1611-fix-trunk.patch
>
>  Time Spent: 6h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Reopened] (HDDS-1611) Evaluate ACL on volume bucket key and prefix to authorize access

[Anu Engineer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anu Engineer reopened HDDS-1611:


> Evaluate ACL on volume bucket key and prefix to authorize access 
> -
>
> Key: HDDS-1611
> URL: https://issues.apache.org/jira/browse/HDDS-1611
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>Reporter: Xiaoyu Yao
>Assignee: Ajay Kumar
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.5.0, 0.4.1
>
>  Time Spent: 6h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-1611) Evaluate ACL on volume bucket key and prefix to authorize access

[Anu Engineer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anu Engineer resolved HDDS-1611.

   Resolution: Fixed
Fix Version/s: 0.4.1
   0.5.0

Thanks for the patch. I have committed this patch to the trunk. I will cherry 
pick this to branch 0.4.1 shortly.

> Evaluate ACL on volume bucket key and prefix to authorize access 
> -
>
> Key: HDDS-1611
> URL: https://issues.apache.org/jira/browse/HDDS-1611
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>Reporter: Xiaoyu Yao
>Assignee: Ajay Kumar
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.5.0, 0.4.1
>
>  Time Spent: 6h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1193/

[Jul 9, 2019 3:12:55 AM] (msingh) HDDS-1750. Add block allocation metrics for 
pipelines in SCM.
[Jul 9, 2019 3:24:12 AM] (aengineer) HDDS-1550. MiniOzoneCluster is not 
shutting down all the threads during
[Jul 9, 2019 4:06:50 AM] (arp7) HDDS-1705. Recon: Add estimatedTotalCount to 
the response of containers
[Jul 9, 2019 11:22:00 AM] (elek) HDDS-1717. MR Job fails as 
OMFailoverProxyProvider has dependency
[Jul 9, 2019 3:21:16 PM] (elek) HDDS-1742. Merge ozone-perf and ozonetrace 
example clusters
[Jul 9, 2019 5:47:50 PM] (msingh) HDDS-1718. Increase Ratis Leader election 
timeout default. Contributed
[Jul 9, 2019 9:43:55 PM] (xyao) HDDS-1586. Allow Ozone RPC client to read with 
topology awareness.




-1 overall


The following subsystems voted -1:
asflicense findbugs hadolint pathlen unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-documentstore
 
   Unread field:TimelineEventSubDoc.java:[line 56] 
   Unread field:TimelineMetricSubDoc.java:[line 44] 

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/hadoop-yarn-applications-mawo-core
 
   Class org.apache.hadoop.applications.mawo.server.common.TaskStatus 
implements Cloneable but does not define or use clone method At 
TaskStatus.java:does not define or use clone method At TaskStatus.java:[lines 
39-346] 
   Equals method for 
org.apache.hadoop.applications.mawo.server.worker.WorkerId assumes the argument 
is of type WorkerId At WorkerId.java:the argument is of type WorkerId At 
WorkerId.java:[line 114] 
   
org.apache.hadoop.applications.mawo.server.worker.WorkerId.equals(Object) does 
not check for null argument At WorkerId.java:null argument At 
WorkerId.java:[lines 114-115] 

FindBugs :

   module:hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra 
   org.apache.hadoop.tools.dynamometer.Client.addFileToZipRecursively(File, 
File, ZipOutputStream) may fail to clean up java.io.InputStream on checked 
exception Obligation to clean up resource created at Client.java:to clean up 
java.io.InputStream on checked exception Obligation to clean up resource 
created at Client.java:[line 863] is not discharged 
   Exceptional return value of java.io.File.mkdirs() ignored in 
org.apache.hadoop.tools.dynamometer.DynoInfraUtils.fetchHadoopTarball(File, 
String, Configuration, Logger) At DynoInfraUtils.java:ignored in 
org.apache.hadoop.tools.dynamometer.DynoInfraUtils.fetchHadoopTarball(File, 
String, Configuration, Logger) At DynoInfraUtils.java:[line 142] 
   Found reliance on default encoding in 
org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.run(String[]):in 
org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.run(String[]): new 
java.io.InputStreamReader(InputStream) At SimulatedDataNodes.java:[line 149] 
   org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.run(String[]) 
invokes System.exit(...), which shuts down the entire virtual machine At 
SimulatedDataNodes.java:down the entire virtual machine At 
SimulatedDataNodes.java:[line 123] 
   org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.run(String[]) may 
fail to close stream At SimulatedDataNodes.java:stream At 
SimulatedDataNodes.java:[line 149] 

FindBugs :

   module:hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-blockgen 
   Self assignment of field BlockInfo.replication in new 
org.apache.hadoop.tools.dynamometer.blockgenerator.BlockInfo(BlockInfo) At 
BlockInfo.java:in new 
org.apache.hadoop.tools.dynamometer.blockgenerator.BlockInfo(BlockInfo) At 
BlockInfo.java:[line 78] 

Failed junit tests :

   hadoop.hdfs.server.datanode.TestBPOfferService 
   hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes 
   hadoop.hdfs.TestMultipleNNPortQOP 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.hdfs.server.datanode.TestDirectoryScanner 
   hadoop.hdfs.server.federation.router.TestRouterWithSecureStartup 
   hadoop.hdfs.server.federation.security.TestRouterHttpDelegationToken 
   hadoop.ozone.freon.TestRandomKeyGenerator 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1193/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1193/artifact/out/diff-compile-javac-root.txt
  [336K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1193/artifact/out/diff-checkstyle-root.txt
  [17M]

   hadolint:

   

[jira] [Created] (HDFS-14643) [Dynamometer] Merge extra commits from GitHub to Hadoop

[Erik Krogen (JIRA)]

Erik Krogen created HDFS-14643:
--

 Summary: [Dynamometer] Merge extra commits from GitHub to Hadoop
 Key: HDFS-14643
 URL: https://issues.apache.org/jira/browse/HDFS-14643
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Erik Krogen
Assignee: Erik Krogen


While Dynamometer was in the process of being committed to Hadoop, a few 
patches went in to the GitHub version that haven't yet made it into the version 
committed here. Some of them are related to TravisCI and Bintray deployment, 
which can safely be ignored in a Hadoop context, but a few are relevant:

{code}
* 2d2591e 2019-05-24 Make XML parsing error message more explicit (PR #97) 
[lfengnan ]
* 755a298 2019-04-04 Fix misimplemented CountTimeWritable setter and update the 
README docs regarding the output file (PR #96) [Christopher Gregorian 
]
* 66d3e19 2019-03-14 Modify AuditReplay workflow to output count and latency of 
operations (PR #92) [Christopher Gregorian ]
* 5c1d8cd 2019-02-28 Fix issues with the start-workload.sh script (PR #84) 
[Erik Krogen ]
{code}

I will use this ticket to track porting these 4 commits into Hadoop's 
Dynamometer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-14642) processMisReplicatedBlocks does not return correct processed count

[Stephen O'Donnell (JIRA)]

Stephen O'Donnell created HDFS-14642:


 Summary: processMisReplicatedBlocks does not return correct 
processed count
 Key: HDFS-14642
 URL: https://issues.apache.org/jira/browse/HDFS-14642
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 3.2.0
Reporter: Stephen O'Donnell
Assignee: Stephen O'Donnell


HDFS-14053 introduced a method "processMisReplicatedBlocks" to the 
blockManager, and it is used by fsck to schedule mis-replicated blocks for 
replication.

The method should return a the number of blocks it processed, but it always 
returns zero as "processed" is never incremented in the method.

It should also drop and re-take the write lock every "numBlocksPerIteration" 
but as processed is never incremented, it will never drop and re-take the write 
lock, giving potential for holding the write lock for a long time.
{code:java}
public int processMisReplicatedBlocks(List blocks) {
  int processed = 0;
  Iterator iter = blocks.iterator();

  try {
while (isPopulatingReplQueues() && namesystem.isRunning()
&& !Thread.currentThread().isInterrupted()
&& iter.hasNext()) {
  int limit = processed + numBlocksPerIteration;
  namesystem.writeLockInterruptibly();
  try {
while (iter.hasNext() && processed < limit) {
  BlockInfo blk = iter.next();
  MisReplicationResult r = processMisReplicatedBlock(blk);
  LOG.debug("BLOCK* processMisReplicatedBlocks: " +
  "Re-scanned block {}, result is {}", blk, r);
}
  } finally {
namesystem.writeUnlock();
  }
}
  } catch (InterruptedException ex) {
LOG.info("Caught InterruptedException while scheduling replication work" +
" for mis-replicated blocks");
Thread.currentThread().interrupt();
  }

  return processed;
}{code}
Due to this, fsck causes a warning to be logged in the NN for every 
mis-replicated file it schedules replication for, as it checks the processed 
count:
{code:java}
2019-07-10 15:46:14,790 WARN namenode.NameNode: Fsck: Block manager is able to 
process only 0 mis-replicated blocks (Total count : 1 ) for path /...{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Any thoughts making Submarine a separate Apache project?

[Wanqiang Ji]

+1  This is a fantastic recommendation. I can see the community grows fast
and good collaborative, submarine can be an independent project at now,
thanks for all contributors.

FYI,
Wanqiang Ji

On Wed, Jul 10, 2019 at 3:34 PM Xun Liu  wrote:

> Hi all,
>
> This is Xun Liu contributing to the Submarine project for deep learning
> workloads running with big data workloads together on Hadoop clusters.
>
> There are a bunch of integrations of Submarine to other projects are
> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next step
> of Submarine is going to integrate with more projects like Apache Arrow,
> Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> cases like model serving, notebook management, advanced training
> optimizations (like auto parameter tuning, memory cache optimizations for
> large datasets for training, etc.), and make it run on other platforms like
> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY project
> to Apache so we can put Submarine and TonY together to the same codebase
> (Page #30.
>
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> ).
>
> This expands the scope of the original Submarine project in exciting new
> ways. Toward that end, would it make sense to create a separate Submarine
> project at Apache? This can make faster adoption of Submarine, and allow
> Submarine to grow to a full-blown machine learning platform.
>
> There will be lots of technical details to work out, but any initial
> thoughts on this?
>
> Best Regards,
> Xun Liu
>

[jira] [Resolved] (HDDS-1459) Docker compose of ozonefs has older hadoop image for hadoop 3.2

[Elek, Marton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elek, Marton resolved HDDS-1459.

Resolution: Duplicate

Thanks the report [~vivekratnavel]

It's fixed with HDDS-1525:

ozonefs compose files are removed because ozone-mr tests are improved and they 
include the same functionality (ozone fs test with hdfs cli AND with mr client).

Versions are fixed (2.7, 3.1, 3.2)

> Docker compose of ozonefs has older hadoop image for hadoop 3.2
> ---
>
> Key: HDDS-1459
> URL: https://issues.apache.org/jira/browse/HDDS-1459
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: Ozone Filesystem
>Affects Versions: 0.4.0
>Reporter: Vivek Ratnavel Subramanian
>Assignee: Vivek Ratnavel Subramanian
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1780) TestFailureHandlingByClient tests are flaky

[Shashikant Banerjee (JIRA)]

Shashikant Banerjee created HDDS-1780:
-

 Summary: TestFailureHandlingByClient tests are flaky
 Key: HDDS-1780
 URL: https://issues.apache.org/jira/browse/HDDS-1780
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
  Components: Ozone Client
Affects Versions: 0.5.0
Reporter: Shashikant Banerjee
Assignee: Shashikant Banerjee
 Fix For: 0.5.0


The tests seem to fail bcoz , when the datanode goes down with stale node 
interval being set to a low value, containers may get closed early and client 
writes might fail with closed container exception rather than pipeline 
failure/Timeout exceptions as excepted in the tests. The fix made here is to 
tune the stale node interval.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/378/

[Jul 9, 2019 3:54:37 PM] (stack) Backport HDFS-3246,HDFS-14111 ByteBuffer pread 
interface to branch-2.9
[Jul 9, 2019 3:57:57 PM] (stack) Revert "Backport HDFS-3246,HDFS-14111 
ByteBuffer pread interface to
[Jul 9, 2019 3:58:16 PM] (stack) HDFS-14483 Backport HDFS-3246,HDFS-14111 
ByteBuffer pread interface to




-1 overall


The following subsystems voted -1:
asflicense findbugs hadolint pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/empty-configuration.xml
 
   hadoop-tools/hadoop-azure/src/config/checkstyle-suppressions.xml 
   hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/public/crossdomain.xml 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml
 

FindBugs :

   module:hadoop-common-project/hadoop-common 
   Class org.apache.hadoop.fs.GlobalStorageStatistics defines non-transient 
non-serializable instance field map In GlobalStorageStatistics.java:instance 
field map In GlobalStorageStatistics.java 

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client
 
   Boxed value is unboxed and then immediately reboxed in 
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result,
 byte[], byte[], KeyConverter, ValueConverter, boolean) At 
ColumnRWHelper.java:then immediately reboxed in 
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result,
 byte[], byte[], KeyConverter, ValueConverter, boolean) At 
ColumnRWHelper.java:[line 335] 

Failed junit tests :

   hadoop.hdfs.server.datanode.TestDirectoryScanner 
   hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys 
   hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.registry.secure.TestSecureLogins 
   hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/378/artifact/out/diff-compile-cc-root-jdk1.7.0_95.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/378/artifact/out/diff-compile-javac-root-jdk1.7.0_95.txt
  [328K]

   cc:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/378/artifact/out/diff-compile-cc-root-jdk1.8.0_212.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/378/artifact/out/diff-compile-javac-root-jdk1.8.0_212.txt
  [308K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/378/artifact/out/diff-checkstyle-root.txt
  [16M]

   hadolint:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/378/artifact/out/diff-patch-hadolint.txt
  [4.0K]

   pathlen:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/378/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/378/artifact/out/diff-patch-pylint.txt
  [24K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/378/artifact/out/diff-patch-shellcheck.txt
  [72K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/378/artifact/out/diff-patch-shelldocs.txt
  [8.0K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/378/artifact/out/whitespace-eol.txt
  [12M]
   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/378/artifact/out/whitespace-tabs.txt
  [1.2M]

   xml:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/378/artifact/out/xml.txt
  [12K]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/378/artifact/out/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/378/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-client-warnings.html
  [8.0K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/378/artifact/out/diff-javadoc-javadoc-root-jdk1.7.0_95.txt
  [16K]
   

[jira] [Created] (HDDS-1778) Fix existing blockade tests

[Nanda kumar (JIRA)]

Nanda kumar created HDDS-1778:
-

 Summary: Fix existing blockade tests
 Key: HDDS-1778
 URL: https://issues.apache.org/jira/browse/HDDS-1778
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
  Components: test
Reporter: Nanda kumar
Assignee: Nanda kumar


This jira is to track and fix existing blockade test cases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Any thoughts making Submarine a separate Apache project?

[Xun Liu]

Hi all,

This is Xun Liu contributing to the Submarine project for deep learning
workloads running with big data workloads together on Hadoop clusters.

There are a bunch of integrations of Submarine to other projects are
finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next step
of Submarine is going to integrate with more projects like Apache Arrow,
Redis, MLflow, etc. & be able to handle end-to-end machine learning use
cases like model serving, notebook management, advanced training
optimizations (like auto parameter tuning, memory cache optimizations for
large datasets for training, etc.), and make it run on other platforms like
Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY project
to Apache so we can put Submarine and TonY together to the same codebase
(Page #30.
https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
).

This expands the scope of the original Submarine project in exciting new
ways. Toward that end, would it make sense to create a separate Submarine
project at Apache? This can make faster adoption of Submarine, and allow
Submarine to grow to a full-blown machine learning platform.

There will be lots of technical details to work out, but any initial
thoughts on this?

Best Regards,
Xun Liu