date:20181102

[jira] [Created] (HDDS-798) Storage-class is showing incorrectly

2018-11-02 Thread Bharat Viswanadham (JIRA)

Bharat Viswanadham created HDDS-798:
---

 Summary: Storage-class is showing incorrectly
 Key: HDDS-798
 URL: https://issues.apache.org/jira/browse/HDDS-798
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Reporter: Bharat Viswanadham
Assignee: Bharat Viswanadham


After HDDS-712, we support storage-class.

For list-objects, even if key has set storage-class to REDUCED_REDUNDANCY, 
still it shows STANDARD.

As in code in list object response, we have hardcoded it as below.

keyMetadata.setStorageClass("STANDARD");



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-14051) Refactor NameNodeHttpServer#initWebHdfs to specify local keytab

2018-11-02 Thread JIRA

Íñigo Goiri created HDFS-14051:
--

 Summary: Refactor NameNodeHttpServer#initWebHdfs to specify local 
keytab
 Key: HDFS-14051
 URL: https://issues.apache.org/jira/browse/HDFS-14051
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Íñigo Goiri
Assignee: CR Hota


We use {{NameNodeHttpServer#initWebHdfs}} from {{RouterHttpServer}}.
However, this relies on {{NameNodeHttpServer#getAuthFilterParams()}} which uses:
{code}
String httpKeytab = conf.get(DFSUtil.getSpnegoKeytabKey(conf,
DFSConfigKeys.DFS_NAMENODE_KEYTAB_FILE_KEY));
{code}
We should refactor this to be able to specify the keytab file.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-14050) Use parameterized logging construct in NamenodeFsck class

2018-11-02 Thread Hrishikesh Gadre (JIRA)

Hrishikesh Gadre created HDFS-14050:
---

 Summary: Use parameterized logging construct in NamenodeFsck class
 Key: HDFS-14050
 URL: https://issues.apache.org/jira/browse/HDFS-14050
 Project: Hadoop HDFS
  Issue Type: Task
Affects Versions: 3.0.0
Reporter: Hrishikesh Gadre
Assignee: Hrishikesh Gadre


HDFS-13695 implemented a change to use slf4j logger (instead of commons 
logging). But the NamenodeFsck class is still not using parameterized logging 
construct. This came up during the code review for HADOOP-11391. We should 
change logging statements in NamenodeFsck to use slf4j style parameterized 
logging apis.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-14049) TestHttpFSServerWebServer fails on Windows because of missing winutils.exe

2018-11-02 Thread JIRA

Íñigo Goiri created HDFS-14049:
--

 Summary: TestHttpFSServerWebServer fails on Windows because of 
missing winutils.exe
 Key: HDFS-14049
 URL: https://issues.apache.org/jira/browse/HDFS-14049
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Íñigo Goiri
Assignee: Íñigo Goiri


2018-11-02 15:00:51,780 WARN  Shell - Did not find winutils.exe: {}
java.io.FileNotFoundException: Could not locate Hadoop executable: 
D:\hadoop-2.9\hadoop-hdfs-project\hadoop-hdfs-httpfs\target\test-dir\bin\winutils.exe
 -see https://wiki.apache.org/hadoop/WindowsProblems
at org.apache.hadoop.util.Shell.getQualifiedBinInner(Shell.java:612)
at org.apache.hadoop.util.Shell.getQualifiedBin(Shell.java:585)
at org.apache.hadoop.util.Shell.(Shell.java:682)
at org.apache.hadoop.util.StringUtils.(StringUtils.java:78)
at 
org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1587)
at 
org.apache.hadoop.fs.http.server.HttpFSServerWebServer.(HttpFSServerWebServer.java:93)
at 
org.apache.hadoop.fs.http.server.TestHttpFSServerWebServer.setUp(TestHttpFSServerWebServer.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at 
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-797) If DN is started before SCM, it does not register

2018-11-02 Thread Hanisha Koneru (JIRA)

Hanisha Koneru created HDDS-797:
---

 Summary: If DN is started before SCM, it does not register
 Key: HDDS-797
 URL: https://issues.apache.org/jira/browse/HDDS-797
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Reporter: Hanisha Koneru
Assignee: Hanisha Koneru






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [DISCUSS] Hadoop RPC encryption performance improvements

2018-11-02 Thread Wei-Chiu Chuang

Thanks all for the inputs,

To offer additional information (while Daryn is working on his stuff),
optimizing RPC encryption opens up another possibility: migrating KMS
service to use Hadoop RPC.

Today's KMS uses HTTPS + REST API, much like webhdfs. It has very
undesirable performance (a few thousand ops per second) compared to
NameNode. Unfortunately for each NameNode namespace operation you also need
to access KMS too.

Migrating KMS to Hadoop RPC greatly improves its performance (if
implemented correctly), and RPC encryption would be a prerequisite. So
please keep that in mind when discussing the Hadoop RPC encryption
improvements. Cloudera is very interested to help with the Hadoop RPC
encryption project because a lot of our customers are using at-rest
encryption, and some of them are starting to hit KMS performance limit.

This whole "migrating KMS to Hadoop RPC" was Daryn's idea. I heard this
idea in the meetup and I am very thrilled to see this happening because it
is a real issue bothering some of our customers, and I suspect it is the
right solution to address this tech debt.

On Fri, Nov 2, 2018 at 1:21 PM Todd Lipcon 
wrote:

> One possibility (which we use in Kudu) is to use SSL for encryption but
> with a self-signed certificate, maintaining the existing SASL/GSSAPI
> handshake for authentication. The one important bit here, security wise, is
> to implement channel binding (RFC 5056 and RFC 5929) to prevent against
> MITMs. The description of the Kudu protocol is here:
>
> https://github.com/apache/kudu/blob/master/docs/design-docs/rpc.md#wire-protocol
>
> If implemented correctly, this provides TLS encryption (with all of its
> performance and security benefits) without requiring the user to deploy a
> custom cert.
>
> -Todd
>
> On Thu, Nov 1, 2018 at 7:14 PM Konstantin Shvachko 
> wrote:
>
> > Hi Wei-Chiu,
> >
> > Thanks for starting the thread and summarizing the problem. Sorry for
> slow
> > response.
> > We've been looking at the encrypted performance as well and are
> interested
> > in this effort.
> > We ran some benchmarks locally. Our benchmarks also showed substantial
> > penalty for turning on wire encryption on rpc.
> > Although it was less drastic - more in the range of -40%. But we ran a
> > different benchmark NNThroughputBenchmark, and we ran it on 2.6 last
> year.
> > Could have published the results, but need to rerun on more recent
> > versions.
> >
> > Three points from me on this discussion:
> >
> > 1. We should settle on the benchmarking tools.
> > For development RPCCallBenchmark is good as it measures directly the
> > improvement on the RPC layer. But for external consumption it is more
> > important to know about e.g. NameNode RPCs performance. So we probably
> > should run both benchmarks.
> > 2. SASL vs SSL.
> > Since current implementation is based on SASL, I think it would make
> sense
> > to make improvements in this direction. I assume switching to SSL would
> > require changes in configuration. Not sure if it will be compatible,
> since
> > we don't have the details. At this point I would go with HADOOP-10768.
> > Given all (Daryn's) concerns are addressed.
> > 3. Performance improvement expectations.
> > Ideally we want to have < 10% penalty for encrypted communication.
> Anything
> > over 30% will probably have very limited usability. And there is the gray
> > area in between, which could be mitigated by allowing mixed encrypted and
> > un-encrypted RPCs on the single NameNode like in HDFS-13566.
> >
> > Thanks,
> > --Konstantin
> >
> > On Wed, Oct 31, 2018 at 7:39 AM Daryn Sharp 
> > wrote:
> >
> > > Various KMS tasks have been delaying my RPC encryption work – which is
> > 2nd
> > > on TODO list.  It's becoming a top priority for us so I'll try my best
> to
> > > get a preliminary netty server patch (sans TLS) up this week if that
> > helps.
> > >
> > > The two cited jiras had some critical flaws.  Skimming my comments,
> both
> > > use blocking IO (obvious nonstarter).  HADOOP-10768 is a hand rolled
> > > TLS-like encryption which I don't feel is something the community can
> or
> > > should maintain from a security standpoint.
> > >
> > > Daryn
> > >
> > > On Wed, Oct 31, 2018 at 8:43 AM Wei-Chiu Chuang 
> > > wrote:
> > >
> > > > Ping. Any one? Cloudera is interested in moving forward with the RPC
> > > > encryption improvements, but I just like to get a consensus which
> > > approach
> > > > to go with.
> > > >
> > > > Otherwise I'll pick HADOOP-10768 since it's ready for commit, and
> I've
> > > > spent time on testing it.
> > > >
> > > > On Thu, Oct 25, 2018 at 11:04 AM Wei-Chiu Chuang  >
> > > > wrote:
> > > >
> > > > > Folks,
> > > > >
> > > > > I would like to invite all to discuss the various Hadoop RPC
> > encryption
> > > > > performance improvements. As you probably know, Hadoop RPC
> encryption
> > > > > currently relies on Java SASL, and have _really_ bad performance
> (in
> > > > terms
> > > > > of number of RPCs per second, around 15~20% of

Re: [DISCUSS] Hadoop RPC encryption performance improvements

2018-11-02 Thread Todd Lipcon

One possibility (which we use in Kudu) is to use SSL for encryption but
with a self-signed certificate, maintaining the existing SASL/GSSAPI
handshake for authentication. The one important bit here, security wise, is
to implement channel binding (RFC 5056 and RFC 5929) to prevent against
MITMs. The description of the Kudu protocol is here:
https://github.com/apache/kudu/blob/master/docs/design-docs/rpc.md#wire-protocol

If implemented correctly, this provides TLS encryption (with all of its
performance and security benefits) without requiring the user to deploy a
custom cert.

-Todd

On Thu, Nov 1, 2018 at 7:14 PM Konstantin Shvachko 
wrote:

> Hi Wei-Chiu,
>
> Thanks for starting the thread and summarizing the problem. Sorry for slow
> response.
> We've been looking at the encrypted performance as well and are interested
> in this effort.
> We ran some benchmarks locally. Our benchmarks also showed substantial
> penalty for turning on wire encryption on rpc.
> Although it was less drastic - more in the range of -40%. But we ran a
> different benchmark NNThroughputBenchmark, and we ran it on 2.6 last year.
> Could have published the results, but need to rerun on more recent
> versions.
>
> Three points from me on this discussion:
>
> 1. We should settle on the benchmarking tools.
> For development RPCCallBenchmark is good as it measures directly the
> improvement on the RPC layer. But for external consumption it is more
> important to know about e.g. NameNode RPCs performance. So we probably
> should run both benchmarks.
> 2. SASL vs SSL.
> Since current implementation is based on SASL, I think it would make sense
> to make improvements in this direction. I assume switching to SSL would
> require changes in configuration. Not sure if it will be compatible, since
> we don't have the details. At this point I would go with HADOOP-10768.
> Given all (Daryn's) concerns are addressed.
> 3. Performance improvement expectations.
> Ideally we want to have < 10% penalty for encrypted communication. Anything
> over 30% will probably have very limited usability. And there is the gray
> area in between, which could be mitigated by allowing mixed encrypted and
> un-encrypted RPCs on the single NameNode like in HDFS-13566.
>
> Thanks,
> --Konstantin
>
> On Wed, Oct 31, 2018 at 7:39 AM Daryn Sharp 
> wrote:
>
> > Various KMS tasks have been delaying my RPC encryption work – which is
> 2nd
> > on TODO list.  It's becoming a top priority for us so I'll try my best to
> > get a preliminary netty server patch (sans TLS) up this week if that
> helps.
> >
> > The two cited jiras had some critical flaws.  Skimming my comments, both
> > use blocking IO (obvious nonstarter).  HADOOP-10768 is a hand rolled
> > TLS-like encryption which I don't feel is something the community can or
> > should maintain from a security standpoint.
> >
> > Daryn
> >
> > On Wed, Oct 31, 2018 at 8:43 AM Wei-Chiu Chuang 
> > wrote:
> >
> > > Ping. Any one? Cloudera is interested in moving forward with the RPC
> > > encryption improvements, but I just like to get a consensus which
> > approach
> > > to go with.
> > >
> > > Otherwise I'll pick HADOOP-10768 since it's ready for commit, and I've
> > > spent time on testing it.
> > >
> > > On Thu, Oct 25, 2018 at 11:04 AM Wei-Chiu Chuang 
> > > wrote:
> > >
> > > > Folks,
> > > >
> > > > I would like to invite all to discuss the various Hadoop RPC
> encryption
> > > > performance improvements. As you probably know, Hadoop RPC encryption
> > > > currently relies on Java SASL, and have _really_ bad performance (in
> > > terms
> > > > of number of RPCs per second, around 15~20% of the one without SASL)
> > > >
> > > > There have been some attempts to address this, most notably,
> > HADOOP-10768
> > > >  (Optimize
> Hadoop
> > > RPC
> > > > encryption performance) and HADOOP-13836
> > > >  (Securing
> Hadoop
> > > RPC
> > > > using SSL). But it looks like both attempts have not been
> progressing.
> > > >
> > > > During the recent Hadoop contributor meetup, Daryn Sharp mentioned
> he's
> > > > working on another approach that leverages Netty for its SSL
> > encryption,
> > > > and then integrate Netty with Hadoop RPC so that Hadoop RPC
> > automatically
> > > > benefits from netty's SSL encryption performance.
> > > >
> > > > So there are at least 3 attempts to address this issue as I see it.
> Do
> > we
> > > > have a consensus that:
> > > > 1. this is an important problem
> > > > 2. which approach we want to move forward with
> > > >
> > > > --
> > > > A very happy Hadoop contributor
> > > >
> > >
> > >
> > > --
> > > A very happy Hadoop contributor
> > >
> >
> >
> > --
> >
> > Daryn
> >
>


-- 
Todd Lipcon
Software Engineer, Cloudera

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2018-11-02 Thread Apache Jenkins Server

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/945/

[Nov 1, 2018 10:13:48 AM] (shashikant) HDDS-771. ChunkGroupOutputStream stream 
entries need to be properly
[Nov 1, 2018 12:56:20 PM] (stevel) HADOOP-15895. [JDK9+] Add missing 
javax.annotation-api dependency to
[Nov 1, 2018 9:22:00 PM] (jhung) YARN-7225. Add queue and partition info to RM 
audit log. Contributed by
[Nov 2, 2018 12:26:00 AM] (weichiu) HDFS-14008. NN should log snapshotdiff 
report. Contributed by Pranay
[Nov 2, 2018 12:36:50 AM] (weichiu) HDFS-13996. Make HttpFS' ACLs RegEx 
configurable. Contributed by Siyao
[Nov 2, 2018 2:50:32 AM] (yqlin) HDDS-751. Replace usage of Guava Optional with 
Java Optional.




-1 overall


The following subsystems voted -1:
asflicense findbugs hadolint pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml
 

Failed CTEST tests :

   test_test_libhdfs_threaded_hdfs_static 
   test_libhdfs_threaded_hdfspp_test_shim_static 

Failed junit tests :

   hadoop.util.TestReadWriteDiskValidator 
   hadoop.util.TestBasicDiskValidator 
   hadoop.util.TestDiskCheckerWithDiskIo 
   hadoop.fs.contract.router.web.TestRouterWebHDFSContractAppend 
   
hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage
 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/945/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/945/artifact/out/diff-compile-javac-root.txt
  [324K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/945/artifact/out/diff-checkstyle-root.txt
  [17M]

   hadolint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/945/artifact/out/diff-patch-hadolint.txt
  [4.0K]

   pathlen:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/945/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/945/artifact/out/diff-patch-pylint.txt
  [40K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/945/artifact/out/diff-patch-shellcheck.txt
  [68K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/945/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/945/artifact/out/whitespace-eol.txt
  [9.3M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/945/artifact/out/whitespace-tabs.txt
  [1.1M]

   xml:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/945/artifact/out/xml.txt
  [4.0K]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/945/artifact/out/branch-findbugs-hadoop-hdds_client.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/945/artifact/out/branch-findbugs-hadoop-hdds_container-service.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/945/artifact/out/branch-findbugs-hadoop-hdds_framework.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/945/artifact/out/branch-findbugs-hadoop-hdds_server-scm.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/945/artifact/out/branch-findbugs-hadoop-hdds_tools.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/945/artifact/out/branch-findbugs-hadoop-ozone_client.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/945/artifact/out/branch-findbugs-hadoop-ozone_common.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/945/artifact/out/branch-findbugs-hadoop-ozone_objectstore-service.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/945/artifact/out/branch-findbugs-hadoop-ozone_ozone-manager.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/945/artifact/out/branch-findbugs-hadoop-ozone_ozonefs.txt
  [16K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/945/artifact/out/branch-findbugs-hadoop-ozone_s3gateway.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/945/artifact/out/branch-findbugs-hadoop-ozone_tools.txt
  [8.0K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/945/artifact/out/diff-javadoc-javadoc-root.txt
  [752K]

   CTEST:

   
https://builds.apache.or

[jira] [Created] (HDFS-14048) DFSOutputStream close() throws exception on subsequent call after DataNode restart

2018-11-02 Thread Erik Krogen (JIRA)

Erik Krogen created HDFS-14048:
--

 Summary: DFSOutputStream close() throws exception on subsequent 
call after DataNode restart
 Key: HDFS-14048
 URL: https://issues.apache.org/jira/browse/HDFS-14048
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: hdfs-client
Affects Versions: 2.3.0
Reporter: Erik Krogen
Assignee: Erik Krogen


We recently discovered an issue in which, during a rolling upgrade, some jobs 
were failing with exceptions like (sadly this is the whole stack trace):
{code}
java.io.IOException: A datanode is restarting: 
DatanodeInfoWithStorage[1.1.1.1:71,BP-,DISK]
at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:877)
{code}
with an earlier statement in the log like:
{code}
INFO [main] org.apache.hadoop.hdfs.DFSClient: A datanode is restarting: 
DatanodeInfoWithStorage[1.1.1.1:71,BP-,DISK]
{code}

Strangely we did not see any other logs about the {{DFSOutputStream}} failing 
after waiting for the DataNode restart. We eventually realized that in some 
cases {{DFSOutputStream#close()}} may be called more than once, and that if so, 
the {{IOException}} above is thrown on the _second_ call to {{close()}} (this 
is even with HDFS-5335; prior to this it would have been thrown on all calls to 
{{close()}} besides the first).

The problem is that in {{DataStreamer#createBlockOutputStream()}}, after the 
new output stream is created, it resets the error states:
{code}
errorState.resetInternalError();
// remove all restarting nodes from failed nodes list
failed.removeAll(restartingNodes);
restartingNodes.clear(); 
{code}
But it forgets to clear {{lastException}}. When {{DFSOutputStream#closeImpl()}} 
is called a second time, this block is triggered:
{code}
if (isClosed()) {
  LOG.debug("Closing an already closed stream. [Stream:{}, streamer:{}]",
  closed, getStreamer().streamerClosed());
  try {
getStreamer().getLastException().check(true);
{code}
The second time, {{isClosed()}} is true, so the exception checking occurs and 
the "Datanode is restarting" exception is thrown even though the stream has 
already been successfully closed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-796) Fix failed test TestStorageContainerManagerHttpServer#testHttpPolicy

2018-11-02 Thread Yiqun Lin (JIRA)

Yiqun Lin created HDDS-796:
--

 Summary: Fix failed test 
TestStorageContainerManagerHttpServer#testHttpPolicy
 Key: HDDS-796
 URL: https://issues.apache.org/jira/browse/HDDS-796
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Reporter: Yiqun Lin
Assignee: Yiqun Lin


After replacing usage of Guava Optional with Java Optional (HDDS-751), there 
are two unit test failures generated 
([https://builds.apache.org/job/PreCommit-HDDS-Build/1594/testReport/]). The 
unit tests are not fully triggered and tested under HDDS-751, :P
{noformat}
java.util.NoSuchElementException: No value present
at java.util.Optional.get(Optional.java:135)
at 
org.apache.hadoop.hdds.server.BaseHttpServer.getBindAddress(BaseHttpServer.java:121)
at 
org.apache.hadoop.hdds.server.BaseHttpServer.getHttpBindAddress(BaseHttpServer.java:148)
at 
org.apache.hadoop.hdds.server.BaseHttpServer.(BaseHttpServer.java:63)
at 
org.apache.hadoop.hdds.scm.server.StorageContainerManagerHttpServer.(StorageContainerManagerHttpServer.java:34)
at 
org.apache.hadoop.hdds.scm.TestStorageContainerManagerHttpServer.testHttpPolicy(TestStorageContainerManagerHttpServer.java:103)
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-14047) [libhdfs++] Fix hdfsGetLastExceptionRootCause bug in test_libhdfs_threaded.c

2018-11-02 Thread Anatoli Shein (JIRA)

Anatoli Shein created HDFS-14047:


 Summary: [libhdfs++] Fix hdfsGetLastExceptionRootCause bug in 
test_libhdfs_threaded.c
 Key: HDFS-14047
 URL: https://issues.apache.org/jira/browse/HDFS-14047
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: libhdfs, native
Reporter: Anatoli Shein
Assignee: Anatoli Shein


Currently the native client CI tests break deterministically with these errors:

Libhdfs

1 - test_test_libhdfs_threaded_hdfs_static (Failed)

[exec] TEST_ERROR: failed on 
/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180
 with NULL return return value (errno: 2): expected substring: File does not 
exist [exec] TEST_ERROR: failed on 
/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336
 with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti, fs, 
&paths) [exec] hdfsOpenFile(/tlhData0001/file1): 
FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;)
 error: [exec] (unable to get root cause for java.io.FileNotFoundException) 
[exec] (unable to get stack trace for java.io.FileNotFoundException)

 

Libhdfs++

34 - test_libhdfs_threaded_hdfspp_test_shim_static (Failed)

[exec] TEST_ERROR: failed on 
/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180
 with NULL return return value (errno: 2): expected substring: File does not 
exist [exec] TEST_ERROR: failed on 
/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336
 with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti, fs, 
&paths) [exec] hdfsOpenFile(/tlhData0001/file1): 
FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;)
 error: [exec] (unable to get root cause for java.io.FileNotFoundException) 
[exec] (unable to get stack trace for java.io.FileNotFoundException)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-795) RocksDb specific classes leak from DBStore/Table interfaces

2018-11-02 Thread Elek, Marton (JIRA)

Elek, Marton created HDDS-795:
-

 Summary: RocksDb specific classes leak from DBStore/Table 
interfaces
 Key: HDDS-795
 URL: https://issues.apache.org/jira/browse/HDDS-795
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
Reporter: Elek, Marton
Assignee: Elek, Marton


org.apache.hadoop.utils.db.RocksDB and Table interfaces provide a 
vendor-independent way to access any key value store. 

The default implementation uses RocksDb but other implementation also could be 
used (for example an InMemory implementation for testing only).

The current Table interface contains methods which depend on RocksDB specific 
classes. For example:

{code}
public interface DBStore extends AutoCloseable {
//...
/**
   * Return the Column Family handle. TODO: This leaks an RockDB abstraction
   * into Ozone code, cleanup later.
   *
   * @return ColumnFamilyHandle
   */
  ColumnFamilyHandle getHandle();
//...
{code}

We need to remove the RocksDB specific classes from the generic interfaces.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-794) Add configs to set StateMachineData write timeout in ContainerStateMachine

2018-11-02 Thread Shashikant Banerjee (JIRA)

Shashikant Banerjee created HDDS-794:


 Summary: Add configs to set StateMachineData write timeout in 
ContainerStateMachine
 Key: HDDS-794
 URL: https://issues.apache.org/jira/browse/HDDS-794
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
  Components: Ozone Datanode
Reporter: Shashikant Banerjee
Assignee: Shashikant Banerjee
 Fix For: 0.3.0, 0.4.0


The patch will address adding config settings in Ozone which will 
enable/disable timeout for StateMachineData write via Ratis. It also adds some 
debug logs in writeChunk handling path inside datanode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-14046) There is no ICON for Maintenance In Datanode UI page and after Datanode moved into Maintenance states still datanode mark status is empty in Datanode UI.

2018-11-02 Thread Harshakiran Reddy (JIRA)

Harshakiran Reddy created HDFS-14046:


 Summary: There is no ICON for Maintenance In Datanode UI page and 
after Datanode moved into Maintenance  states still datanode mark status is 
empty in Datanode UI.
 Key: HDFS-14046
 URL: https://issues.apache.org/jira/browse/HDFS-14046
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: datanode
Affects Versions: 3.1.1
Reporter: Harshakiran Reddy






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-793) Support custom key/value annotations on volume/bucket/key

2018-11-02 Thread Elek, Marton (JIRA)

Elek, Marton created HDDS-793:
-

 Summary: Support custom key/value annotations on volume/bucket/key
 Key: HDDS-793
 URL: https://issues.apache.org/jira/browse/HDDS-793
 Project: Hadoop Distributed Data Store
  Issue Type: New Feature
  Components: OM
Reporter: Elek, Marton
Assignee: Elek, Marton


I propose to add a custom Map annotation field to 
objects/buckets and keys in Ozone Manager.

It would enable to build any extended functionality on top of the OM's generic 
interface. For example:

 * Support tags in Ozone S3 gateway 
(https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectGETtagging.html)
 * Support md5 based ETags in s3g
 * Store s3 related authorization data (ACLs, policies) together with the 
parent objects

As an optional feature (could be implemented later) the client can defined the 
exposed annotations. For example s3g can defined which annotations should be 
read from rocksdb on OM side and sent the the client (s3g)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-792) Use md5 hash as ETag for Ozone S3 objects

2018-11-02 Thread Elek, Marton (JIRA)

Elek, Marton created HDDS-792:
-

 Summary: Use md5 hash as ETag for Ozone S3 objects
 Key: HDDS-792
 URL: https://issues.apache.org/jira/browse/HDDS-792
 Project: Hadoop Distributed Data Store
  Issue Type: Sub-task
  Components: S3
Reporter: Elek, Marton


AWS S3 uses md5 hash of the files as ETag. 

Not a strict requirement, but s3 tests (https://github.com/gaul/s3-tests/) can 
not been executed without that.

It requires to support custom key/value annotations on key objects.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-791) Support Range header for Oozne s3 object download

2018-11-02 Thread Elek, Marton (JIRA)

Elek, Marton created HDDS-791:
-

 Summary: Support Range header for Oozne s3 object download
 Key: HDDS-791
 URL: https://issues.apache.org/jira/browse/HDDS-791
 Project: Hadoop Distributed Data Store
  Issue Type: Sub-task
  Components: S3
Reporter: Elek, Marton


Using s3 rest api smaller chunks of an object could be uploaded with using 
Range headers:

For example:

{code}
GET /example-object HTTP/1.1
Host: example-bucket.s3.amazonaws.com
x-amz-date: Fri, 28 Jan 2011 21:32:02 GMT
Range: bytes=0-9
Authorization: AWS AKIAIOSFODNN7EXAMPLE:Yxg83MZaEgh3OZ3l0rLo5RTX11o=
Sample Response with Specified Range of the Object Bytes
{code}

Can be implemented with using the seek method on OzoneInputStream.

The Range header  support is one of the missing piece for fully support s3a 
interface.

References:
Range header spec:
https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35

Aws s3 doc:
https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectGET.html



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-790) Support multipart upload on Ozone S3 gateway interface

2018-11-02 Thread Elek, Marton (JIRA)

Elek, Marton created HDDS-790:
-

 Summary: Support multipart upload on Ozone S3 gateway interface
 Key: HDDS-790
 URL: https://issues.apache.org/jira/browse/HDDS-790
 Project: Hadoop Distributed Data Store
  Issue Type: Sub-task
  Components: S3
Reporter: Elek, Marton


AWS S3 Muptipart uploads API enables to upload huge files in chunks. 

Documented here:
https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html

Should be investigated how is it possible with OzoneClient.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-798) Storage-class is showing incorrectly

[jira] [Created] (HDFS-14051) Refactor NameNodeHttpServer#initWebHdfs to specify local keytab

[jira] [Created] (HDFS-14050) Use parameterized logging construct in NamenodeFsck class

[jira] [Created] (HDFS-14049) TestHttpFSServerWebServer fails on Windows because of missing winutils.exe

[jira] [Created] (HDDS-797) If DN is started before SCM, it does not register

Re: [DISCUSS] Hadoop RPC encryption performance improvements

Re: [DISCUSS] Hadoop RPC encryption performance improvements

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[jira] [Created] (HDFS-14048) DFSOutputStream close() throws exception on subsequent call after DataNode restart

[jira] [Created] (HDDS-796) Fix failed test TestStorageContainerManagerHttpServer#testHttpPolicy

[jira] [Created] (HDFS-14047) [libhdfs++] Fix hdfsGetLastExceptionRootCause bug in test_libhdfs_threaded.c

[jira] [Created] (HDDS-795) RocksDb specific classes leak from DBStore/Table interfaces

[jira] [Created] (HDDS-794) Add configs to set StateMachineData write timeout in ContainerStateMachine

[jira] [Created] (HDFS-14046) There is no ICON for Maintenance In Datanode UI page and after Datanode moved into Maintenance states still datanode mark status is empty in Datanode UI.

[jira] [Created] (HDDS-793) Support custom key/value annotations on volume/bucket/key

[jira] [Created] (HDDS-792) Use md5 hash as ETag for Ozone S3 objects

[jira] [Created] (HDDS-791) Support Range header for Oozne s3 object download

[jira] [Created] (HDDS-790) Support multipart upload on Ozone S3 gateway interface

18 matches

Site Navigation

Mail list logo

Footer information