[jira] [Resolved] (HADOOP-18592) Sasl connection failure should log remote address
[
https://issues.apache.org/jira/browse/HADOOP-18592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu resolved HADOOP-18592.
Fix Version/s: 3.4.0
3.3.9
Hadoop Flags: Reviewed
Resolution: Fixed
Committed to branch-3.3 and trunk branches. Thank you for your contribution,
[~vjasani] . Thank you all for review.
> Sasl connection failure should log remote address
> -
>
> Key: HADOOP-18592
> URL: https://issues.apache.org/jira/browse/HADOOP-18592
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.3.4
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.9
>
>
> If Sasl connection fails with some generic error, we miss logging remote
> server that the client was trying to connect to.
> Sample log:
> {code:java}
> 2023-01-12 00:22:28,148 WARN [20%2C1673404849949,1] ipc.Client - Exception
> encountered while connecting to the server
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> at sun.nio.ch.IOUtil.read(IOUtil.java:197)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
> at
> org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:141)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
> at java.io.FilterInputStream.read(FilterInputStream.java:133)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
> at java.io.DataInputStream.readInt(DataInputStream.java:387)
> at org.apache.hadoop.ipc.Client$IpcStreams.readResponse(Client.java:1950)
> at
> org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:367)
> at
> org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:623)
> at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:414)
> ...
> ... {code}
> We should log the remote server address.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-17728) Deadlock in FileSystem StatisticsDataReferenceCleaner cleanUp
[
https://issues.apache.org/jira/browse/HADOOP-17728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu reopened HADOOP-17728:
> Deadlock in FileSystem StatisticsDataReferenceCleaner cleanUp
> -
>
> Key: HADOOP-17728
> URL: https://issues.apache.org/jira/browse/HADOOP-17728
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.2.1
>Reporter: yikf
>Assignee: yikf
>Priority: Minor
> Labels: pull-request-available, reverted
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> Cleaner thread will be blocked if we remove reference from ReferenceQueue
> unless the `queue.enqueue` called.
>
> As shown below, We call ReferenceQueue.remove() now while cleanUp, Call
> chain as follow：
> *StatisticsDataReferenceCleaner#queue.remove() ->
> ReferenceQueue.remove(0) -> lock.wait(0)*
> But, lock.notifyAll is called when queue.enqueue only, so Cleaner thread
> will be blocked.
>
> ThreadDump:
> {code:java}
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x7f7afc088800
> nid=0x2119 in Object.wait() [0x7f7b0023]
>java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xc00c2f58> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:502)
> at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
> - locked <0xc00c2f58> (a java.lang.ref.Reference$Lock)
> at
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17728) Deadlock in FileSystem StatisticsDataReferenceCleaner cleanUp
[
https://issues.apache.org/jira/browse/HADOOP-17728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu resolved HADOOP-17728.
Fix Version/s: 3.2.3
3.4.0
3.3.1
Hadoop Flags: Reviewed
Resolution: Fixed
Committed to 3.1+ Thanks for your reporting and contribution [~kaifeiYi].
Thanks for your review [~ste...@apache.org]
> Deadlock in FileSystem StatisticsDataReferenceCleaner cleanUp
> -
>
> Key: HADOOP-17728
> URL: https://issues.apache.org/jira/browse/HADOOP-17728
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.2.1
>Reporter: yikf
>Assignee: yikf
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.3.1, 3.4.0, 3.2.3
>
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> Cleaner thread will be blocked if we remove reference from ReferenceQueue
> unless the `queue.enqueue` called.
>
> As shown below, We call ReferenceQueue.remove() now while cleanUp, Call
> chain as follow：
> *StatisticsDataReferenceCleaner#queue.remove() ->
> ReferenceQueue.remove(0) -> lock.wait(0)*
> But, lock.notifyAll is called when queue.enqueue only, so Cleaner thread
> will be blocked.
>
> ThreadDump:
> {code:java}
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x7f7afc088800
> nid=0x2119 in Object.wait() [0x7f7b0023]
>java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xc00c2f58> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:502)
> at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
> - locked <0xc00c2f58> (a java.lang.ref.Reference$Lock)
> at
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16355) ZookeeperMetadataStore: Use Zookeeper as S3Guard backend store
[
https://issues.apache.org/jira/browse/HADOOP-16355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu resolved HADOOP-16355.
Resolution: Abandoned
As documented by [[HADOOP-17480]], AWS S3 is consistent and that S3Guard is not
needed.
> ZookeeperMetadataStore: Use Zookeeper as S3Guard backend store
> --
>
> Key: HADOOP-16355
> URL: https://issues.apache.org/jira/browse/HADOOP-16355
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Mingliang Liu
>Priority: Major
>
> When S3Guard was proposed, there are a couple of valid reasons to choose
> DynamoDB as its default backend store: 0) seamless integration as part of AWS
> ecosystem e.g. client library 1) it's a managed web service which is zero
> operational cost, highly available and infinitely scalable 2) it's performant
> with single digit latency 3) it's proven by Netflix's S3mper (not actively
> maintained) and EMRFS (closed source and usage). As it's pluggable, it's
> possible to implement {{MetadataStore}} with other backend store without
> changing semantics, besides null and in-memory local ones.
> Here we propose {{ZookeeperMetadataStore}} which uses Zookeeper as S3Guard
> backend store. Its main motivation is to provide a new MetadataStore option
> which:
> # can be easily integrated as Zookeeper is heavily used in Hadoop community
> # affordable performance as both client and Zookeeper ensemble are usually
> "local" in a Hadoop cluster (ZK/HBase/Hive etc)
> # removes DynamoDB dependency
> Obviously all use cases will not prefer this to default DynamoDB store. For
> e.g. ZK might not scale well if there are dozens of S3 buckets and each has
> millions of objects. Our use case is targeting HBase to store HFiles on S3
> instead of HDFS. A total solution for HBase on S3 must be HBOSS (see
> HBASE-22149) for recovering atomicity of metadata operations like rename, and
> S3Guard for consistent enumeration and access to object store bucket
> metadata. We would like to use Zookeeper as backend store for both.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17252) Website to link to latest Hadoop wiki
Mingliang Liu created HADOOP-17252: -- Summary: Website to link to latest Hadoop wiki Key: HADOOP-17252 URL: https://issues.apache.org/jira/browse/HADOOP-17252 Project: Hadoop Common Issue Type: Bug Reporter: Mingliang Liu Currently the website links to the [old wiki|https://wiki.apache.org/hadoop]. Shall we update that to the latest one: https://cwiki.apache.org/confluence/display/HADOOP2/Home -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17159) Make UGI support forceful relogin from keytab ignoring the last login time
[ https://issues.apache.org/jira/browse/HADOOP-17159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu resolved HADOOP-17159. Fix Version/s: 2.10.1 Hadoop Flags: Reviewed Resolution: Fixed Committed to 2.10.1 and 3.1.5+ see "Fix Version/s". Thank you for your contribution, [~sandeep.guggilam] > Make UGI support forceful relogin from keytab ignoring the last login time > -- > > Key: HADOOP-17159 > URL: https://issues.apache.org/jira/browse/HADOOP-17159 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.10.0, 3.3.0, 3.2.1, 3.1.3 >Reporter: Sandeep Guggilam >Assignee: Sandeep Guggilam >Priority: Major > Fix For: 3.2.2, 2.10.1, 3.3.1, 3.4.0, 3.1.5 > > Time Spent: 40m > Remaining Estimate: 0h > > Currently we have a relogin() method in UGI which attempts to login if there > is no login attempted in the last 10 minutes or configured amount of time > We should also have provision for doing a forceful relogin irrespective of > the time window that the client can choose to use it if needed . Consider the > below scenario: > # SASL Server is reimaged and new keytabs are fetched with refreshing the > password > # SASL client connection to the server would fail when it tries with the > cached service ticket > # We should try to logout to clear the service tickets in cache and then try > to login back in such scenarios. But since the current relogin() doesn't > guarantee a login, it could cause an issue > # A forceful relogin in this case would help after logout > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17184) Add --mvn-custom-repos parameter to yetus calls
Mingliang Liu created HADOOP-17184:
--
Summary: Add --mvn-custom-repos parameter to yetus calls
Key: HADOOP-17184
URL: https://issues.apache.org/jira/browse/HADOOP-17184
Project: Hadoop Common
Issue Type: Bug
Components: build
Reporter: Mingliang Liu
In my request PR [#2188|https://github.com/apache/hadoop/pull/2188], I see the
QA build fails with unrelated errors.
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-install-plugin:2.5.1:install (default-install)
on project hadoop-project: Failed to install metadata
org.apache.hadoop:hadoop-project:3.4.0-SNAPSHOT/maven-metadata.xml: Could not
parse metadata
/home/jenkins/.m2/repository/org/apache/hadoop/hadoop-project/3.4.0-SNAPSHOT/maven-metadata-local.xml:
in epilog non whitespace content is not allowed but got n (position: END_TAG
seen ...\nn... @21:2) -> [Help 1]
{code}
As reported by HBASE-22474 and HBASE-22801, PreCommit validation from yetus
uses a shared .m2 repository. By adding {{--mvn-custom-repos}} and
{{--jenkins}} paramters, yetus will use a custom .m2 directory for executions
for PR validations.
This is a change to mimic that for Hadoop project.
CC: [~aajisaka]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17052) NetUtils.connect() throws unchecked exception (UnresolvedAddressException) causing clients to abort
[ https://issues.apache.org/jira/browse/HADOOP-17052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu resolved HADOOP-17052. Fix Version/s: 3.4.0 3.3.1 2.10.1 3.2.2 3.1.4 2.9.3 Hadoop Flags: Reviewed Resolution: Fixed Committed to all fixed versions branches. Thanks [~dhegde] for the reporting and providing a fix; thanks [~hemanthboyina] for review. > NetUtils.connect() throws unchecked exception (UnresolvedAddressException) > causing clients to abort > --- > > Key: HADOOP-17052 > URL: https://issues.apache.org/jira/browse/HADOOP-17052 > Project: Hadoop Common > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.10.0, 2.9.2, 3.2.1, 3.1.3 >Reporter: Dhiraj Hegde >Assignee: Dhiraj Hegde >Priority: Major > Fix For: 2.9.3, 3.1.4, 3.2.2, 2.10.1, 3.3.1, 3.4.0 > > Attachments: read_failure.log, write_failure1.log, write_failure2.log > > > Hadoop components are increasingly being deployed on VMs and containers. One > aspect of this environment is that DNS is dynamic. Hostname records get > modified (or deleted/recreated) as a container in Kubernetes (or even VM) is > being created/recreated. In such dynamic environments, the initial DNS > resolution request might return resolution failure briefly as DNS client > doesn't always get the latest records. This has been observed in Kubernetes > in particular. In such cases NetUtils.connect() appears to throw > java.nio.channels.UnresolvedAddressException. In much of Hadoop code (like > DFSInputStream and DFSOutputStream), the code is designed to retry > IOException. However, since UnresolvedAddressException is not child of > IOException, no retry happens and the code aborts immediately. It is much > better if NetUtils.connect() throws java.net.UnknownHostException as that is > derived from IOException and the code will treat this as a retry-able error. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17013) this bug is bla bla bla
[ https://issues.apache.org/jira/browse/HADOOP-17013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu resolved HADOOP-17013. Fix Version/s: (was: 0.24.0) Target Version/s: (was: 3.2.1) Resolution: Invalid This is not fun [~islam.saied]. Please stop doing this. > this bug is bla bla bla > --- > > Key: HADOOP-17013 > URL: https://issues.apache.org/jira/browse/HADOOP-17013 > Project: Hadoop Common > Issue Type: Bug > Components: bin >Affects Versions: 3.2.1 >Reporter: islam >Priority: Major > Labels: bulk-closed > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16853) ITestS3GuardOutOfBandOperations failing on versioned S3 buckets
[
https://issues.apache.org/jira/browse/HADOOP-16853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu resolved HADOOP-16853.
Fix Version/s: 3.3.0
Hadoop Flags: Reviewed
Resolution: Fixed
Committed to {{trunk}}. Thanks [~ste...@apache.org] for reporting and fixing.
Thanks [~gabor.bota] for review.
> ITestS3GuardOutOfBandOperations failing on versioned S3 buckets
> ---
>
> Key: HADOOP-16853
> URL: https://issues.apache.org/jira/browse/HADOOP-16853
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Fix For: 3.3.0
>
>
> org.apache.hadoop.fs.s3a.ITestS3GuardOutOfBandOperations.testListingDelete[auth=true]
> failing because the deleted file can still be read when the s3guard entry has
> the versionId.
> Proposed: if the FS is versioned and the file status has versionID then we
> switch to tests which assert the file is readable, rather than tests which
> assert it isn't there
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16827) TestHarFileSystem.testInheritedMethodsImplemented broken
[ https://issues.apache.org/jira/browse/HADOOP-16827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu resolved HADOOP-16827. Fix Version/s: 3.3.0 Hadoop Flags: Reviewed Resolution: Fixed > TestHarFileSystem.testInheritedMethodsImplemented broken > > > Key: HADOOP-16827 > URL: https://issues.apache.org/jira/browse/HADOOP-16827 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs, test >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Fix For: 3.3.0 > > > caused by HADOOP-16759. > I'm surprised this didn't surface earlier -or embarrassed that if it did, > somehow I missed it. Will fix. > > will also review the checksum FS to make sure it's gone through there too. > > Given I was using the IDE to refactor, it should have all been automatic -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16759) Filesystem openFile() builder to take a FileStatus param
[
https://issues.apache.org/jira/browse/HADOOP-16759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu resolved HADOOP-16759.
Fix Version/s: 3.3.0
Hadoop Flags: Reviewed
Resolution: Fixed
Committed to {{trunk}} branch. I see major conflicts when backporting to 3.2.
[~ste...@apache.org] Do you plan to bulk backport some time later those patches
which apply, or you upload a new patch for branch-3.2/3.1? Thanks!
> Filesystem openFile() builder to take a FileStatus param
>
>
> Key: HADOOP-16759
> URL: https://issues.apache.org/jira/browse/HADOOP-16759
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs, fs/azure, fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Fix For: 3.3.0
>
>
> Let us pass in a file status to openFile() so that S3A & ABFS will skip
> their own HEAD requests just to see if a file is there, a normal file and get
> its length + etag,
> {code}
> CompletableFuture streamF = fs.openFile(stat.getPath())
> .withFileStatus(stat).build()
> {code}
> code opening files off a listing of everything in a directory can eliminate a
> lot of requests here.
> Also: change the specification of openFile's completable future to say
> "returned stream may only raise FNFE, access restrictions on the first read"
> That is: it's not just potentially an async open, it's possibly lazy
> evaluated entirely.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16758) Refine testing.md to tell user better how to use auth-keys.xml
Mingliang Liu created HADOOP-16758:
--
Summary: Refine testing.md to tell user better how to use
auth-keys.xml
Key: HADOOP-16758
URL: https://issues.apache.org/jira/browse/HADOOP-16758
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Reporter: Mingliang Liu
Assignee: Mingliang Liu
I have a small patch to refine \{{testing.md}} about `auth-keys.xml`. I think
this might be applicable to others.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16757) Increase timeout unit test rule for ITestDynamoDBMetadataStore
Mingliang Liu created HADOOP-16757:
--
Summary: Increase timeout unit test rule for
ITestDynamoDBMetadataStore
Key: HADOOP-16757
URL: https://issues.apache.org/jira/browse/HADOOP-16757
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Mingliang Liu
Not sure if this is a good proposal, but I saw a few cases where some
integration test methods in {{ITestDynamoDBMetadataStore}} timed out simply.
Specially, the one keeps failing me today is {{testAncestorOverwriteConflict}}.
I increase the timeout to 20s and it works for me happily. Am I using VPN and a
slow home network, I'm afraid so.
The time out rule, as inherited from base class {{HadoopTestBase}}, is 10s by
default. Though that 10s time out default value can be overridden in base class
via system property {{test.default.timeout}}, that's system wide affecting all
other tests. Changing that time out value for one test is no better than
overriding in this test {{ITestDynamoDBMetadataStore}} explicitly. I think
doubling it to 20s would not be very crazy considering we are testing against a
remote web service, create and destroy tables sometimes.
{code}
@Rule
public Timeout timeout = new Timeout(20 * 1000);
{code}
Thoughts?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16735) Make it clearer in config default that EnvironmentVariableCredentialsProvider supports AWS_SESSION_TOKEN
Mingliang Liu created HADOOP-16735:
--
Summary: Make it clearer in config default that
EnvironmentVariableCredentialsProvider supports AWS_SESSION_TOKEN
Key: HADOOP-16735
URL: https://issues.apache.org/jira/browse/HADOOP-16735
Project: Hadoop Common
Issue Type: Improvement
Components: documentation, fs/s3
Reporter: Mingliang Liu
Assignee: Mingliang Liu
In the great doc {{hadoop-aws/tools/hadoop-aws/index.html}}, user can find that
authenticating via the AWS Environment Variables supports session token.
However, the config description in core-default.xml does not make it clear.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16732) S3Guard to support encrypted DynamoDB table
Mingliang Liu created HADOOP-16732: -- Summary: S3Guard to support encrypted DynamoDB table Key: HADOOP-16732 URL: https://issues.apache.org/jira/browse/HADOOP-16732 Project: Hadoop Common Issue Type: Improvement Components: fs/s3 Reporter: Mingliang Liu S3Guard is not yet supporting [encrypted DynamoDB table|https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/encryption.tutorial.html]. We can provide an option to enable encrypted DynamoDB table so data at rest could be encrypted. S3Guard data in DynamoDB usually is not sensitive since it's the S3 namespace mirroring, but some times even this is a concern. By default it's not enabled. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16722) S3GuardTool to support FilterFileSystem
Mingliang Liu created HADOOP-16722:
--
Summary: S3GuardTool to support FilterFileSystem
Key: HADOOP-16722
URL: https://issues.apache.org/jira/browse/HADOOP-16722
Project: Hadoop Common
Issue Type: Improvement
Components: fs/s3
Reporter: Mingliang Liu
Currently S3GuardTool is operating against a S3AFileSystem implementation.
There are cases where the {{fs.hboss.fs.s3a.impl}} is a FilterFileSystem
wrapping another implementation. For example, [[HBASE-23314]] made
{{HBaseObjectStoreSemantics}} a {{FilterFileSystem}} from {{FileSystem}}.
S3GuardTool could use {{FilterFileSystem::getRawFileSystem}} method to get the
wrapped S3AFileSystem implementation. Without this support, a simple
S3GuardTool against HBOSS will get confusing error like "s3a://mybucket is not
a S3A file system".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16355) ZookeeperMetadataStore: Use Zookeeper as S3Guard backend store
Mingliang Liu created HADOOP-16355:
--
Summary: ZookeeperMetadataStore: Use Zookeeper as S3Guard backend
store
Key: HADOOP-16355
URL: https://issues.apache.org/jira/browse/HADOOP-16355
Project: Hadoop Common
Issue Type: Sub-task
Components: fs
Reporter: Mingliang Liu
When S3Guard was proposed, there are a couple of valid reasons to choose
DynamoDB as its default backend store: 0) seamless integration as part of AWS
ecosystem e.g. client library 1) it's a managed web service which is zero
operational cost, highly available and infinitely scalable 2) it's performant
with single digit latency 3) it's proven by Netflix's S3mper (not actively
maintained) and EMRFS (closed source and usage). As it's pluggable, it's
possible to implement {{MetadataStore}} with other backend store without
changing semantics, besides null and in-memory local ones.
Here we propose {{ZookeeperMetadataStore}} which uses Zookeeper as S3Guard
backend store. Its main motivation is to provide a new MetadataStore option
which:
# can be easily integrated as Zookeeper is heavily used in Hadoop community
# affordable performance as both client and Zookeeper ensemble are usually
"local" in a Hadoop cluster (ZK/HBase/Hive etc)
# removes DynamoDB dependency
Obviously all use cases will not prefer this to default DynamoDB store. For
e.g. ZK might not scale well if there are dozens of S3 buckets and each has
millions of objects.
Our use case is targeting HBase to store HFiles on S3 instead of HDFS. A total
solution for HBase on S3 must be HBOSS (see HBASE-22149) for recovering
atomicity of metadata operations like rename, and S3Guard for consistent
enumeration and access to object store bucket metadata. We would like to use
Zookeeper as backend store for both.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14601) Azure: Reuse ObjectMapper
Mingliang Liu created HADOOP-14601:
--
Summary: Azure: Reuse ObjectMapper
Key: HADOOP-14601
URL: https://issues.apache.org/jira/browse/HADOOP-14601
Project: Hadoop Common
Issue Type: Sub-task
Components: fa/azure
Reporter: Mingliang Liu
Assignee: Mingliang Liu
Currently there are a few places in {{hadoop-azure}} module that uses creates
{{ObjectMapper}} for each request/call. We should re-use the object mapper for
performance purpose.
The general caveat is about thread safety; I think the change will be safe
though.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14594) ITestS3AFileOperationCost::testFakeDirectoryDeletion to uncomment metric assertions
Mingliang Liu created HADOOP-14594: -- Summary: ITestS3AFileOperationCost::testFakeDirectoryDeletion to uncomment metric assertions Key: HADOOP-14594 URL: https://issues.apache.org/jira/browse/HADOOP-14594 Project: Hadoop Common Issue Type: Bug Reporter: Mingliang Liu Assignee: Mingliang Liu Per discussion [HADOOP-14255] and [HADOOP-13222], we can delete the TODO comment in tests for metric assertions. See the attached patch for more details. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14536) Update azure-storage sdk to version 5.2.0
Mingliang Liu created HADOOP-14536: -- Summary: Update azure-storage sdk to version 5.2.0 Key: HADOOP-14536 URL: https://issues.apache.org/jira/browse/HADOOP-14536 Project: Hadoop Common Issue Type: Improvement Components: fs/azure Affects Versions: 3.0.0-alpha3 Reporter: Mingliang Liu Update WASB driver to use the latest version (5.2.0) of SDK for Microsoft Azure Storage Clients. We are currently using version 4.2.0 of the SDK. Azure Storage Clients changes between 4.2 and 5.2: * Fixed Exists() calls on Shares and Directories to now populate metadata. This was already being done for Files. * Changed blob constants to support up to 256 MB on put blob for block blobs. The default value for put blob threshold has also been updated to half of the maximum, or 128 MB currently. * Fixed a bug that prevented setting content MD5 to true when creating a new file. * Fixed a bug where access conditions, options, and operation context were not being passed when calling openWriteExisting() on a page blob or a file. * Fixed a bug where an exception was being thrown on a range get of a blob or file when the options disableContentMD5Validation is set to false and useTransactionalContentMD5 is set to true and there is no overall MD5. * Fixed a bug where retries were happening immediately if a socket exception was thrown. * In CloudFileShareProperties, setShareQuota() no longer asserts in bounds. This check has been moved to create() and uploadProperties() in CloudFileShare. * Prefix support for listing files and directories. * Added support for setting public access when creating a blob container * The public access setting on a blob container is now a container property returned from downloadProperties. * Add Message now modifies the PopReceipt, Id, NextVisibleTime, InsertionTime, and ExpirationTime properties of its CloudQueueMessage parameter. * Populate content MD5 for range gets on Blobs and Files. * Added support in Page Blob for incremental copy. * Added large BlockBlob upload support. Blocks can now support sizes up to 100 MB. * Added a new, memory-optimized upload strategy for the upload* APIs. This algorithm only applies for blocks greater than 4MB and when storeBlobContentMD5 and Client-Side Encryption are disabled. * getQualifiedUri() has been deprecated for Blobs. Please use getSnapshotQualifiedUri() instead. This new function will return the blob including the snapshot (if present) and no SAS token. * getQualifiedStorageUri() has been deprecated for Blobs. Please use getSnapshotQualifiedStorageUri() instead. This new function will return the blob including the snapshot (if present) and no SAS token. * Fixed a bug where copying from a blob that included a SAS token and a snapshot ommitted the SAS token. * Fixed a bug in client-side encryption for tables that was preventing the Java client from decrypting entities encrypted with the .NET client, and vice versa. * Added support for server-side encryption. * Added support for getBlobReferenceFromServer methods on CloudBlobContainer to support retrieving a blob without knowing its type. * Fixed a bug in the retry policies where 300 status codes were being retried when they shouldn't be. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14516) Update WASB driver to use the latest version (5.2.0) of SDK for Microsoft Azure Storage Clients
[ https://issues.apache.org/jira/browse/HADOOP-14516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu resolved HADOOP-14516. Resolution: Duplicate Closing as duplicates. Please see [HADOOP-14490] and comment there. Thanks, > Update WASB driver to use the latest version (5.2.0) of SDK for Microsoft > Azure Storage Clients > --- > > Key: HADOOP-14516 > URL: https://issues.apache.org/jira/browse/HADOOP-14516 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Georgi Chalakov > > Update WASB driver to use the latest version (5.2.0) of SDK for Microsoft > Azure Storage Clients. We are currently using version 4.2.0 of the SDK. > Azure Storage Clients changes between 4.2 and 5.2: > * Fixed Exists() calls on Shares and Directories to now populate metadata. > This was already being done for Files. > * Changed blob constants to support up to 256 MB on put blob for block > blobs. The default value for put blob threshold has also been updated to half > of the maximum, or 128 MB currently. > * Fixed a bug that prevented setting content MD5 to true when creating a new > file. > * Fixed a bug where access conditions, options, and operation context were > not being passed when calling openWriteExisting() on a page blob or a file. > * Fixed a bug where an exception was being thrown on a range get of a blob > or file when the options disableContentMD5Validation is set to false and > useTransactionalContentMD5 is set to true and there is no overall MD5. > * Fixed a bug where retries were happening immediately if a socket exception > was thrown. > * In CloudFileShareProperties, setShareQuota() no longer asserts in bounds. > This check has been moved to create() and uploadProperties() in > CloudFileShare. > * Prefix support for listing files and directories. > * Added support for setting public access when creating a blob container > * The public access setting on a blob container is now a container property > returned from downloadProperties. > * Add Message now modifies the PopReceipt, Id, NextVisibleTime, > InsertionTime, and ExpirationTime properties of its CloudQueueMessage > parameter. > * Populate content MD5 for range gets on Blobs and Files. > * Added support in Page Blob for incremental copy. > * Added large BlockBlob upload support. Blocks can now support sizes up to > 100 MB. > * Added a new, memory-optimized upload strategy for the upload* APIs. This > algorithm only applies for blocks greater than 4MB and when > storeBlobContentMD5 and Client-Side Encryption are disabled. > * getQualifiedUri() has been deprecated for Blobs. Please use > getSnapshotQualifiedUri() instead. This new function will return the blob > including the snapshot (if present) and no SAS token. > * getQualifiedStorageUri() has been deprecated for Blobs. Please use > getSnapshotQualifiedStorageUri() instead. This new function will return the > blob including the snapshot (if present) and no SAS token. > * Fixed a bug where copying from a blob that included a SAS token and a > snapshot ommitted the SAS token. > * Fixed a bug in client-side encryption for tables that was preventing the > Java client from decrypting entities encrypted with the .NET client, and vice > versa. > * Added support for server-side encryption. > * Added support for getBlobReferenceFromServer methods on CloudBlobContainer > to support retrieving a blob without knowing its type. > * Fixed a bug in the retry policies where 300 status codes were being > retried when they shouldn't be. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14500) Azure: TestFileSystemOperationExceptionHandling{,MultiThreaded} fails
Mingliang Liu created HADOOP-14500:
--
Summary: Azure:
TestFileSystemOperationExceptionHandling{,MultiThreaded} fails
Key: HADOOP-14500
URL: https://issues.apache.org/jira/browse/HADOOP-14500
Project: Hadoop Common
Issue Type: Bug
Components: fs/azure, test
Reporter: Mingliang Liu
The following test fails:
{code}
TestFileSystemOperationExceptionHandling.testSingleThreadBlockBlobSeekScenario
Expected exception:
java.io.FileNotFoundExceptionTestFileSystemOperationsExceptionHandlingMultiThreaded.testMultiThreadBlockBlobSeekScenario
Expected exception: java.io.FileNotFoundException
{code}
I did early analysis and found [HADOOP-14478] maybe the reason. I think we can
fix the test itself here.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14498) HADOOP_OPTIONAL_TOOLS not parsed correctly
Mingliang Liu created HADOOP-14498:
--
Summary: HADOOP_OPTIONAL_TOOLS not parsed correctly
Key: HADOOP-14498
URL: https://issues.apache.org/jira/browse/HADOOP-14498
Project: Hadoop Common
Issue Type: Bug
Components: build
Affects Versions: 3.0.0-alpha1
Reporter: Mingliang Liu
# This will make hadoop-azure not show up in the hadoop classpath, though both
hadoop-aws and hadoop-azure-datalake are in the
classpath.{code:title=hadoop-env.sh}
export HADOOP_OPTIONAL_TOOLS="hadoop-azure,hadoop-aws,hadoop-azure-datalake"
{code}
# And if we put only hadoop-azure and hadoop-aws, both of them are shown in the
classpath.
{code:title=hadoop-env.sh}
export HADOOP_OPTIONAL_TOOLS="hadoop-azure,hadoop-aws"
{code}
This makes me guess that, while parsing the {{HADOOP_OPTIONAL_TOOLS}}, we make
some assumptions that hadoop tool modules have a single "-" in names, and the
_hadoop-azure-datalake_ overrides the _hadoop-azure_. Or any other assumptions
about the {{${project.artifactId\}}}?
Ping [~aw].
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14491) Azure has messed doc structure
Mingliang Liu created HADOOP-14491: -- Summary: Azure has messed doc structure Key: HADOOP-14491 URL: https://issues.apache.org/jira/browse/HADOOP-14491 Project: Hadoop Common Issue Type: Improvement Components: documentation, fs/azure Reporter: Mingliang Liu Assignee: Mingliang Liu # The _WASB Secure mode and configuration_ and _Authorization Support in WASB_ sections are missing in the navigation # _Authorization Support in WASB_ should be header level 3 instead of level 2 # Some of the code format is not specified # Sample code indent not unified. Let's use the auto-generated navigation instead of manually updating it, just as other documents. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14490) Upgrade azure-storage sdk version
Mingliang Liu created HADOOP-14490:
--
Summary: Upgrade azure-storage sdk version
Key: HADOOP-14490
URL: https://issues.apache.org/jira/browse/HADOOP-14490
Project: Hadoop Common
Issue Type: Improvement
Reporter: Mingliang Liu
As required by [HADOOP-14478], we're expecting the {{BlobInputStream}} to
support advanced {{readFully()}} by taking hints of mark. This can only be done
by means of sdk version bump.
cc: [~rajesh.balamohan].
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14472) Azure: TestReadAndSeekPageBlobAfterWrite fails intermittently
Mingliang Liu created HADOOP-14472:
--
Summary: Azure: TestReadAndSeekPageBlobAfterWrite fails
intermittently
Key: HADOOP-14472
URL: https://issues.apache.org/jira/browse/HADOOP-14472
Project: Hadoop Common
Issue Type: Bug
Components: fs/azure
Reporter: Mingliang Liu
Reported by [HADOOP-14461]
{code}
testManySmallWritesWithHFlush(org.apache.hadoop.fs.azure.TestReadAndSeekPageBlobAfterWrite)
Time elapsed: 1.051 sec <<< FAILURE!
java.lang.AssertionError: hflush duration of 13, less than minimum expected of
20
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.fs.azure.TestReadAndSeekPageBlobAfterWrite.writeAndReadOneFile(TestReadAndSeekPageBlobAfterWrite.java:286)
at
org.apache.hadoop.fs.azure.TestReadAndSeekPageBlobAfterWrite.testManySmallWritesWithHFlush(TestReadAndSeekPageBlobAfterWrite.java:247)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14460) Azure: fs.azure.account.key.youraccount.blob.core.windows.net -> fs.azure.account.key.youraccount
Mingliang Liu created HADOOP-14460:
--
Summary: Azure:
fs.azure.account.key.youraccount.blob.core.windows.net ->
fs.azure.account.key.youraccount
Key: HADOOP-14460
URL: https://issues.apache.org/jira/browse/HADOOP-14460
Project: Hadoop Common
Issue Type: Bug
Components: documentation, fs/azure
Reporter: Mingliang Liu
Assignee: Mingliang Liu
In {{SimpleKeyProvider}}, we have following code for getting the key:
{code}
protected static final String KEY_ACCOUNT_KEY_PREFIX =
"fs.azure.account.key.";
...
protected String getStorageAccountKeyName(String accountName) {
return KEY_ACCOUNT_KEY_PREFIX + accountName;
}
{code}
While in documentation {{index.md}}, we have:
{code}
fs.azure.account.key.youraccount.blob.core.windows.net
YOUR ACCESS KEY
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14458) TestAliyunOSSFileSystemContract missing imports/
Mingliang Liu created HADOOP-14458:
--
Summary: TestAliyunOSSFileSystemContract missing imports/
Key: HADOOP-14458
URL: https://issues.apache.org/jira/browse/HADOOP-14458
Project: Hadoop Common
Issue Type: Bug
Components: fs/oss, test
Reporter: Mingliang Liu
Assignee: Mingliang Liu
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
(default-testCompile) on project hadoop-aliyun: Compilation failure:
Compilation failure:
[ERROR]
/Users/mliu/Workspace/hadoop/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSFileSystemContract.java:[71,5]
cannot find symbol
[ERROR] symbol: method assertTrue(java.lang.String,boolean)
[ERROR] location: class
org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract
[ERROR]
/Users/mliu/Workspace/hadoop/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSFileSystemContract.java:[90,5]
cannot find symbol
[ERROR] symbol: method assertTrue(java.lang.String,boolean)
[ERROR] location: class
org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract
[ERROR]
/Users/mliu/Workspace/hadoop/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSFileSystemContract.java:[91,5]
cannot find symbol
[ERROR] symbol: method assertTrue(java.lang.String,boolean)
[ERROR] location: class
org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract
[ERROR]
/Users/mliu/Workspace/hadoop/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSFileSystemContract.java:[92,5]
cannot find symbol
[ERROR] symbol: method assertTrue(java.lang.String,boolean)
[ERROR] location: class
org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract
[ERROR]
/Users/mliu/Workspace/hadoop/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSFileSystemContract.java:[93,5]
cannot find symbol
[ERROR] symbol: method assertTrue(java.lang.String,boolean)
[ERROR] location: class
org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract
[ERROR]
/Users/mliu/Workspace/hadoop/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSFileSystemContract.java:[95,5]
cannot find symbol
[ERROR] symbol: method assertTrue(java.lang.String,boolean)
[ERROR] location: class
org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract
[ERROR]
/Users/mliu/Workspace/hadoop/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSFileSystemContract.java:[96,5]
cannot find symbol
[ERROR] symbol: method assertTrue(java.lang.String,boolean)
[ERROR] location: class
org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract
[ERROR]
/Users/mliu/Workspace/hadoop/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSFileSystemContract.java:[98,5]
cannot find symbol
[ERROR] symbol: method assertTrue(java.lang.String,boolean)
[ERROR] location: class
org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract
[ERROR]
/Users/mliu/Workspace/hadoop/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSFileSystemContract.java:[99,5]
cannot find symbol
[ERROR] symbol: method assertTrue(java.lang.String,boolean)
[ERROR] location: class
org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract
[ERROR]
/Users/mliu/Workspace/hadoop/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSFileSystemContract.java:[115,7]
cannot find symbol
[ERROR] symbol: method fail(java.lang.String)
[ERROR] location: class
org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract
[ERROR]
/Users/mliu/Workspace/hadoop/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSFileSystemContract.java:[129,7]
cannot find symbol
[ERROR] symbol: method fail(java.lang.String)
[ERROR] location: class
org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract
[ERROR]
/Users/mliu/Workspace/hadoop/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSFileSystemContract.java:[143,7]
cannot find symbol
[ERROR] symbol: method fail(java.lang.String)
[ERROR] location: class
org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract
[ERROR]
/Users/mliu/Workspace/hadoop/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSFileSystemContract.java:[163,7]
cannot find symbol
[ERROR] symbol: method fail(java.lang.String)
[ERROR] location: class
org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract
[ERROR]
[jira] [Created] (HADOOP-14438) Make ADLS doc of setting up client key up to date
Mingliang Liu created HADOOP-14438:
--
Summary: Make ADLS doc of setting up client key up to date
Key: HADOOP-14438
URL: https://issues.apache.org/jira/browse/HADOOP-14438
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/adl
Reporter: Mingliang Liu
In the doc {{hadoop-tools/hadoop-azure-datalake/src/site/markdown/index.md}},
we have such a statement:
{code:title=Note down the properties you will need to auth}
...
- Resource: Always https://management.core.windows.net/ , for all customers
{code}
Is the {{Resource}} useful here? It seems not necessary to me.
{code:title=Adding the service principal to your ADL Account}
- ...
- Select Users under Settings
...
{code}
According to the portal, it should be "Access control (IAM)" under "Settings"
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14363) Inconsistent default block location in FileSystem javadoc
Mingliang Liu created HADOOP-14363:
--
Summary: Inconsistent default block location in FileSystem javadoc
Key: HADOOP-14363
URL: https://issues.apache.org/jira/browse/HADOOP-14363
Project: Hadoop Common
Issue Type: Bug
Components: fs
Reporter: Mingliang Liu
Assignee: Chen Liang
Priority: Trivial
{code:title=FileSystem::getFileBlockLocations()}
..
*
* The default implementation returns an array containing one element:
*
* BlockLocation( { "localhost:50010" }, { "localhost" }, 0, file.getLen())
* >
*
* @param file FilesStatus to get data from
* @param start offset into the given file
* @param len length for which to get locations for
* @throws IOException IO failure
*/
public BlockLocation[] getFileBlockLocations(FileStatus file,
long start, long len) throws IOException {
...
String[] name = {"localhost:9866"};
String[] host = {"localhost"};
return new BlockLocation[] {
new BlockLocation(name, host, 0, file.getLen()) };
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14349) Rename ADLS CONTRACT_ENABLE_KEY
Mingliang Liu created HADOOP-14349: -- Summary: Rename ADLS CONTRACT_ENABLE_KEY Key: HADOOP-14349 URL: https://issues.apache.org/jira/browse/HADOOP-14349 Project: Hadoop Common Issue Type: Sub-task Components: fs/adl Affects Versions: 2.8.0 Reporter: Mingliang Liu dfs.adl.test.contract.enable -> fs.adl.test.contract.enable -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14165) Add S3Guard.dirListingUnion in S3AFileSystem#listFiles, listLocatedStatus
[
https://issues.apache.org/jira/browse/HADOOP-14165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu resolved HADOOP-14165.
Resolution: Duplicate
Resolving as duplicate to [HADOOP-13926] and [HADOOP-14266].
> Add S3Guard.dirListingUnion in S3AFileSystem#listFiles, listLocatedStatus
> -
>
> Key: HADOOP-14165
> URL: https://issues.apache.org/jira/browse/HADOOP-14165
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Rajesh Balamohan
>Priority: Minor
>
> {{S3Guard::dirListingUnion}} merges information from backing store and DDB to
> create consistent view. This needs to be added in
> {{S3AFileSystem::listFiles}} and {{S3AFileSystem::listLocatedStatus}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14215) DynamoDB client should waitForActive on existing tables
[ https://issues.apache.org/jira/browse/HADOOP-14215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu resolved HADOOP-14215. Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: HADOOP-13345 Committed to feature branch. Thanks for your contribution [~mackrorysd]. Thanks [~fabbri] for review and [~rajesh.balamohan] for offline discussion. > DynamoDB client should waitForActive on existing tables > --- > > Key: HADOOP-14215 > URL: https://issues.apache.org/jira/browse/HADOOP-14215 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Critical > Fix For: HADOOP-13345 > > Attachments: HADOOP-14215-HADOOP-13345.000.patch, > HADOOP-14215-HADOOP-13345.001.patch, HADOOP-14215-HADOOP-13345.002.patch > > > I saw a case where 2 separate applications tried to use the same > non-pre-existing table with table.create = true at about the same time. One > failed with a ResourceInUse exception. If a table does not exist, we attempt > to create it and then wait for it to enter the active state. If another jumps > in in the middle of that, the table may exist, thus bypassing our call to > waitForActive(), and then try to use the table immediately. > While we're at it, let's also make sure that the race condition where a table > might get created between checking if it exists and attempting to create it > is handled gracefully. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14282) S3Guard: DynamoDBMetadata::prune() should self interrupt correctly
Mingliang Liu created HADOOP-14282: -- Summary: S3Guard: DynamoDBMetadata::prune() should self interrupt correctly Key: HADOOP-14282 URL: https://issues.apache.org/jira/browse/HADOOP-14282 Project: Hadoop Common Issue Type: Sub-task Reporter: Mingliang Liu Assignee: Mingliang Liu -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14266) S3Guard: S3AFileSystem::listFiles() to employ MetadataStore
Mingliang Liu created HADOOP-14266:
--
Summary: S3Guard: S3AFileSystem::listFiles() to employ
MetadataStore
Key: HADOOP-14266
URL: https://issues.apache.org/jira/browse/HADOOP-14266
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: HADOOP-13345
Reporter: Mingliang Liu
Similar to [HADOOP-13926], this is to track the effort of employing
MetadataStore in {{S3AFileSystem::listFiles()}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14255) S3A to delete unnecessary fake directory objects in mkdirs()
Mingliang Liu created HADOOP-14255:
--
Summary: S3A to delete unnecessary fake directory objects in
mkdirs()
Key: HADOOP-14255
URL: https://issues.apache.org/jira/browse/HADOOP-14255
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Reporter: Mingliang Liu
Assignee: Mingliang Liu
In S3AFileSystem, as an optimization, we delete unnecessary fake directory
objects if that directory contains at least one (nested) file. That is done in
closing stream of newly created file. However, if the directory becomes
non-empty after we just create an empty subdirectory, we do not delete its fake
directory object though that fake directory object becomes "unnecessary".
So in {{S3AFileSystem::mkdirs()}}, we have a pending TODO:
{quote}
// TODO: If we have created an empty file at /foo/bar and we then call
// mkdirs for /foo/bar/baz/roo what happens to the empty file /foo/bar/?
private boolean innerMkdirs(Path p, FsPermission permission)
{quote}
This JIRA is to fix the TODO: provide consistent behavior for a fake directory
object between its nested subdirectory and nested file by deleting it.
See related discussion in [HADOOP-14236].
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14248) Retire SharedInstanceProfileCredentialsProvider after AWS SDK upgrade
Mingliang Liu created HADOOP-14248:
--
Summary: Retire SharedInstanceProfileCredentialsProvider after AWS
SDK upgrade
Key: HADOOP-14248
URL: https://issues.apache.org/jira/browse/HADOOP-14248
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.0.0-alpha3
Reporter: Mingliang Liu
Assignee: Mingliang Liu
This is from the discussion in [HADOOP-13050].
So [HADOOP-13727] added the SharedInstanceProfileCredentialsProvider, which
effectively reduces high number of connections to EC2 Instance Metadata Service
caused by InstanceProfileCredentialsProvider. That patch, in order to prevent
the throttling problem, defined new class
{{SharedInstanceProfileCredentialsProvider}} as a subclass of
{{InstanceProfileCredentialsProvider}}, which enforces creation of only a
single instance.
Per [HADOOP-13050], we upgraded the AWS Java SDK. Since then, the
{{InstanceProfileCredentialsProvider}} in SDK code internally enforces a
singleton. That confirms that our effort in [HADOOP-13727] makes 100% sense.
Meanwhile, {{SharedInstanceProfileCredentialsProvider}} can retire gracefully
in trunk branch.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14247) FileContextMainOperationsBaseTest should clean up test root path
Mingliang Liu created HADOOP-14247: -- Summary: FileContextMainOperationsBaseTest should clean up test root path Key: HADOOP-14247 URL: https://issues.apache.org/jira/browse/HADOOP-14247 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.8.0 Reporter: Mingliang Liu Assignee: Mingliang Liu -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14236) S3Guard: S3AFileSystem::rename() should move non-listed sub-directory entries in metadata store
Mingliang Liu created HADOOP-14236:
--
Summary: S3Guard: S3AFileSystem::rename() should move non-listed
sub-directory entries in metadata store
Key: HADOOP-14236
URL: https://issues.apache.org/jira/browse/HADOOP-14236
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Reporter: Mingliang Liu
Assignee: Mingliang Liu
After running integration test {{ITestS3AFileSystemContract}}, I found the
following items are not cleaned up in DynamoDB:
{code}
parent=/mliu-s3guard/user/mliu/s3afilesystemcontract/testRenameDirectoryAsExisting/dir,
child=subdir
parent=/mliu-s3guard/user/mliu/s3afilesystemcontract/testRenameDirectoryAsExistingNew/newdir/subdir,
child=file2
{code}
At first I thought it’s similar to [HADOOP-14226] or [HADOOP-14227], and we
need to be careful when cleaning up test data.
Then I found it’s a bug in the code of integrating S3Guard with S3AFileSystem:
for rename we miss sub-directory items to put (dest) and delete (src). The
reason is that in S3A, we delete those fake directory objects if they are not
necessary, e.g. non-empty. So when we list the objects to rename, the object
summaries will only return _file_ objects. This has two consequences after
rename:
# there will be left items for src path in metadata store - left-overs will
confuse {{get(Path)}} which should return null
# we are not persisting the whole subtree for dest path to metadata store -
this will break the DynamoDBMetadataStore invariant: _if a path exists, all its
ancestors will also exist in the table_.
Existing tests are not complaining about this though. If this is a real bug,
let’s address it here.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14227) S3Guard: ITestS3AConcurrentOps is not cleaning up test data
Mingliang Liu created HADOOP-14227:
--
Summary: S3Guard: ITestS3AConcurrentOps is not cleaning up test
data
Key: HADOOP-14227
URL: https://issues.apache.org/jira/browse/HADOOP-14227
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Reporter: Mingliang Liu
Assignee: Mingliang Liu
After running {{ITestS3AConcurrentOps}}, the test data is not cleanup in
DynamoDB. There are two reasons:
# The {{ITestS3AConcurrentOps::teardown()}} method is not calling super
teardown() method to clean up the default test directory.
# The {{auxFs}} is not S3Guard aware even though the {{fs}} to test is. That's
because the {{auxFs}} is creating a new Configuration object without patching
in S3Guard options (via {{maybeEnableS3Guard(conf);}}).
This JIRA is to clean up the data after test.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14226) S3Guard: ITestDynamoDBMetadataStoreScale is not cleaning up test data
Mingliang Liu created HADOOP-14226:
--
Summary: S3Guard: ITestDynamoDBMetadataStoreScale is not cleaning
up test data
Key: HADOOP-14226
URL: https://issues.apache.org/jira/browse/HADOOP-14226
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: HADOOP-13345
Reporter: Mingliang Liu
Assignee: Mingliang Liu
After running {{ITestDynamoDBMetadataStoreScale}}, the test data is not cleaned
up. There is a call to {{clearMetadataStore(ms, count);}} in the finally clause
though. The reason is that, the internally called method
{{DynamoDBMetadataStore::deleteSubtree()}} is assuming there should be an item
for the parent dest path:
{code}
parent=/fake-bucket, child=moved-here, is_dir=true
{code}
In DynamoDBMetadataStore implementation, we assume that _if a path exists, all
its ancestors will also exist in the table_. We need to pre-create dest path to
maintain this invariant so that test data can be cleaned up successfully.
I think there may be other tests with the same problem. Let's identify/address
them separately.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14214) DomainSocketWatcher::add()/delete() should not self interrupt while looping await()
Mingliang Liu created HADOOP-14214:
--
Summary: DomainSocketWatcher::add()/delete() should not self
interrupt while looping await()
Key: HADOOP-14214
URL: https://issues.apache.org/jira/browse/HADOOP-14214
Project: Hadoop Common
Issue Type: Bug
Reporter: Mingliang Liu
Assignee: Mingliang Liu
Our hive team found a TPCDS job whose queries running on LLAP seem to be
getting stuck. Dozens of threads were waiting for the
{{DfsClientShmManager::lock}}, as following jstack:
{code}
Thread 251 (IO-Elevator-Thread-5):
State: WAITING
Blocked count: 3871
Wtaited count: 4565
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@16ead198
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1976)
org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager.allocSlot(DfsClientShmManager.java:255)
org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager.allocSlot(DfsClientShmManager.java:434)
org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.allocShmSlot(ShortCircuitCache.java:1017)
org.apache.hadoop.hdfs.BlockReaderFactory.createShortCircuitReplicaInfo(BlockReaderFactory.java:476)
org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.create(ShortCircuitCache.java:784)
org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.fetchOrCreate(ShortCircuitCache.java:718)
org.apache.hadoop.hdfs.BlockReaderFactory.getBlockReaderLocal(BlockReaderFactory.java:422)
org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:333)
org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1181)
org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:1118)
org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1478)
org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1441)
org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:121)
org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:111)
org.apache.orc.impl.RecordReaderUtils$DefaultDataReader.readStripeFooter(RecordReaderUtils.java:166)
org.apache.hadoop.hive.llap.io.metadata.OrcStripeMetadata.(OrcStripeMetadata.java:64)
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.readStripesMetadata(OrcEncodedDataReader.java:622)
{code}
The thread that is expected to signal those threads is calling
{{DomainSocketWatcher::add()}} method, but it gets stuck there dealing with
InterruptedException infinitely. The jstack is like:
{code}
Thread 44417 (TezTR-257387_2840_12_10_52_0):
State: RUNNABLE
Blocked count: 3
Wtaited count: 5
Stack:
java.lang.Throwable.fillInStackTrace(Native Method)
java.lang.Throwable.fillInStackTrace(Throwable.java:783)
java.lang.Throwable.(Throwable.java:250)
java.lang.Exception.(Exception.java:54)
java.lang.InterruptedException.(InterruptedException.java:57)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2034)
org.apache.hadoop.net.unix.DomainSocketWatcher.add(DomainSocketWatcher.java:325)
org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager.allocSlot(DfsClientShmManager.java:266)
org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager.allocSlot(DfsClientShmManager.java:434)
org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.allocShmSlot(ShortCircuitCache.java:1017)
org.apache.hadoop.hdfs.BlockReaderFactory.createShortCircuitReplicaInfo(BlockReaderFactory.java:476)
org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.create(ShortCircuitCache.java:784)
org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.fetchOrCreate(ShortCircuitCache.java:718)
org.apache.hadoop.hdfs.BlockReaderFactory.getBlockReaderLocal(BlockReaderFactory.java:422)
org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:333)
org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1181)
org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:1118)
org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1478)
org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1441)
org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:121)
{code}
The whole job makes no progress because of this.
The thread in {{DomainSocketWatcher::add()}} is expected to eventually break
the while loop where it waits for the newly added entry being deleted by
another thread. However, if this thread is ever interrupted, chances are that
it will hold the lock forever so
[jira] [Created] (HADOOP-14194) Alyun OSS should not use empty endpoint as default
Mingliang Liu created HADOOP-14194:
--
Summary: Alyun OSS should not use empty endpoint as default
Key: HADOOP-14194
URL: https://issues.apache.org/jira/browse/HADOOP-14194
Project: Hadoop Common
Issue Type: Bug
Components: fs/oss
Reporter: Mingliang Liu
Assignee: Xiaobing Zhou
In {{AliyunOSSFileSystemStore::initialize()}}, it retrieves the endPoint and
using empty string as a default value.
{code}
String endPoint = conf.getTrimmed(ENDPOINT_KEY, "");
{code}
The plain value without validation is passed to OSSClient. If the endPoint is
not provided (empty string) or the endPoint is not valid, users will get
exception from Aliyun OSS sdk with raw exception message like:
{code}
java.lang.IllegalArgumentException: java.net.URISyntaxException: Expected
authority at index 8: https://
at com.aliyun.oss.OSSClient.toURI(OSSClient.java:359)
at com.aliyun.oss.OSSClient.setEndpoint(OSSClient.java:313)
at com.aliyun.oss.OSSClient.(OSSClient.java:297)
at
org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.initialize(AliyunOSSFileSystemStore.java:134)
at
org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem.initialize(AliyunOSSFileSystem.java:272)
at
org.apache.hadoop.fs.aliyun.oss.AliyunOSSTestUtils.createTestFileSystem(AliyunOSSTestUtils.java:63)
at
org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract.setUp(TestAliyunOSSFileSystemContract.java:47)
at junit.framework.TestCase.runBare(TestCase.java:139)
at junit.framework.TestResult$1.protect(TestResult.java:122)
at junit.framework.TestResult.runProtected(TestResult.java:142)
at junit.framework.TestResult.run(TestResult.java:125)
at junit.framework.TestCase.run(TestCase.java:129)
at junit.framework.TestSuite.runTest(TestSuite.java:255)
at junit.framework.TestSuite.run(TestSuite.java:250)
at
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:237)
at
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
Caused by: java.net.URISyntaxException: Expected authority at index 8: https://
at java.net.URI$Parser.fail(URI.java:2848)
at java.net.URI$Parser.failExpecting(URI.java:2854)
at java.net.URI$Parser.parseHierarchical(URI.java:3102)
at java.net.URI$Parser.parse(URI.java:3053)
at java.net.URI.(URI.java:588)
at com.aliyun.oss.OSSClient.toURI(OSSClient.java:357)
{code}
Let's check endPoint is not null or empty, catch the IllegalArgumentException
and log it, wrapping the exception with clearer message stating the
misconfiguration in endpoint or credentials.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14192) Aliyun OSS FileSystem contract test should implement getTestBaseDir()
Mingliang Liu created HADOOP-14192:
--
Summary: Aliyun OSS FileSystem contract test should implement
getTestBaseDir()
Key: HADOOP-14192
URL: https://issues.apache.org/jira/browse/HADOOP-14192
Project: Hadoop Common
Issue Type: Bug
Components: fs/oss
Reporter: Mingliang Liu
Assignee: Mingliang Liu
[HADOOP-14170] is the recent effort of improving the file system contract tests
{{FileSystemContractBaseTest}}, which make {{path()}} method final and add a
new method {{getTestBaseDir()}} for subclasses to implement. Aliyun OSS should
override that as it uses unique directory (naming with fork id) for supporting
parallel tests. Plus, the current {{testWorkingDirectory}} needs not override
per changes in {{FileSystemContractBaseTest}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13959) S3guard: replace dynamo.describe() call in init with more efficient query
[
https://issues.apache.org/jira/browse/HADOOP-13959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu resolved HADOOP-13959.
Resolution: Fixed
As [HADOOP-13985] is committed, I think we can resolve this one. Feel free to
re-open if necessary.
> S3guard: replace dynamo.describe() call in init with more efficient query
> -
>
> Key: HADOOP-13959
> URL: https://issues.apache.org/jira/browse/HADOOP-13959
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Steve Loughran
>Assignee: Mingliang Liu
>Priority: Minor
>
> HADOOP-13908 adds initialization when a table isn't created, using the
> {{describe()}} call.
> AWS document this as inefficient, and throttle it. We should be able to get
> away with a simple table lookup as the probe
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14180) FileSystem contract tests to replace JUnit 3 with 4
Mingliang Liu created HADOOP-14180:
--
Summary: FileSystem contract tests to replace JUnit 3 with 4
Key: HADOOP-14180
URL: https://issues.apache.org/jira/browse/HADOOP-14180
Project: Hadoop Common
Issue Type: Bug
Components: fs
Reporter: Mingliang Liu
This is from discussion in [HADOOP-14170], as Steve commented:
{quote}
...it's time to move this to JUnit 4, annotate all tests with @test, and make
the test cases skip if they don't have the test FS defined. JUnit 3 doesn't
support Assume, so when I do test runs without the s3n or s3 fs specced, I get
lots of errors I just ignore.
...Move to Junit 4, and, in our own code, find everywhere we've subclassed a
method to make the test a no-op, and insert an Assume.assumeTrue(false) in
there so they skip properly.
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14170) FileSystemContractBaseTest is not cleaning up test directory clearly
Mingliang Liu created HADOOP-14170:
--
Summary: FileSystemContractBaseTest is not cleaning up test
directory clearly
Key: HADOOP-14170
URL: https://issues.apache.org/jira/browse/HADOOP-14170
Project: Hadoop Common
Issue Type: Bug
Components: fs
Reporter: Mingliang Liu
Assignee: Mingliang Liu
In {{FileSystemContractBaseTest::tearDown()}} method, it cleans up the
{{path("/test")}} directory, which will be qualified as {{/test}} (against root
instead of working directory because it's absolute):
{code}
@Override
protected void tearDown() throws Exception {
try {
if (fs != null) {
fs.delete(path("/test"), true);
}
} catch (IOException e) {
LOG.error("Error deleting /test: " + e, e);
}
}
{code}
But in the test, it uses {{path("test")}} sometimes, which will be made
qualified against the working directory (e.g. {{/user/bob/test}}).
This makes some tests fail intermittently, e.g. {{ITestS3AFileSystemContract}}.
Also see the discussion in [HADOOP-13934].
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14153) ADL module has messed doc structure
Mingliang Liu created HADOOP-14153: -- Summary: ADL module has messed doc structure Key: HADOOP-14153 URL: https://issues.apache.org/jira/browse/HADOOP-14153 Project: Hadoop Common Issue Type: Bug Components: fs/adl Reporter: Mingliang Liu Assignee: Mingliang Liu -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-14129) ITestS3ACredentialsInURL sometimes fails
[ https://issues.apache.org/jira/browse/HADOOP-14129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu reopened HADOOP-14129: > ITestS3ACredentialsInURL sometimes fails > > > Key: HADOOP-14129 > URL: https://issues.apache.org/jira/browse/HADOOP-14129 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: HADOOP-13345 >Reporter: Sean Mackrory >Assignee: Sean Mackrory > Fix For: HADOOP-13345 > > Attachments: HADOOP-14129-HADOOP-13345.001.patch, > HADOOP-14129-HADOOP-13345.002.patch > > > This test sometimes fails. I believe it's expected that DynamoDB doesn't have > access to the credentials if they're embedded in the URL instead of the > configuration (and IMO that's fine - since the functionality hasn't been in > previous releases and since we want to discourage this practice especially > now that there are better alternatives). Weirdly, I only sometimes get this > failure on the HADOOP-13345 branch. But if the problem turns out to be what I > think it is, a simple Assume should fix it. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14135) Remove URI parameter in AWSCredentialProvider constructors
Mingliang Liu created HADOOP-14135:
--
Summary: Remove URI parameter in AWSCredentialProvider constructors
Key: HADOOP-14135
URL: https://issues.apache.org/jira/browse/HADOOP-14135
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Mingliang Liu
Assignee: Mingliang Liu
This was from comment in [HADOOP-13252].
It looks like the URI parameter is not needed for our AWSCredentialProvider
constructors. This was useful because we relied on URI parameter for retrieving
user:pass. Now in binding URIs, we have
{code}
279 S3xLoginHelper.Login creds = getAWSAccessKeys(binding, conf);
280 credentials.add(new BasicAWSCredentialsProvider(
281 creds.getUser(), creds.getPassword()));
{code}
This way, we only need configuration object (if necessary) for all
AWSCredentialProvider implementations. The benefit is that, if we create
AWSCredentialProvider list for DynamoDB, we don't have to pass down the
associated file system URI. This might be useful to S3Guard tools.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14130) Simplify DynamoDBClientFactory for creating Amazon DynamoDB clients
Mingliang Liu created HADOOP-14130:
--
Summary: Simplify DynamoDBClientFactory for creating Amazon
DynamoDB clients
Key: HADOOP-14130
URL: https://issues.apache.org/jira/browse/HADOOP-14130
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Reporter: Mingliang Liu
Assignee: Mingliang Liu
So, we are using deprecated {{AmazonDynamoDBClient}} class to create a DynamoDB
client instead of the recommended builder. We had discussion in [HADOOP-13345]
for preferring region to endpoints for user to specify the DynamoDB region (if
associated S3 region is unknown or different). We have reported inconsistent
behavior if endpoint and S3 region are different in [HADOOP-14027]. We also
noticed that {{DynamoDBMetadataStore}} may sometimes logs nonsense region. And
in [HADOOP-13252], we also have feelings that file system URI is not needed to
create a {{AWSCredentialProvider}}. Resultantly we don't need to pass down file
system URI for creating a DynamoDB client.
So this JIRA is to change this, best effort.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13994) explicitly declare the commons-lang3 dependency as 3.4
[ https://issues.apache.org/jira/browse/HADOOP-13994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu resolved HADOOP-13994. Resolution: Fixed > explicitly declare the commons-lang3 dependency as 3.4 > -- > > Key: HADOOP-13994 > URL: https://issues.apache.org/jira/browse/HADOOP-13994 > Project: Hadoop Common > Issue Type: Bug > Components: build, fs/azure, fs/s3 >Affects Versions: 3.0.0-alpha2, HADOOP-13345 >Reporter: Steve Loughran >Assignee: Steve Loughran > Fix For: HADOOP-13345 > > Attachments: HADOOP-13994-HADOOP-13345-001.patch, > HADOOP-13994-HADOOP-13445-001.patch > > > Other people aren't seeing this (yet?), but unless you explicitly exclude v > 3.4 of commons-lang3 from the azure build (which HADOOP-13660 does), then the > dependency declaration of commons-lang3 v 3.3.2 is creating a resolution > conflict. That's a dependency only needed for the local dynamodb & tests. > I propose to fix this in s3guard by explicitly declaring the version used in > the tests to be that of the azure-storage one, excluding that you get for > free. It doesn't impact anything shipped in production, but puts the hadoop > build in control of what versions of commons-lang are coming in everywhere by > way of the commons-config version declared in hadoop-common -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-13994) explicitly declare the commons-lang3 dependency as 3.4
[ https://issues.apache.org/jira/browse/HADOOP-13994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu reopened HADOOP-13994: > explicitly declare the commons-lang3 dependency as 3.4 > -- > > Key: HADOOP-13994 > URL: https://issues.apache.org/jira/browse/HADOOP-13994 > Project: Hadoop Common > Issue Type: Bug > Components: build, fs/azure, fs/s3 >Affects Versions: 3.0.0-alpha2, HADOOP-13345 >Reporter: Steve Loughran >Assignee: Steve Loughran > Fix For: HADOOP-13345 > > Attachments: HADOOP-13994-HADOOP-13345-001.patch, > HADOOP-13994-HADOOP-13445-001.patch > > > Other people aren't seeing this (yet?), but unless you explicitly exclude v > 3.4 of commons-lang3 from the azure build (which HADOOP-13660 does), then the > dependency declaration of commons-lang3 v 3.3.2 is creating a resolution > conflict. That's a dependency only needed for the local dynamodb & tests. > I propose to fix this in s3guard by explicitly declaring the version used in > the tests to be that of the azure-storage one, excluding that you get for > free. It doesn't impact anything shipped in production, but puts the hadoop > build in control of what versions of commons-lang are coming in everywhere by > way of the commons-config version declared in hadoop-common -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14091) AbstractFileSystem implementaion for 'wasbs' scheme
[ https://issues.apache.org/jira/browse/HADOOP-14091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu resolved HADOOP-14091. Resolution: Fixed Sure, but let's track in our private channel. This is for community effort. Thanks > AbstractFileSystem implementaion for 'wasbs' scheme > --- > > Key: HADOOP-14091 > URL: https://issues.apache.org/jira/browse/HADOOP-14091 > Project: Hadoop Common > Issue Type: Task > Components: fs/azure > Environment: humboldt >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: SECURE, WASB > Fix For: 2.8.0, 3.0.0-alpha3 > > Attachments: HADOOP-14091.001.patch, HADOOP-14091.002.patch > > > Currently org.apache.hadoop.fs.azure.Wasb provides AbstractFileSystem > implementation for 'wasb' scheme. > This task refers to providing AbstractFileSystem implementation for 'wasbs' > scheme -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14107) ITestS3GuardListConsistency fails intermittently
Mingliang Liu created HADOOP-14107:
--
Summary: ITestS3GuardListConsistency fails intermittently
Key: HADOOP-14107
URL: https://issues.apache.org/jira/browse/HADOOP-14107
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: HADOOP-13345
Reporter: Mingliang Liu
{code}
mvn -Dit.test='ITestS3GuardListConsistency' -Dtest=none -Dscale -Ds3guard
-Ddynamo -q clean verify
---
T E S T S
---
Running org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.544 sec <<<
FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency
testListStatusWriteBack(org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency)
Time elapsed: 3.147 sec <<< FAILURE!
java.lang.AssertionError: Unexpected number of results from metastore.
Metastore should only know about /XYZ:
DirListingMetadata{path=s3a://mliu-s3guard/test/ListStatusWriteBack,
listMap={s3a://mliu-s3guard/test/ListStatusWriteBack/XYZ=PathMetadata{fileStatus=S3AFileStatus{path=s3a://mliu-s3guard/test/ListStatusWriteBack/XYZ;
isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu;
permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true},
s3a://mliu-s3guard/test/ListStatusWriteBack/123=PathMetadata{fileStatus=S3AFileStatus{path=s3a://mliu-s3guard/test/ListStatusWriteBack/123;
isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu;
permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true}},
isAuthoritative=false}
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency.testListStatusWriteBack(ITestS3GuardListConsistency.java:127)
{code}
See discussion on the parent JIRA [HADOOP-13345].
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14106) Fix deprecated FileSystem inefficient calls in unit test
Mingliang Liu created HADOOP-14106:
--
Summary: Fix deprecated FileSystem inefficient calls in unit test
Key: HADOOP-14106
URL: https://issues.apache.org/jira/browse/HADOOP-14106
Project: Hadoop Common
Issue Type: Improvement
Reporter: Mingliang Liu
[HADOOP-13321] deprecates FileSystem APIs that promote inefficient call
patterns. There are existing code patterns in tests and this JIRA is to address
them.
{code}
[WARNING]
/testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractFSContractTestBase.java:[372,10]
[deprecation] isDirectory(Path) in FileSystem has been deprecated
[WARNING]
/testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java:[249,14]
[deprecation] isFile(Path) in FileSystem has been deprecated
[WARNING]
/testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java:[251,16]
[deprecation] isFile(Path) in FileSystem has been deprecated
[WARNING]
/testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java:[264,14]
[deprecation] isFile(Path) in FileSystem has been deprecated
[WARNING]
/testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java:[266,16]
[deprecation] isFile(Path) in FileSystem has been deprecated
[WARNING]
/testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java:[280,14]
[deprecation] isFile(Path) in FileSystem has been deprecated
[WARNING]
/testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java:[282,16]
[deprecation] isFile(Path) in FileSystem has been deprecated
[WARNING]
/testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java:[288,14]
[deprecation] isFile(Path) in FileSystem has been deprecated
[WARNING]
/testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java:[290,16]
[deprecation] isFile(Path) in FileSystem has been deprecated
[WARNING]
/testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java:[306,14]
[deprecation] isDirectory(Path) in FileSystem has been deprecated
[WARNING]
/testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java:[308,16]
[deprecation] isDirectory(Path) in FileSystem has been deprecated
[WARNING]
/testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java:[313,14]
[deprecation] isDirectory(Path) in FileSystem has been deprecated
[WARNING]
/testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java:[315,16]
[deprecation] isDirectory(Path) in FileSystem has been deprecated
[WARNING]
/testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java:[340,14]
[deprecation] isFile(Path) in FileSystem has been deprecated
[WARNING]
/testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java:[342,16]
[deprecation] isFile(Path) in FileSystem has been deprecated
[WARNING]
/testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java:[351,14]
[deprecation] isDirectory(Path) in FileSystem has been deprecated
[WARNING]
/testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java:[353,16]
[deprecation] isDirectory(Path) in FileSystem has been deprecated
[WARNING]
/testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java:[400,14]
[deprecation] isFile(Path) in FileSystem has been deprecated
[WARNING]
/testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java:[759,14]
[deprecation] isFile(Path) in FileSystem has been deprecated
[WARNING]
/testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java:[761,16]
[deprecation] isFile(Path) in FileSystem has been deprecated
[WARNING]
/testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContractBaseTest.java:[124,18]
[deprecation] isFile(Path) in FileSystem has been deprecated
[WARNING]
[jira] [Created] (HADOOP-14102) Relax error message assertion in S3A test ITestS3AEncryptionSSEC::testCreateFileAndReadWithDifferentEncryptionKey
Mingliang Liu created HADOOP-14102:
--
Summary: Relax error message assertion in S3A test
ITestS3AEncryptionSSEC::testCreateFileAndReadWithDifferentEncryptionKey
Key: HADOOP-14102
URL: https://issues.apache.org/jira/browse/HADOOP-14102
Project: Hadoop Common
Issue Type: Test
Components: fs/s3
Reporter: Mingliang Liu
Assignee: Mingliang Liu
Priority: Trivial
[HADOOP-13075] added support for SSE-KMS and SSE-C in s3a filesystem, along
with integration test
{{ITestS3AEncryptionSSEC::testCreateFileAndReadWithDifferentEncryptionKey}}.
For {{AccessDeniedException}} test case, it assumes the error message contains
string _Forbidden (Service: Amazon S3; Status Code: 403;_, which is true in the
current AWS java sdk and current S3AFileSystem code path.
When enabling S3Guard (see feature JIRA [HADOOP-13345]), the code path that
fails in {{S3AFileSystem}} changes for that test. Specifically, the request w/o
S3Guard was calling {{getFileStatus()}} and fails with access denied exception
containing _Forbidden_ keyword; while the request w/ S3Guard is able to call
{{getFileStatus()}} successfully and then fails later for read operations with
access denied exception containing _Access Denied_ keyword.
AWS sdk does not provide the exactly same error message for different
AccessDeniedExceptions. In the meantime, the AWS sdk may evolve (as we have
been upgrading the sdk version recently in a timely manner) and the error
message may be different in the future. This is to relax exception message
assertion in test.
Thanks [~fabbri] for discussion. See [HADOOP-13075].
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14079) Fix breaking link in s3guard.md
Mingliang Liu created HADOOP-14079: -- Summary: Fix breaking link in s3guard.md Key: HADOOP-14079 URL: https://issues.apache.org/jira/browse/HADOOP-14079 Project: Hadoop Common Issue Type: Bug Components: fs/s3 Affects Versions: HADOOP-13345 Reporter: Mingliang Liu Assignee: Mingliang Liu Priority: Trivial See the initial patch. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13960) Initialize DynamoDBMetadataStore without associated S3AFileSystem
Mingliang Liu created HADOOP-13960:
--
Summary: Initialize DynamoDBMetadataStore without associated
S3AFileSystem
Key: HADOOP-13960
URL: https://issues.apache.org/jira/browse/HADOOP-13960
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Reporter: Mingliang Liu
Assignee: Mingliang Liu
Per the discussion in [HADOOP-13650], it's helpful to initialize a
DynamoDBMetadataStore object without S3AFileSystem. In the current code, we can
achieve this via {{DynamoDBMetadataStore#initialize(Configuration)}}. However,
users still have to provide the associated S3AFileSystem URI in the
configuration, by means of either setting the {{fs.defaultFS}} in configuration
file or {{-fs s3://bucket}} command line parameter. Setting the default
FileSystem in configuration seems not necessary as the command line is to
manipulate metadata store, e.g. command line tools on an existing HDFS cluster.
This JIRA is to track the effort of initializing a DynamoDBMetadataStore
without associating any S3 buckets, so that S3AFileSystem is not needed. Users
have to specify in configuration the DynamoDB endpoints (for region) and table
name along with credentials, AWS client settings.
See [~eddyxu] and [~liuml07]'s comments in [HADOOP-13650] for more details.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-13946) Document how HDFS updates timestamps in the FS spec; compare with object stores
[ https://issues.apache.org/jira/browse/HADOOP-13946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu reopened HADOOP-13946: > Document how HDFS updates timestamps in the FS spec; compare with object > stores > --- > > Key: HADOOP-13946 > URL: https://issues.apache.org/jira/browse/HADOOP-13946 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation, fs >Affects Versions: 2.7.3 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HADOOP-13946-001.patch > > > SPARK-17159 shows that the behavior of when HDFS updates timestamps isn't > well documented. Document these in the FS spec. > I'm not going to add tests for this, as it is so very dependent on FS > implementations, as in "POSIX filesystems may behave differently from HDFS". > If someone knows what happens there, their contribution is welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13938) Address javadoc errors in Azure WASB.
[ https://issues.apache.org/jira/browse/HADOOP-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu resolved HADOOP-13938. Resolution: Duplicate > Address javadoc errors in Azure WASB. > - > > Key: HADOOP-13938 > URL: https://issues.apache.org/jira/browse/HADOOP-13938 > Project: Hadoop Common > Issue Type: Bug > Components: azure, fs/azure >Affects Versions: 2.8.0 >Reporter: Dushyanth > > As observed in HADOOP-13863, there are few javadocs error that are thrown > while building WASB in QA builds. This JIRA is created to track the fix for > these javadoc errors. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13937) Mock bucket locations in MockS3ClientFactory
Mingliang Liu created HADOOP-13937:
--
Summary: Mock bucket locations in MockS3ClientFactory
Key: HADOOP-13937
URL: https://issues.apache.org/jira/browse/HADOOP-13937
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: HADOOP-13345
Reporter: Mingliang Liu
Assignee: Mingliang Liu
Priority: Minor
Currently the MockS3ClientFactory}} does not mock the bucket locations. One
effect is that {{TestDynamoDBMetadataStore}} will have null region for the
bucket.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13908) Existing tables may not be initialized correctly in DynamoDBMetadataStore
Mingliang Liu created HADOOP-13908:
--
Summary: Existing tables may not be initialized correctly in
DynamoDBMetadataStore
Key: HADOOP-13908
URL: https://issues.apache.org/jira/browse/HADOOP-13908
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: HADOOP-13345
Reporter: Mingliang Liu
Assignee: Mingliang Liu
This was based on discussion in [HADOOP-13455]. Though we should not create
table unless the config {{fs.s3a.s3guard.ddb.table.create}} is set true, we
still have to get the existing table in {{DynamoDBMetadataStore#initialize()}}
and wait for its becoming active, before any table/item operations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13813) TestDelegationTokenFetcher#testDelegationTokenWithoutRenewer is failing
Mingliang Liu created HADOOP-13813:
--
Summary:
TestDelegationTokenFetcher#testDelegationTokenWithoutRenewer is failing
Key: HADOOP-13813
URL: https://issues.apache.org/jira/browse/HADOOP-13813
Project: Hadoop Common
Issue Type: Bug
Reporter: Mingliang Liu
Assignee: Mingliang Liu
[HADOOP-13720] added more info to the msgs printed in
{{AbstractDelegationTokenSecretManager}} for better supportability, which is
good. Unfortunately the unit test
{{TestDelegationTokenFetcher#testDelegationTokenWithoutRenewer}} that asserts
the message string was not updated accordingly. The unit test is failing in
both {{trunk}} and {{branch-2}} branches, see example builds
[1|https://issues.apache.org/jira/browse/HDFS-11129?focusedCommentId=15657488=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15657488],
[2|https://issues.apache.org/jira/browse/HDFS-11130?focusedCommentId=15658086=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15658086],
and
[3|https://issues.apache.org/jira/browse/HDFS-7?focusedCommentId=15656939=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15656939].
This JIRA is to track the effort of fixing this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13697) LogLevel#main throws exception if no arguments provided
Mingliang Liu created HADOOP-13697:
--
Summary: LogLevel#main throws exception if no arguments provided
Key: HADOOP-13697
URL: https://issues.apache.org/jira/browse/HADOOP-13697
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.9.0
Reporter: Mingliang Liu
Assignee: Mingliang Liu
{code}
root@b9ab37566005:/# hadoop daemonlog
Usage: General options are:
[-getlevel [-protocol (http|https)]
[-setlevel[-protocol (http|https)]
Exception in thread "main" org.apache.hadoop.HadoopIllegalArgumentException: No
arguments specified
at org.apache.hadoop.log.LogLevel$CLI.parseArguments(LogLevel.java:138)
at org.apache.hadoop.log.LogLevel$CLI.run(LogLevel.java:106)
at org.apache.hadoop.log.LogLevel.main(LogLevel.java:70)
{code}
I think we can catch the exception in the main method, and dump a log error
message instead of throw the stack which may frustrate users.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-11513) Artifact errors with Maven build
[
https://issues.apache.org/jira/browse/HADOOP-11513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu resolved HADOOP-11513.
Resolution: Cannot Reproduce
Feel free to re-open if it happens again. Thanks.
> Artifact errors with Maven build
>
>
> Key: HADOOP-11513
> URL: https://issues.apache.org/jira/browse/HADOOP-11513
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 2.7.0
>Reporter: Arpit Agarwal
>
> I recently started getting the following errors with _mvn -q clean compile
> install_ on Linux and OS X.
> {code}
> [ERROR] Artifact: org.xerial.snappy:snappy-java:jar:1.0.4.1 has no file.
> [ERROR] Artifact: xerces:xercesImpl:jar:2.9.1 has no file.
> [ERROR] Artifact: xml-apis:xml-apis:jar:1.3.04 has no file.
> [ERROR] Artifact: xmlenc:xmlenc:jar:0.52 has no file.
> [ERROR] Artifact: org.xerial.snappy:snappy-java:jar:1.0.4.1 has no file.
> [ERROR] Artifact: xerces:xercesImpl:jar:2.9.1 has no file.
> [ERROR] Artifact: xml-apis:xml-apis:jar:1.3.04 has no file.
> [ERROR] Artifact: xmlenc:xmlenc:jar:0.52 has no file.
> {code}
> _mvn --version_ on Linux reports:
> {code}
> Apache Maven 3.2.5 (12a6b3acb947671f09b81f49094c53f426d8cea1;
> 2014-12-14T09:29:23-08:00)
> Maven home: /home/vagrant/usr/share/maven
> Java version: 1.7.0_65, vendor: Oracle Corporation
> Java home: /usr/lib/jvm/java-7-openjdk-amd64/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "3.13.0-24-generic", arch: "amd64", family: "unix"
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13644) Replace config key literal strings with config key names
Mingliang Liu created HADOOP-13644:
--
Summary: Replace config key literal strings with config key names
Key: HADOOP-13644
URL: https://issues.apache.org/jira/browse/HADOOP-13644
Project: Hadoop Common
Issue Type: Bug
Reporter: Mingliang Liu
Assignee: Chen Liang
Priority: Minor
There are some places that use config key literal strings instead of config key
names, e.g.
{code:title=IOUtils.java}
copyBytes(in, out, conf.getInt("io.file.buffer.size", 4096), true);
{code}
We should replace places like this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-12642) Update documentation to cover fs.s3.buffer.dir enhancements
[ https://issues.apache.org/jira/browse/HADOOP-12642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu resolved HADOOP-12642. Resolution: Implemented Resolving this issue as [HADOOP-12982] covered this. Feel free to re-open if not. > Update documentation to cover fs.s3.buffer.dir enhancements > --- > > Key: HADOOP-12642 > URL: https://issues.apache.org/jira/browse/HADOOP-12642 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/s3 >Affects Versions: 2.6.0 >Reporter: Jason Archip >Priority: Minor > > Could you please update the documentation at > https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html > to include the options for the updated fs.s3.buffer.dir > Please let me know if there is a different place to put my request > Thanks, > Jason Archip -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13621) s3:// should have been fully cut off from trunk
Mingliang Liu created HADOOP-13621:
--
Summary: s3:// should have been fully cut off from trunk
Key: HADOOP-13621
URL: https://issues.apache.org/jira/browse/HADOOP-13621
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.0.0-alpha1
Reporter: Mingliang Liu
Assignee: Mingliang Liu
[HADOOP-12709] should have cut off S3 fully from the {{trunk}} branch for
Hadoop 3. However, during the long rebasing period, there are some omissions
left in documentation aka
{{hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md}}. This
jira is to address this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13512) ReloadingX509TrustManager should keep reloading in case of exception
Mingliang Liu created HADOOP-13512:
--
Summary: ReloadingX509TrustManager should keep reloading in case
of exception
Key: HADOOP-13512
URL: https://issues.apache.org/jira/browse/HADOOP-13512
Project: Hadoop Common
Issue Type: Bug
Components: security
Affects Versions: 2.8.0
Reporter: Mingliang Liu
Assignee: Mingliang Liu
{{org.apache.hadoop.security.ssl.TestReloadingX509TrustManager}} checks the key
store file's last modified time to decide whether to reload. This is to avoid
unnecessary reload if the key store file is not changed. To do this, it
maintains an internal state {{lastLoaded}} whenever it tries to reload a file.
It also updates the {{lastLoaded}} variable in case of exception so failing
reload will not be retried until the key store file's last modified time
changes again.
Chances are that the reload happens when the key store file is being written.
The reload fails (probably with EOFException) and won't load until key store
files's last modified time changes. After a short period, the key store file is
closed after update. However, the last modified time may not be updated as if
it's in the same precision period (e.g. 1 second). In this case, the updated
key store file is never reloaded.
A simple fix is to update the {{lastLoaded}} only when the reload succeeds.
{{ReloadingX509TrustManager}} will keep reloading in case of exception.
Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-13470) GenericTestUtils$LogCapturer is flaky
[
https://issues.apache.org/jira/browse/HADOOP-13470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu reopened HADOOP-13470:
> GenericTestUtils$LogCapturer is flaky
> -
>
> Key: HADOOP-13470
> URL: https://issues.apache.org/jira/browse/HADOOP-13470
> Project: Hadoop Common
> Issue Type: Bug
> Components: test, util
>Affects Versions: 2.8.0
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Labels: reviewed
> Fix For: 2.8.0
>
> Attachments: HADOOP-13470.000.patch, HADOOP-13470.001.patch
>
>
> {{GenericTestUtils$LogCapturer}} is useful for assertions against service
> logs. However it should be fixed in following aspects:
> # In the constructor, it uses the stdout appender's layout.
> {code}
> Layout layout = Logger.getRootLogger().getAppender("stdout").getLayout();
> {code}
> However, the stdout appender may be named "console" or alike which makes the
> constructor throw NPE. Actually the layout does not matter and we can use a
> default pattern layout that only captures application logs.
> # {{stopCapturing()}} method is not working. The major reason is that the
> {{appender}} internal variable is never assigned and thus removing it to stop
> capturing makes no sense.
> # It does not support {{org.slf4j.Logger}} which is preferred to log4j in
> many modules.
> # There is no unit test for it.
> This jira is to address these.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13470) GenericTestUtils$LogCapturer is flaky
Mingliang Liu created HADOOP-13470:
--
Summary: GenericTestUtils$LogCapturer is flaky
Key: HADOOP-13470
URL: https://issues.apache.org/jira/browse/HADOOP-13470
Project: Hadoop Common
Issue Type: Bug
Components: test, util
Affects Versions: 2.8.0
Reporter: Mingliang Liu
Assignee: Mingliang Liu
{{GenericTestUtils$LogCapturer}} is useful for assertions against service logs.
However it should be fixed in following aspects:
# In the constructor, it uses the stdout appender's layout.
{code}
Layout layout = Logger.getRootLogger().getAppender("stdout").getLayout();
{code}
However, the stdout appender may be named "console" or alike which makes the
constructor throw NPE. Actually the layout does not matter and we can use a
default pattern layout that only captures application logs.
# {{stopCapturing()}} method is not working. The major reason is that the
{{appender}} internal variable is never assigned and thus removing it to stop
capturing makes no sense.
# It does not support {{org.slf4j.Logger}} which is preferred to log4j in many
modules.
# There is no unit test for it.
This jira is to address these.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13435) Add thread local mechanism for aggregating file system storage stats
Mingliang Liu created HADOOP-13435:
--
Summary: Add thread local mechanism for aggregating file system
storage stats
Key: HADOOP-13435
URL: https://issues.apache.org/jira/browse/HADOOP-13435
Project: Hadoop Common
Issue Type: Sub-task
Components: fs
Reporter: Mingliang Liu
Assignee: Mingliang Liu
As discussed in [HADOOP-13032], this is to add thread local mechanism for
aggregating file system storage stats. This class will also be used in
[HADOOP-13031], which is to separate the distance-oriented rack-aware read
bytes logic from {{FileSystemStorageStatistics}} to new
DFSRackAwareStorageStatistics as it's DFS-specific. After this patch, the
{{FileSystemStorageStatistics}} can live without the to-be-removed
{{FileSystem$Statistics}} implementation.
A unit test should also be added.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13368) DFSOpsCountStatistics$OpType#fromSymbol and s3a.Statistic#fromSymbol should be O(1) operation
Mingliang Liu created HADOOP-13368:
--
Summary: DFSOpsCountStatistics$OpType#fromSymbol and
s3a.Statistic#fromSymbol should be O(1) operation
Key: HADOOP-13368
URL: https://issues.apache.org/jira/browse/HADOOP-13368
Project: Hadoop Common
Issue Type: Sub-task
Affects Versions: 2.8.0
Reporter: Mingliang Liu
Assignee: Mingliang Liu
To lookup, {{DFSOpsCountStatistics$OpType#fromSymbol}} and
{{s3a.Statistic#fromSymbol}} iterates all the enums to get the entry by its
symbol. Usages of {{fromSymbol()}} include {{isTracked()}} and {{getLong()}}.
As there are dozens of enum entries, it merits to make these two similar
operations O(1) complexity. This point is especially true if downstream app
probes a dozen of stats in an outer loop (see [TEZ-3331]).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13305) Define common statistics names across schemes
Mingliang Liu created HADOOP-13305:
--
Summary: Define common statistics names across schemes
Key: HADOOP-13305
URL: https://issues.apache.org/jira/browse/HADOOP-13305
Project: Hadoop Common
Issue Type: Sub-task
Components: fs
Affects Versions: 2.8.0
Reporter: Mingliang Liu
Assignee: Mingliang Liu
The {{StorageStatistics}} provides a pretty general interface, i.e.
{{getLong(name)}} and {{getLongStatistics()}}. There is no shared or standard
names for the storage statistics and thus the getLong(name) is up to the
implementation of storage statistics. The problems:
# For the common statistics, downstream applications expect the same statistics
name across different storage statistics and/or file system schemes. Chances
are they have to use {{DFSOpsCountStorageStatistics#getLong(“getStatus”)}} and
{{S3A.Statistics#getLong(“get_status”)}} for retrieving the getStatus operation
stat.
# Moreover, probing per-operation stats is hard if there is no standard/shared
common names.
It makes a lot of sense for different schemes to issue the per-operation stats
of the same name. Meanwhile, every FS will have its own internal things to
count, which can't be centrally defined or managed. But there are some common
which would be easier managed if they all had the same name.
Another motivation is that having a common set of names here will encourage
uniform instrumentation of all filesystems; it will also make it easier to
analyze the output of runs, were the stats to be published to a "performance
log" similar to the audit log. See Steve's work for S3 (e.g. [HADOOP-13171])
This jira is track the effort of defining common StorageStatistics entry names.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13291) DFSOpsCountStatistics/S3AStorageStatistics should implement isTracked() correctly
Mingliang Liu created HADOOP-13291:
--
Summary: DFSOpsCountStatistics/S3AStorageStatistics should
implement isTracked() correctly
Key: HADOOP-13291
URL: https://issues.apache.org/jira/browse/HADOOP-13291
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Mingliang Liu
Assignee: Mingliang Liu
Priority: Minor
{{StorageStatistics#isTracked()}} is a simple and useful API for probing a stat
with key name. Currently {{DFSOpsCountStatistics}} implements this method
wrongly. {{S3AStorageStatistics}} borrowed the same idea and also has the same
error.
This jira is to make simple things right.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13288) Guard null stats key in FileSystemStorageStatistics
Mingliang Liu created HADOOP-13288:
--
Summary: Guard null stats key in FileSystemStorageStatistics
Key: HADOOP-13288
URL: https://issues.apache.org/jira/browse/HADOOP-13288
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Mingliang Liu
Assignee: Mingliang Liu
Currently in {{FileSystemStorageStatistics}} we simply returns data from
{{FileSystem#Statistics}}. However there is no null key check, which leads to
NPE problems to downstream applications. For example, we got a NPE when passing
a null key to {{FileSystemStorageStatistics#getLong()}}, exception stack as
following:
{quote}
NullPointerException
at
org.apache.hadoop.fs.FileSystemStorageStatistics.fetch(FileSystemStorageStatistics.java:80)
at
org.apache.hadoop.fs.FileSystemStorageStatistics.getLong(FileSystemStorageStatistics.java:108)
at
org.apache.tez.runtime.metrics.FileSystemStatisticsUpdater2.updateCounters(FileSystemStatisticsUpdater2.java:60)
at
org.apache.tez.runtime.metrics.TaskCounterUpdater.updateCounters(TaskCounterUpdater.java:118)
at org.apache.tez.runtime.RuntimeTask.setFrameworkCounters(RuntimeTask.java:172)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:100)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{quote}
This jira is to add null stat key check to {{FileSystemStorageStatistics}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13284) Remove the rack-aware read stats in FileSystemStorageStatistics from branch-2
Mingliang Liu created HADOOP-13284:
--
Summary: Remove the rack-aware read stats in
FileSystemStorageStatistics from branch-2
Key: HADOOP-13284
URL: https://issues.apache.org/jira/browse/HADOOP-13284
Project: Hadoop Common
Issue Type: Sub-task
Affects Versions: 2.8.0
Reporter: Mingliang Liu
Assignee: Mingliang Liu
As [HDFS-9579] was not committed to {{branch-2.8}},
{{FileSystemStorageStatistics#KEYS}} should not include those rack aware read
stats brought by [HDFS-9579], including {{bytesReadLocalHost,
bytesReadDistanceOfOneOrTwo, bytesReadDistanceOfThreeOrFour,
bytesReadDistanceOfFiveOrLarger}}. Or else, the long iterator will throw NPE
when traversing. See detailed exception stack as following (it happens when Tez
uses the new FileSystemStorageStatistics).
{code}
2016-06-15 15:56:59,242 [DEBUG] [TezChild] |impl.TezProcessorContextImpl|:
Cleared TezProcessorContextImpl related information
2016-06-15 15:56:59,243 [WARN] [main] |task.TezTaskRunner2|: Exception from
RunnerCallable
java.lang.NullPointerException
at
org.apache.hadoop.fs.FileSystemStorageStatistics$LongStatisticIterator.next(FileSystemStorageStatistics.java:74)
at
org.apache.hadoop.fs.FileSystemStorageStatistics$LongStatisticIterator.next(FileSystemStorageStatistics.java:51)
at
org.apache.tez.runtime.metrics.FileSystemStatisticsUpdater2.updateCounters(FileSystemStatisticsUpdater2.java:51)
at
org.apache.tez.runtime.metrics.TaskCounterUpdater.updateCounters(TaskCounterUpdater.java:118)
at
org.apache.tez.runtime.RuntimeTask.setFrameworkCounters(RuntimeTask.java:172)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:100)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-06-15 15:56:59,245 [DEBUG] [main] |task.TaskReporter|: Sending heartbeat
to AM, request={ containerId=container_1466028486194_0005_01_02,
requestId=10, startIndex=0, preRoutedStartIndex=1, maxEventsToGet=500,
taskAttemptId=attempt_1466028486194_0005_1_00_00_0, eventCount=4 }
{code}
Thanks [~hitesh] for reporting this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13283) Support reset operation for new global storage statistics and per FS storage stats
Mingliang Liu created HADOOP-13283:
--
Summary: Support reset operation for new global storage statistics
and per FS storage stats
Key: HADOOP-13283
URL: https://issues.apache.org/jira/browse/HADOOP-13283
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Mingliang Liu
Assignee: Mingliang Liu
Applications may reuse the file system object across jobs and its storage
statistics should be reset. Specially the {{FileSystem.Statistics}} supports
reset and [HADOOP-13032] needs to keep that use case valid.
This jira is for supporting reset operations for storage statistics.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13032) Refactor FileSystem$Statistics as a separate source file
Mingliang Liu created HADOOP-13032:
--
Summary: Refactor FileSystem$Statistics as a separate source file
Key: HADOOP-13032
URL: https://issues.apache.org/jira/browse/HADOOP-13032
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Reporter: Mingliang Liu
Assignee: Mingliang Liu
Priority: Minor
This jira is to track the effort of moving the {{Statistics}} class out of
{{FileSystem}} for:
# shorter source code ({{FileSystem.java}} is gonna have 3780 LoC after
[HDFS-10175])
# simpler class structure (the {{Statistics}} is not very straightforward as
it's optimized for performance in [HDFS-5276] and evloving rapidly recently,
see [HDFS-9579], [HDFS-10175] and [MAPREDUCE-6660]).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-13031) Refactor the code that maintains rack-aware counters in FileSystem$Statistics
Mingliang Liu created HADOOP-13031:
--
Summary: Refactor the code that maintains rack-aware counters in
FileSystem$Statistics
Key: HADOOP-13031
URL: https://issues.apache.org/jira/browse/HADOOP-13031
Project: Hadoop Common
Issue Type: Improvement
Reporter: Mingliang Liu
Priority: Minor
According to discussion in [HDFS-10175], using a composite (e.g. enum map,
array) data structure to manage the distance->bytesRead mapping will probably
make the code simpler.
# {{StatisticsData}} will be a bit shorter by delegating the operations to the
composite data structure.
# The {{incrementBytesReadByDistance(int distance, long newBytes)}} and
{{getBytesReadByDistance(int distance)}} which switch-case all hard-code
variables, may be simplified as we can set/get the {{bytesRead}} by distance
directly from map/array.
This jira is to track the discussion and effort of refactoring the code that
maintains rack-aware counters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-12849) TestSymlinkLocalFSFileSystem fails intermittently
Mingliang Liu created HADOOP-12849: -- Summary: TestSymlinkLocalFSFileSystem fails intermittently Key: HADOOP-12849 URL: https://issues.apache.org/jira/browse/HADOOP-12849 Project: Hadoop Common Issue Type: Bug Components: test Reporter: Mingliang Liu *Error Message* expected:<1456523612000> but was:<1456523613000> *Stacktrace* java.lang.AssertionError: expected:<1456523612000> but was:<1456523613000> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.junit.Assert.assertEquals(Assert.java:542) at org.apache.hadoop.fs.SymlinkBaseTest.testSetTimesDanglingLink(SymlinkBaseTest.java:1410) at org.apache.hadoop.fs.TestSymlinkLocalFS.testSetTimesDanglingLink(TestSymlinkLocalFS.java:239) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) It happens in recent builds: * https://builds.apache.org/job/PreCommit-HADOOP-Build/8732/testReport/org.apache.hadoop.fs/TestSymlinkLocalFSFileSystem/testSetTimesDanglingLink/ * https://builds.apache.org/job/PreCommit-HADOOP-Build/8721/testReport/org.apache.hadoop.fs/TestSymlinkLocalFSFileSystem/testSetTimesSymlinkToFile/ * https://builds.apache.org/job/PreCommit-HADOOP-Build/8590/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12514) Make
Mingliang Liu created HADOOP-12514: -- Summary: Make Key: HADOOP-12514 URL: https://issues.apache.org/jira/browse/HADOOP-12514 Project: Hadoop Common Issue Type: Task Reporter: Mingliang Liu Assignee: Mingliang Liu Priority: Minor This is a follow up of [HADOOP-12472]. It makes sense to make the following static fields package private instead of protected, as they are for test purpose (protected keyword makes more sense to sub-classes). - protected static String E_NULL_THROWABLE = "Null Throwable"; - protected static String E_NULL_THROWABLE_STRING = "Null Throwable.toString() value"; - protected static String E_UNEXPECTED_EXCEPTION = "but got unexpected exception"; Meanwhile, we may need to make them final. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12474) Fix data race when allocating server port in MiniKMS
Mingliang Liu created HADOOP-12474:
--
Summary: Fix data race when allocating server port in MiniKMS
Key: HADOOP-12474
URL: https://issues.apache.org/jira/browse/HADOOP-12474
Project: Hadoop Common
Issue Type: Bug
Reporter: Mingliang Liu
Assignee: Mingliang Liu
In {{MiniKMS}}, there is potential data race when create jetty server in
{{createJettyServer}}. It looks like the code searches for a free port and then
starts jetty, but maybe there's enough of a race condition between port
location and jetty.start to cause intermittent failures.
{code}
ServerSocket ss = new ServerSocket((inPort < 0) ? 0 : inPort, 50,
localhost);
int port = ss.getLocalPort();
Server server = new Server(0);
if (!ssl) {
server.getConnectors()[0].setHost(host);
server.getConnectors()[0].setPort(port);
} else {
...
c.setPort(port);
{code}
We've seen test failures saying {{java.net.BindException: Address already in
use}}, e.g.
[https://builds.apache.org/job/PreCommit-HDFS-Build/12942/testReport/]
As in [HADOOP-12417], we should always bind port 0 which gives us an ephemeral
port, instead of searching a free port before starting jetty.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-12473) distcp's ignoring failures should be mutually exclusive with the atomic option
Mingliang Liu created HADOOP-12473:
--
Summary: distcp's ignoring failures should be mutually exclusive
with the atomic option
Key: HADOOP-12473
URL: https://issues.apache.org/jira/browse/HADOOP-12473
Project: Hadoop Common
Issue Type: Bug
Components: tools/distcp
Affects Versions: 2.7.1
Reporter: Mingliang Liu
Assignee: Mingliang Liu
Priority: Critical
Fix For: 2.8.0
{{RetriableFileCopyCommand.CopyReadException}} is double-wrapped via
# via {{RetriableCommand::execute}}
# via {{CopyMapper#copyFileWithRetry}}
before {{CopyMapper::handleFailure}} tests
{code}
if (ignoreFailures && exception.getCause() instanceof
RetriableFileCopyCommand.CopyReadException
{code}
which is always false.
Orthogonally, ignoring failures should be mutually exclusive with the atomic
option otherwise an incomplete dir is eligible for commit defeating the purpose.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
