[jira] [Commented] (HADOOP-15428) s3guard bucket-info -unguarded will guard bucket if FS is set to do this automatically

2018-04-30 Thread lqjack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HADOOP-15428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458671#comment-16458671
 ] 

lqjack commented on HADOOP-15428:
-

https://github.com/apache/hadoop/pull/373

> s3guard bucket-info -unguarded will guard bucket if FS is set to do this 
> automatically
> --
>
> Key: HADOOP-15428
> URL: https://issues.apache.org/jira/browse/HADOOP-15428
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Priority: Major
>
> If you call hadoop s3guard bucket-info on a bucket where the fs is set to 
> create a s3guard table on demand, then the DDB table is automatically 
> created. As a result
> the {{bucket-info -unguarded}} option cannot be used, and the call has 
> significant side effects (i.e. it can run up bills)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15428) s3guard bucket-info -unguarded will guard bucket if FS is set to do this automatically

2018-04-28 Thread lqjack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HADOOP-15428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457884#comment-16457884
 ] 

lqjack commented on HADOOP-15428:
-

[~ste...@apache.org] I do’t think there exist the issue. 
If you user disable the function to create the new Table when the table not 
exist . so we just throw the exception.
In my opinion , we can just change the error message for user . 
What do you think of it ? 

> s3guard bucket-info -unguarded will guard bucket if FS is set to do this 
> automatically
> --
>
> Key: HADOOP-15428
> URL: https://issues.apache.org/jira/browse/HADOOP-15428
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Priority: Major
>
> If you call hadoop s3guard bucket-info on a bucket where the fs is set to 
> create a s3guard table on demand, then the DDB table is automatically 
> created. As a result
> the {{bucket-info -unguarded}} option cannot be used, and the call has 
> significant side effects (i.e. it can run up bills)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15427) hadoop shell complains needlessly about "ERROR: Tools helper .. not found"

2018-04-28 Thread lqjack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HADOOP-15427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457690#comment-16457690
 ] 

lqjack commented on HADOOP-15427:
-

https://github.com/apache/hadoop/pull/370

> hadoop shell complains needlessly about "ERROR: Tools helper .. not found"
> --
>
> Key: HADOOP-15427
> URL: https://issues.apache.org/jira/browse/HADOOP-15427
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: scripts
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Priority: Minor
>
> toolshelper.sh prints error messages like
> {code}
> ERROR: Tools helper...hadoop/libexec/tools/hadoop-aws.sh was not found.
> {code}
> even when they aren't neede,d here in the case of hadoop s3guard shell 
> commands.
> Can I downgrade these to hadoop_debug?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15423) Use single hash Path -> tuple(DirListingMetadata, PathMetadata) in LocalMetadataStore

2018-04-27 Thread lqjack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HADOOP-15423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456846#comment-16456846
 ] 

lqjack commented on HADOOP-15423:
-

[~gabor.bota] What about provide only one map 

private LruHashMap metaMap;

interface PathMetadataable {

  boolean pathMetadata();


  PathMetadata   pathInfo();


  DirListingMetadata dirInfo();
}

> Use single hash Path -> tuple(DirListingMetadata, PathMetadata) in 
> LocalMetadataStore
> -
>
> Key: HADOOP-15423
> URL: https://issues.apache.org/jira/browse/HADOOP-15423
> Project: Hadoop Common
>  Issue Type: Sub-task
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Minor
>
> Right now the s3guard.LocalMetadataStore uses two HashMap in the 
> implementation - one for the file and one for the dir hash.
> {code:java}
>   /** Contains directories and files. */
>   private LruHashMap fileHash;
>   /** Contains directory listings. */
>   private LruHashMap dirHash;
> {code}
> It would be nice to have only one hash instead of these two for storing the 
> values. An idea for the implementation would be to have a class with nullable 
> fields:
> {code:java}
>   static class LocalMetaEntry {
> @Nullable
> public PathMetadata pathMetadata;
> @Nullable
> public DirListingMetadata dirListingMetadata;
>   }
> {code}
> or a Pair (tuple):
> {code:java}
> Pair metaEntry;
> {code}
> And only one hash/cache for these elements.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15418) Hadoop KMSAuthenticationFilter needs to use getPropsByPrefix instead of iterator to avoid ConcurrentModificationException

2018-04-27 Thread lqjack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HADOOP-15418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456840#comment-16456840
 ] 

lqjack commented on HADOOP-15418:
-

https://github.com/apache/hadoop/pull/369

> Hadoop KMSAuthenticationFilter needs to use getPropsByPrefix instead of 
> iterator to avoid ConcurrentModificationException
> -
>
> Key: HADOOP-15418
> URL: https://issues.apache.org/jira/browse/HADOOP-15418
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: common
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
>Priority: Major
>
> The issue is similar to what was fixed in HADOOP-15411. Fixing this in 
> KMSAuthenticationFilter as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15380) TestViewFileSystemLocalFileSystem#testTrashRoot leaves an unnecessary file

2018-04-27 Thread lqjack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HADOOP-15380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456812#comment-16456812
 ] 

lqjack commented on HADOOP-15380:
-

https://github.com/apache/hadoop/pull/368

> TestViewFileSystemLocalFileSystem#testTrashRoot leaves an unnecessary file
> --
>
> Key: HADOOP-15380
> URL: https://issues.apache.org/jira/browse/HADOOP-15380
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Andras Bokor
>Assignee: Andras Bokor
>Priority: Major
>
> After running
> {code}mvn test -Dtest=TestViewFileSystemLocalFileSystem#testTrashRoot
> git status{code}
> Git reports an untracked file: 
> {{hadoop-common-project/hadoop-common/.debug.log.crc}}
> It seems some cleanup issue.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15409) S3AFileSystem.verifyBucketExists to move to s3.doesBucketExistV2

2018-04-25 Thread lqjack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HADOOP-15409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452067#comment-16452067
 ] 

lqjack commented on HADOOP-15409:
-

https://github.com/apache/hadoop/pull/367

> S3AFileSystem.verifyBucketExists to move to s3.doesBucketExistV2
> 
>
> Key: HADOOP-15409
> URL: https://issues.apache.org/jira/browse/HADOOP-15409
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Priority: Major
>
> in S3AFileSystem.initialize(), we check for the bucket existing with 
> verifyBucketExists(), which calls s3.doesBucketExist(). But that doesn't 
> check for auth issues. 
> s3. doesBucketExistV2() does at least validate credentials, and should be 
> switched to. This will help things fail faster 
> See SPARK-24000



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15410) hoop-auth org.apache.hadoop.security.authentication.util.TestZKSignerSecretProvider org.apache.log4j package compile error

2018-04-24 Thread lqjack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HADOOP-15410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451603#comment-16451603
 ] 

lqjack commented on HADOOP-15410:
-

Thanks your advices. I have changed to provided. 

> hoop-auth 
> org.apache.hadoop.security.authentication.util.TestZKSignerSecretProvider 
> org.apache.log4j package compile error
> --
>
> Key: HADOOP-15410
> URL: https://issues.apache.org/jira/browse/HADOOP-15410
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: lqjack
>Priority: Major
>
> When run the 
> org.apache.hadoop.security.authentication.util.TestZKSignerSecretProvider , 
> IDE will automatic compile the java class , but unlucky org.apache.log4j 
> compile failed. 
> should change the pom.xml 
> 
>   log4j
>   log4j
>   runtime
> 
> to 
> 
>   log4j
>   log4j
>   compile
> 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15410) hoop-auth org.apache.hadoop.security.authentication.util.TestZKSignerSecretProvider org.apache.log4j package compile error

2018-04-24 Thread lqjack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HADOOP-15410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450138#comment-16450138
 ] 

lqjack commented on HADOOP-15410:
-

https://github.com/apache/hadoop/pull/366

> hoop-auth 
> org.apache.hadoop.security.authentication.util.TestZKSignerSecretProvider 
> org.apache.log4j package compile error
> --
>
> Key: HADOOP-15410
> URL: https://issues.apache.org/jira/browse/HADOOP-15410
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: lqjack
>Priority: Major
>
> When run the 
> org.apache.hadoop.security.authentication.util.TestZKSignerSecretProvider , 
> IDE will automatic compile the java class , but unlucky org.apache.log4j 
> compile failed. 
> should change the pom.xml 
> 
>   log4j
>   log4j
>   runtime
> 
> to 
> 
>   log4j
>   log4j
>   compile
> 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15410) hoop-auth org.apache.hadoop.security.authentication.util.TestZKSignerSecretProvider org.apache.log4j package compile error

2018-04-24 Thread lqjack (JIRA) 
lqjack created HADOOP-15410:
---

 Summary: hoop-auth 
org.apache.hadoop.security.authentication.util.TestZKSignerSecretProvider 
org.apache.log4j package compile error
 Key: HADOOP-15410
 URL: https://issues.apache.org/jira/browse/HADOOP-15410
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.9.0
Reporter: lqjack


When run the 
org.apache.hadoop.security.authentication.util.TestZKSignerSecretProvider , IDE 
will automatic compile the java class , but unlucky org.apache.log4j compile 
failed. 

should change the pom.xml 

  log4j
  log4j
  runtime



to 


  log4j
  log4j
  compile





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org