[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and provide negotiation capabilities
[ https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13693307#comment-13693307 ] Ashwin Shankar commented on HADOOP-9421: Hi, I getting the same exception mentioned my Steve when I run a sleep job against the latest code in trunk : Caused by: java.io.IOException: java.io.IOException: Server asks us to fall back to SIMPLE auth, but this client is configured to only allow secure connections. Could you please let me know how I can rectify this ? I have an insecure pseudo distributed cluster. Convert SASL to use ProtoBuf and provide negotiation capabilities - Key: HADOOP-9421 URL: https://issues.apache.org/jira/browse/HADOOP-9421 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 2.0.3-alpha Reporter: Sanjay Radia Assignee: Daryn Sharp Priority: Blocker Fix For: 3.0.0, 2.1.0-beta, 2.2.0 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421-v2-demo.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-9582) Non-existent file to hadoop fs -conf doesn't throw error
[ https://issues.apache.org/jira/browse/HADOOP-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar reassigned HADOOP-9582: -- Assignee: Ashwin Shankar Non-existent file to hadoop fs -conf doesn't throw error -- Key: HADOOP-9582 URL: https://issues.apache.org/jira/browse/HADOOP-9582 Project: Hadoop Common Issue Type: Bug Components: conf Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha Reporter: Ashwin Shankar Assignee: Ashwin Shankar Attachments: HADOOP-9582.txt, HADOOP-9582.txt, HADOOP-9582.txt When we run : hadoop fs -conf BAD_FILE -ls / we expect hadoop to throw an error,but it doesn't. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9582) Non-existent file to hadoop fs -conf doesn't throw error
[ https://issues.apache.org/jira/browse/HADOOP-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated HADOOP-9582: --- Attachment: HADOOP-9582-4-b23.txt Non-existent file to hadoop fs -conf doesn't throw error -- Key: HADOOP-9582 URL: https://issues.apache.org/jira/browse/HADOOP-9582 Project: Hadoop Common Issue Type: Bug Components: conf Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha Reporter: Ashwin Shankar Assignee: Ashwin Shankar Attachments: HADOOP-9582-4-b23.txt, HADOOP-9582.txt, HADOOP-9582.txt, HADOOP-9582.txt When we run : hadoop fs -conf BAD_FILE -ls / we expect hadoop to throw an error,but it doesn't. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9582) Non-existent file to hadoop fs -conf doesn't throw error
[ https://issues.apache.org/jira/browse/HADOOP-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated HADOOP-9582: --- Attachment: HADOOP-9582-4.txt Non-existent file to hadoop fs -conf doesn't throw error -- Key: HADOOP-9582 URL: https://issues.apache.org/jira/browse/HADOOP-9582 Project: Hadoop Common Issue Type: Bug Components: conf Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha Reporter: Ashwin Shankar Assignee: Ashwin Shankar Attachments: HADOOP-9582-4-b23.txt, HADOOP-9582-4.txt, HADOOP-9582.txt, HADOOP-9582.txt, HADOOP-9582.txt When we run : hadoop fs -conf BAD_FILE -ls / we expect hadoop to throw an error,but it doesn't. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9582) Non-existent file to hadoop fs -conf doesn't throw error
[
https://issues.apache.org/jira/browse/HADOOP-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13683829#comment-13683829
]
Ashwin Shankar commented on HADOOP-9582:
{quote}
TestGenericOptionsParser change now seems moot, since we didn't modify
GenericOptionsParser and it doesn't test the FsShell change.
{quote}
Yes,right. I've reverted that change in the latest patch.
{quote}
Nit: In TestFsShell rather than wielding AssertionFailedError directly we can
simply use assertTrue(th instanceof RuntimeException)
{quote}
I did do this in one my previous patch,however Steve(on 22/May/13 06:01)
recommended that raising AssertionFailedError would be better for diagnostics.
Should I still use assertTrue ?
Non-existent file to hadoop fs -conf doesn't throw error
--
Key: HADOOP-9582
URL: https://issues.apache.org/jira/browse/HADOOP-9582
Project: Hadoop Common
Issue Type: Bug
Components: conf
Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha
Reporter: Ashwin Shankar
Assignee: Ashwin Shankar
Attachments: HADOOP-9582-4-b23.txt, HADOOP-9582-4.txt,
HADOOP-9582.txt, HADOOP-9582.txt, HADOOP-9582.txt
When we run :
hadoop fs -conf BAD_FILE -ls /
we expect hadoop to throw an error,but it doesn't.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9582) Non-existent file to hadoop fs -conf doesn't throw error
[ https://issues.apache.org/jira/browse/HADOOP-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13675372#comment-13675372 ] Ashwin Shankar commented on HADOOP-9582: Hi Jason, Thanks for your comments. Yes this patch doesn't address the problem in other places apart from FsShell. But could we check this in so that we are good atleast from the FsShell standpoint. Maybe we could create another JIRA to investigate whether we break any use-case if we make the change directly in GenericOptionsParser. Non-existent file to hadoop fs -conf doesn't throw error -- Key: HADOOP-9582 URL: https://issues.apache.org/jira/browse/HADOOP-9582 Project: Hadoop Common Issue Type: Bug Components: conf Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha Reporter: Ashwin Shankar Attachments: HADOOP-9582.txt, HADOOP-9582.txt, HADOOP-9582.txt When we run : hadoop fs -conf BAD_FILE -ls / we expect hadoop to throw an error,but it doesn't. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9582) Non-existent file to hadoop fs -conf doesn't throw error
[ https://issues.apache.org/jira/browse/HADOOP-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated HADOOP-9582: --- Attachment: HADOOP-9582.txt I've turned on the 'quiet' flag in FsShell. This would cause a Runtime exception to be raised when the user gives a nonexistent resource. I've also changed INFO logs(which depends on the flag) to debug so that output is not verbose. Non-existent file to hadoop fs -conf doesn't throw error -- Key: HADOOP-9582 URL: https://issues.apache.org/jira/browse/HADOOP-9582 Project: Hadoop Common Issue Type: Bug Components: conf Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha Reporter: Ashwin Shankar Attachments: HADOOP-9582.txt, HADOOP-9582.txt, HADOOP-9582.txt When we run : hadoop fs -conf BAD_FILE -ls / we expect hadoop to throw an error,but it doesn't. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9581) Hadoop --config non-existent directory should result in error
[
https://issues.apache.org/jira/browse/HADOOP-9581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664185#comment-13664185
]
Ashwin Shankar commented on HADOOP-9581:
This fix was in shell scripts which don't have unit tests.
I've manually verified it.
Hadoop --config non-existent directory should result in error
--
Key: HADOOP-9581
URL: https://issues.apache.org/jira/browse/HADOOP-9581
Project: Hadoop Common
Issue Type: Bug
Components: scripts
Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha
Reporter: Ashwin Shankar
Attachments: HADOOP-9581.txt
Courtesy : [~cwchung]
{quote}Providing a non-existent config directory should result in error.
$ hadoop dfs -ls / : shows Hadoop DFS directory
$ hadoop --config bad_config_dir dfs -ls : successful, showing Linux directory
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9582) Non-existent file to hadoop fs -conf doesn't throw error
[ https://issues.apache.org/jira/browse/HADOOP-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated HADOOP-9582: --- Attachment: HADOOP-9582.txt Added AssertionFailedError in unit tests and corrected spacing issues. Non-existent file to hadoop fs -conf doesn't throw error -- Key: HADOOP-9582 URL: https://issues.apache.org/jira/browse/HADOOP-9582 Project: Hadoop Common Issue Type: Bug Components: conf Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha Reporter: Ashwin Shankar Attachments: HADOOP-9582.txt, HADOOP-9582.txt When we run : hadoop fs -conf BAD_FILE -ls / we expect hadoop to throw an error,but it doesn't. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9582) Non-existent file to hadoop fs -conf doesn't throw error
[
https://issues.apache.org/jira/browse/HADOOP-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664558#comment-13664558
]
Ashwin Shankar commented on HADOOP-9582:
{quote}I've been looking at this a bit more, and now I'm worried about some
compatibility issue. It looks like -conf can be used to specify any resource
path. Is this right, or am I misreading it?{quote}
I'm not sure whether URIs are allowed for '-conf' option.I checked up the
documentation and javadoc, it doesn't talk about URIs. But I would tend to
believe that URIs are allowed ,since its allowed for other options like hadoop
fs -fs URI.If this is the case,then you're right,we have a problem.
{quote}At the same time, delving into Configuration.loadResource() looks like
it gets the URI of the Path instance, calls getPath() on it and then converts
it to a file for loading (skipping missing entries).{quote}
Yes,that's right. Here is where it gets interesting and I have a question for
you.
Configuration.loadResource() has the following code :
{code}
if (doc == null root == null) {
if (quiet)
return null;
throw new RuntimeException(resource + not found);
}
{code}
Looking at the code,it seems that if 'file' doesn't exist we do throw
RuntimeException if the 'quiet' flag is false,which means removing the 'quiet'
flag seems to solve our problem.My question is why is there a 'quiet' flag in
the first place ? I tried to look at svn file history and this particular code
snippet goes back to early days(2006),hence no documentation about it.
Do you know about this flag ?
Is there some ancient use-case where its legal to give a non-existent config
file as arguments while loading it 'quietly' ? because this patch will break
that.
Non-existent file to hadoop fs -conf doesn't throw error
--
Key: HADOOP-9582
URL: https://issues.apache.org/jira/browse/HADOOP-9582
Project: Hadoop Common
Issue Type: Bug
Components: conf
Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha
Reporter: Ashwin Shankar
Attachments: HADOOP-9582.txt, HADOOP-9582.txt
When we run :
hadoop fs -conf BAD_FILE -ls /
we expect hadoop to throw an error,but it doesn't.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9581) Hadoop --config non-existent directory should result in error
[
https://issues.apache.org/jira/browse/HADOOP-9581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13663476#comment-13663476
]
Ashwin Shankar commented on HADOOP-9581:
I've added code to validate if the argument to --config option is a directory.
If not,I throw an error.
Hadoop --config non-existent directory should result in error
--
Key: HADOOP-9581
URL: https://issues.apache.org/jira/browse/HADOOP-9581
Project: Hadoop Common
Issue Type: Bug
Components: scripts
Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha
Reporter: Ashwin Shankar
Courtesy : [~cwchung]
{quote}Providing a non-existent config directory should result in error.
$ hadoop dfs -ls / : shows Hadoop DFS directory
$ hadoop --config bad_config_dir dfs -ls : successful, showing Linux directory
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9581) Hadoop --config non-existent directory should result in error
[
https://issues.apache.org/jira/browse/HADOOP-9581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashwin Shankar updated HADOOP-9581:
---
Status: Open (was: Patch Available)
Hadoop --config non-existent directory should result in error
--
Key: HADOOP-9581
URL: https://issues.apache.org/jira/browse/HADOOP-9581
Project: Hadoop Common
Issue Type: Bug
Components: scripts
Affects Versions: 2.0.4-alpha, 0.23.7, 3.0.0
Reporter: Ashwin Shankar
Courtesy : [~cwchung]
{quote}Providing a non-existent config directory should result in error.
$ hadoop dfs -ls / : shows Hadoop DFS directory
$ hadoop --config bad_config_dir dfs -ls : successful, showing Linux directory
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9581) Hadoop --config non-existent directory should result in error
[
https://issues.apache.org/jira/browse/HADOOP-9581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashwin Shankar updated HADOOP-9581:
---
Status: Patch Available (was: Open)
Hadoop --config non-existent directory should result in error
--
Key: HADOOP-9581
URL: https://issues.apache.org/jira/browse/HADOOP-9581
Project: Hadoop Common
Issue Type: Bug
Components: scripts
Affects Versions: 2.0.4-alpha, 0.23.7, 3.0.0
Reporter: Ashwin Shankar
Courtesy : [~cwchung]
{quote}Providing a non-existent config directory should result in error.
$ hadoop dfs -ls / : shows Hadoop DFS directory
$ hadoop --config bad_config_dir dfs -ls : successful, showing Linux directory
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9581) Hadoop --config non-existent directory should result in error
[
https://issues.apache.org/jira/browse/HADOOP-9581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashwin Shankar updated HADOOP-9581:
---
Attachment: HADOOP-9581.txt
Hadoop --config non-existent directory should result in error
--
Key: HADOOP-9581
URL: https://issues.apache.org/jira/browse/HADOOP-9581
Project: Hadoop Common
Issue Type: Bug
Components: scripts
Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha
Reporter: Ashwin Shankar
Attachments: HADOOP-9581.txt
Courtesy : [~cwchung]
{quote}Providing a non-existent config directory should result in error.
$ hadoop dfs -ls / : shows Hadoop DFS directory
$ hadoop --config bad_config_dir dfs -ls : successful, showing Linux directory
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9581) Hadoop --config non-existent directory should result in error
[
https://issues.apache.org/jira/browse/HADOOP-9581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashwin Shankar updated HADOOP-9581:
---
Status: Patch Available (was: Open)
I've added code to validate if the argument to --config option is a directory.
If not,I throw an error.
Hadoop --config non-existent directory should result in error
--
Key: HADOOP-9581
URL: https://issues.apache.org/jira/browse/HADOOP-9581
Project: Hadoop Common
Issue Type: Bug
Components: scripts
Affects Versions: 2.0.4-alpha, 0.23.7, 3.0.0
Reporter: Ashwin Shankar
Attachments: HADOOP-9581.txt
Courtesy : [~cwchung]
{quote}Providing a non-existent config directory should result in error.
$ hadoop dfs -ls / : shows Hadoop DFS directory
$ hadoop --config bad_config_dir dfs -ls : successful, showing Linux directory
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9582) Non-existent file to hadoop fs -conf doesn't throw error
[ https://issues.apache.org/jira/browse/HADOOP-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated HADOOP-9582: --- Attachment: HADOOP-9582.txt Non-existent file to hadoop fs -conf doesn't throw error -- Key: HADOOP-9582 URL: https://issues.apache.org/jira/browse/HADOOP-9582 Project: Hadoop Common Issue Type: Bug Components: conf Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha Reporter: Ashwin Shankar Attachments: HADOOP-9582.txt When we run : hadoop fs -conf BAD_FILE -ls / we expect hadoop to throw an error,but it doesn't. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9582) Non-existent file to hadoop fs -conf doesn't throw error
[ https://issues.apache.org/jira/browse/HADOOP-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated HADOOP-9582: --- Status: Patch Available (was: Open) I've added code to check if the argument to -conf option is a file(which exist) or not. I throw an exception if file doesn't exist. Non-existent file to hadoop fs -conf doesn't throw error -- Key: HADOOP-9582 URL: https://issues.apache.org/jira/browse/HADOOP-9582 Project: Hadoop Common Issue Type: Bug Components: conf Affects Versions: 2.0.4-alpha, 0.23.7, 3.0.0 Reporter: Ashwin Shankar Attachments: HADOOP-9582.txt When we run : hadoop fs -conf BAD_FILE -ls / we expect hadoop to throw an error,but it doesn't. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9581) Hadoop --config non-existent directory should result in error
Ashwin Shankar created HADOOP-9581:
--
Summary: Hadoop --config non-existent directory should result in
error
Key: HADOOP-9581
URL: https://issues.apache.org/jira/browse/HADOOP-9581
Project: Hadoop Common
Issue Type: Bug
Components: scripts
Affects Versions: 2.0.4-alpha, 0.23.7, 3.0.0
Reporter: Ashwin Shankar
Courtesy : [~cwchung]
{quote}Providing a non-existent config directory should result in error.
$ hadoop dfs -ls / : shows Hadoop DFS directory
$ hadoop --config bad_config_dir dfs -ls : successful, showing Linux directory
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9582) Non-existent file to hadoop fs -conf doesn't throw error
Ashwin Shankar created HADOOP-9582: -- Summary: Non-existent file to hadoop fs -conf doesn't throw error Key: HADOOP-9582 URL: https://issues.apache.org/jira/browse/HADOOP-9582 Project: Hadoop Common Issue Type: Bug Components: conf Affects Versions: 2.0.4-alpha, 0.23.7, 3.0.0 Reporter: Ashwin Shankar When we run : hadoop fs -conf BAD_FILE -ls / we expect hadoop to throw an error,but it doesn't. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
