[jira] [Commented] (HADOOP-14450) ADLS Python client inconsistent when used in tandem with AdlFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-14450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16048510#comment-16048510 ] Atul Sikaria commented on HADOOP-14450: --- The pythin tool with fix is released now. Run this to upgrade the command-line python tool: run pip install azure-datalake-store --upgrade > ADLS Python client inconsistent when used in tandem with AdlFileSystem > -- > > Key: HADOOP-14450 > URL: https://issues.apache.org/jira/browse/HADOOP-14450 > Project: Hadoop Common > Issue Type: Bug > Components: fs/adl >Reporter: Sailesh Mukil >Assignee: Atul Sikaria > Labels: infrastructure > > Impala uses the AdlFileSystem connector to talk to ADLS. As a part of the > Impala tests, we drop tables and verify that the files belonging to that > table have been dropped for all filesystems that Impala supports. These tests > however, fail with ADLS. > If I use the Hadoop ADLS connector to delete a file, and then list the parent > directory of that file using the above Python client within the second, the > client still says that the file is available in ADLS. > This is the Python client from Microsoft that we're using in our testing: > https://github.com/Azure/azure-data-lake-store-python > Their release notes say that it's still a "pre-release preview": > https://github.com/Azure/azure-data-lake-store-python/releases > Questions for the ADLS folks: > Is this a known issue? If so, will it be fixed soon? > Or is this expected behavior? > I'm able to deterministically reproduce it in my tests, with Impala on ADLS. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14450) ADLS Python client inconsistent when used in tandem with AdlFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-14450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023309#comment-16023309 ] Atul Sikaria commented on HADOOP-14450: --- Following up with the developers of the Python SDK. My understanding is that this is a bug in the SDK, but following up for more. > ADLS Python client inconsistent when used in tandem with AdlFileSystem > -- > > Key: HADOOP-14450 > URL: https://issues.apache.org/jira/browse/HADOOP-14450 > Project: Hadoop Common > Issue Type: Bug > Components: fs/adl >Reporter: Sailesh Mukil >Assignee: Atul Sikaria > Labels: infrastructure > > Impala uses the AdlFileSystem connector to talk to ADLS. As a part of the > Impala tests, we drop tables and verify that the files belonging to that > table have been dropped for all filesystems that Impala supports. These tests > however, fail with ADLS. > If I use the Hadoop ADLS connector to delete a file, and then list the parent > directory of that file using the above Python client within the second, the > client still says that the file is available in ADLS. > This is the Python client from Microsoft that we're using in our testing: > https://github.com/Azure/azure-data-lake-store-python > Their release notes say that it's still a "pre-release preview": > https://github.com/Azure/azure-data-lake-store-python/releases > Questions for the ADLS folks: > Is this a known issue? If so, will it be fixed soon? > Or is this expected behavior? > I'm able to deterministically reproduce it in my tests, with Impala on ADLS. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14450) ADLS Python client inconsistent when used in tandem with AdlFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-14450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021792#comment-16021792 ] Mingliang Liu commented on HADOOP-14450: I'd suggest a simple unit test to reproduce the bug (if any), or commands in FSShell. Ad hoc pythons clients wrapping ADLFileSystem may have it's own bug. > ADLS Python client inconsistent when used in tandem with AdlFileSystem > -- > > Key: HADOOP-14450 > URL: https://issues.apache.org/jira/browse/HADOOP-14450 > Project: Hadoop Common > Issue Type: Bug > Components: fs/adl >Reporter: Sailesh Mukil >Assignee: Atul Sikaria > Labels: infrastructure > > Impala uses the AdlFileSystem connector to talk to ADLS. As a part of the > Impala tests, we drop tables and verify that the files belonging to that > table have been dropped for all filesystems that Impala supports. These tests > however, fail with ADLS. > If I use the Hadoop ADLS connector to delete a file, and then list the parent > directory of that file using the above Python client within the second, the > client still says that the file is available in ADLS. > This is the Python client from Microsoft that we're using in our testing: > https://github.com/Azure/azure-data-lake-store-python > Their release notes say that it's still a "pre-release preview": > https://github.com/Azure/azure-data-lake-store-python/releases > Questions for the ADLS folks: > Is this a known issue? If so, will it be fixed soon? > Or is this expected behavior? > I'm able to deterministically reproduce it in my tests, with Impala on ADLS. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14450) ADLS Python client inconsistent when used in tandem with AdlFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-14450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021753#comment-16021753 ] Steve Loughran commented on HADOOP-14450: - sounds like an impala bug, not Hadoop > ADLS Python client inconsistent when used in tandem with AdlFileSystem > -- > > Key: HADOOP-14450 > URL: https://issues.apache.org/jira/browse/HADOOP-14450 > Project: Hadoop Common > Issue Type: Bug > Components: fs/adl >Reporter: Sailesh Mukil >Assignee: Atul Sikaria > Labels: infrastructure > > Impala uses the AdlFileSystem connector to talk to ADLS. As a part of the > Impala tests, we drop tables and verify that the files belonging to that > table have been dropped for all filesystems that Impala supports. These tests > however, fail with ADLS. > If I use the Hadoop ADLS connector to delete a file, and then list the parent > directory of that file using the above Python client within the second, the > client still says that the file is available in ADLS. > This is the Python client from Microsoft that we're using in our testing: > https://github.com/Azure/azure-data-lake-store-python > Their release notes say that it's still a "pre-release preview": > https://github.com/Azure/azure-data-lake-store-python/releases > Questions for the ADLS folks: > Is this a known issue? If so, will it be fixed soon? > Or is this expected behavior? > I'm able to deterministically reproduce it in my tests, with Impala on ADLS. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14450) ADLS Python client inconsistent when used in tandem with AdlFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-14450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021720#comment-16021720 ] John Zhuge commented on HADOOP-14450: - [~ASikaria], could you please take a look? > ADLS Python client inconsistent when used in tandem with AdlFileSystem > -- > > Key: HADOOP-14450 > URL: https://issues.apache.org/jira/browse/HADOOP-14450 > Project: Hadoop Common > Issue Type: Bug > Components: fs/adl >Reporter: Sailesh Mukil >Assignee: Atul Sikaria > Labels: infrastructure > > Impala uses the AdlFileSystem connector to talk to ADLS. As a part of the > Impala tests, we drop tables and verify that the files belonging to that > table have been dropped for all filesystems that Impala supports. These tests > however, fail with ADLS. > If I use the Hadoop ADLS connector to delete a file, and then list the parent > directory of that file using the above Python client within the second, the > client still says that the file is available in ADLS. > This is the Python client from Microsoft that we're using in our testing: > https://github.com/Azure/azure-data-lake-store-python > Their release notes say that it's still a "pre-release preview": > https://github.com/Azure/azure-data-lake-store-python/releases > Questions for the ADLS folks: > Is this a known issue? If so, will it be fixed soon? > Or is this expected behavior? > I'm able to deterministically reproduce it in my tests, with Impala on ADLS. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
