[jira] [Commented] (IMPALA-7860) Tests use partition name that isn't supported on ABFS
[ https://issues.apache.org/jira/browse/IMPALA-7860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16690031#comment-16690031 ] Joe McDonnell commented on IMPALA-7860: --- Ok, great! Thanks for looking into this. I looked at the test, and it was added six years ago for IMPALA-72 (two digits!). It is 100% happenstance that it ends in a period. > Tests use partition name that isn't supported on ABFS > - > > Key: IMPALA-7860 > URL: https://issues.apache.org/jira/browse/IMPALA-7860 > Project: IMPALA > Issue Type: Bug > Components: Infrastructure >Reporter: Sean Mackrory >Priority: Major > > IMPALA-7681 introduced support for the ADLS Gen2 service / ABFS client. As > mentioned in the code review for that > (https://gerrit.cloudera.org/#/c/11630/) a couple of tests were failing > because they use a partition name that ends with a period. If the tests are > modified to end with anything other than a period, they work just fine. > In HADOOP-15860, that's sounding like it's just a known limitation of the > blob storage that shares infrastructure with ADLS Gen2 that won't be changing > any time soon. I propose we modify the tests to just use a slightly different > partition name. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7860) Tests use partition name that isn't supported on ABFS
[ https://issues.apache.org/jira/browse/IMPALA-7860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16690015#comment-16690015 ] Sean Mackrory commented on IMPALA-7860: --- I agree with Joe that #1 is the right fix. I will push for that and submit a patch. I'll leave this particular JIRA open though, as we will also need to tweak the tests, otherwise they'll still fail. It'll just be more apparent why they fail. > Tests use partition name that isn't supported on ABFS > - > > Key: IMPALA-7860 > URL: https://issues.apache.org/jira/browse/IMPALA-7860 > Project: IMPALA > Issue Type: Bug > Components: Infrastructure >Reporter: Sean Mackrory >Priority: Major > > IMPALA-7681 introduced support for the ADLS Gen2 service / ABFS client. As > mentioned in the code review for that > (https://gerrit.cloudera.org/#/c/11630/) a couple of tests were failing > because they use a partition name that ends with a period. If the tests are > modified to end with anything other than a period, they work just fine. > In HADOOP-15860, that's sounding like it's just a known limitation of the > blob storage that shares infrastructure with ADLS Gen2 that won't be changing > any time soon. I propose we modify the tests to just use a slightly different > partition name. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7860) Tests use partition name that isn't supported on ABFS
[ https://issues.apache.org/jira/browse/IMPALA-7860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16690009#comment-16690009 ] Joe McDonnell commented on IMPALA-7860: --- [~jbapple] I think ABFS support is useful for Impala and partitions ending in a period are not super-common, so I don't really want to kill off ABFS support unless we have no workaround. Some solutions: # Hadoop ABFS connector throws an error if something tries to create a directory or file ending in a period (or other invalid character) on ABFS. This gets propagated through Impala so Impala fails to create partitions that end in a period. This fixes the whole ecosystem, because the call would fail for Hive, Spark, Impala, and everything else. # Impala detects that it is doing something on ABFS and throws an error if something tries to have a directory or file that ends in a period on ABFS. # Hadoop ABFS gets it working. # Impala does something special with partition names ending in period. #3 sounds like it isn't happening, and if it was implemented at the connector level, then other non-Hadoop ABFS utilities might not understand those directories/files (which might be ok). For #4, if Impala does something special, we need to be careful about compatibility with Hive, etc. #1 is my preferred fix for now. This bug is not unique to Impala. I think Hive and Spark are likely to have the same problem. (If they don't, let me know, because that is interesting.) If Hadoop does nothing, Impala will probably do #2 or disable ABFS support. For either #1 or #2, users would need to work around it, but that may not impact very many users. > Tests use partition name that isn't supported on ABFS > - > > Key: IMPALA-7860 > URL: https://issues.apache.org/jira/browse/IMPALA-7860 > Project: IMPALA > Issue Type: Bug > Components: Infrastructure >Reporter: Sean Mackrory >Priority: Major > > IMPALA-7681 introduced support for the ADLS Gen2 service / ABFS client. As > mentioned in the code review for that > (https://gerrit.cloudera.org/#/c/11630/) a couple of tests were failing > because they use a partition name that ends with a period. If the tests are > modified to end with anything other than a period, they work just fine. > In HADOOP-15860, that's sounding like it's just a known limitation of the > blob storage that shares infrastructure with ADLS Gen2 that won't be changing > any time soon. I propose we modify the tests to just use a slightly different > partition name. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7860) Tests use partition name that isn't supported on ABFS
[ https://issues.apache.org/jira/browse/IMPALA-7860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16689967#comment-16689967 ] Jim Apple commented on IMPALA-7860: --- [~joemcdonnell], I agree, but if this is a limitation of a proprietary storage system, we're left with a few options: 1. Say Impala does not support ADLS Gen2. Maybe back out the patch. 2. Append something on the end of partition names to prevent a period being something we end a partition name with 3. Accept that some storage systems are buggy and that Impala does the best it can It sounds like #3 is not OK in your view. We might differ on that point. Putting that aside for a moment, do you prefer #1 or #2? > Tests use partition name that isn't supported on ABFS > - > > Key: IMPALA-7860 > URL: https://issues.apache.org/jira/browse/IMPALA-7860 > Project: IMPALA > Issue Type: Bug > Components: Infrastructure >Reporter: Sean Mackrory >Priority: Major > > IMPALA-7681 introduced support for the ADLS Gen2 service / ABFS client. As > mentioned in the code review for that > (https://gerrit.cloudera.org/#/c/11630/) a couple of tests were failing > because they use a partition name that ends with a period. If the tests are > modified to end with anything other than a period, they work just fine. > In HADOOP-15860, that's sounding like it's just a known limitation of the > blob storage that shares infrastructure with ADLS Gen2 that won't be changing > any time soon. I propose we modify the tests to just use a slightly different > partition name. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7860) Tests use partition name that isn't supported on ABFS
[ https://issues.apache.org/jira/browse/IMPALA-7860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16689760#comment-16689760 ] Joe McDonnell commented on IMPALA-7860: --- Partitions ending in a period should work or it should fail with an error. Silently doing something wrong is not ok. > Tests use partition name that isn't supported on ABFS > - > > Key: IMPALA-7860 > URL: https://issues.apache.org/jira/browse/IMPALA-7860 > Project: IMPALA > Issue Type: Bug > Components: Infrastructure >Reporter: Sean Mackrory >Priority: Major > > IMPALA-7681 introduced support for the ADLS Gen2 service / ABFS client. As > mentioned in the code review for that > (https://gerrit.cloudera.org/#/c/11630/) a couple of tests were failing > because they use a partition name that ends with a period. If the tests are > modified to end with anything other than a period, they work just fine. > In HADOOP-15860, that's sounding like it's just a known limitation of the > blob storage that shares infrastructure with ADLS Gen2 that won't be changing > any time soon. I propose we modify the tests to just use a slightly different > partition name. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
