[
https://issues.apache.org/jira/browse/IMPALA-7860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16690009#comment-16690009
]
Joe McDonnell commented on IMPALA-7860:
---------------------------------------
[~jbapple]
I think ABFS support is useful for Impala and partitions ending in a period are
not super-common, so I don't really want to kill off ABFS support unless we
have no workaround. Some solutions:
# Hadoop ABFS connector throws an error if something tries to create a
directory or file ending in a period (or other invalid character) on ABFS. This
gets propagated through Impala so Impala fails to create partitions that end in
a period. This fixes the whole ecosystem, because the call would fail for Hive,
Spark, Impala, and everything else.
# Impala detects that it is doing something on ABFS and throws an error if
something tries to have a directory or file that ends in a period on ABFS.
# Hadoop ABFS gets it working.
# Impala does something special with partition names ending in period.
#3 sounds like it isn't happening, and if it was implemented at the connector
level, then other non-Hadoop ABFS utilities might not understand those
directories/files (which might be ok). For #4, if Impala does something
special, we need to be careful about compatibility with Hive, etc.
#1 is my preferred fix for now. This bug is not unique to Impala. I think Hive
and Spark are likely to have the same problem. (If they don't, let me know,
because that is interesting.) If Hadoop does nothing, Impala will probably do
#2 or disable ABFS support. For either #1 or #2, users would need to work
around it, but that may not impact very many users.
> Tests use partition name that isn't supported on ABFS
> -----------------------------------------------------
>
> Key: IMPALA-7860
> URL: https://issues.apache.org/jira/browse/IMPALA-7860
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Reporter: Sean Mackrory
> Priority: Major
>
> IMPALA-7681 introduced support for the ADLS Gen2 service / ABFS client. As
> mentioned in the code review for that
> (https://gerrit.cloudera.org/#/c/11630/) a couple of tests were failing
> because they use a partition name that ends with a period. If the tests are
> modified to end with anything other than a period, they work just fine.
> In HADOOP-15860, that's sounding like it's just a known limitation of the
> blob storage that shares infrastructure with ADLS Gen2 that won't be changing
> any time soon. I propose we modify the tests to just use a slightly different
> partition name.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
