[jira] [Commented] (HADOOP-13134) WASB's file delete still throwing Blob not found exception
[ https://issues.apache.org/jira/browse/HADOOP-13134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346339#comment-16346339 ] Rohan Garg commented on HADOOP-13134: - Thanks for the analysis and the advice. Yes, I will start a spark JIRA for this. Yes, I will turn on caching to ensure correctness and performance. I agree that finalize in azure FS was trying to help by closing the fs. It should be the responsibility of the client to manage or preserve the FS object if certain behaviors are to be expected from it (esp. the ones which rely on GC, close etc.). > WASB's file delete still throwing Blob not found exception > -- > > Key: HADOOP-13134 > URL: https://issues.apache.org/jira/browse/HADOOP-13134 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.7.1 >Reporter: Lin Chan >Assignee: Thomas Marquardt >Priority: Major > > WASB is still throwing blob not found exception as shown in the following > stack. Need to catch that and convert to Boolean return code in WASB delete. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13134) WASB's file delete still throwing Blob not found exception
[ https://issues.apache.org/jira/browse/HADOOP-13134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332677#comment-16332677 ] Steve Loughran commented on HADOOP-13134: - # Thats unrelated, start with a JIRA for spark and see where it goes. # deleteOnExit isn't called on VM exit, it's called on close of the FS instance. If you aren't using the FS cache, then you have to manage the FS lifecycle itself and call close() when its time. Looks like Wasb is being helpful by closing the fs, which is then closing staging entries. # There's no finalize() call in s3a. I'm a fan of them because you can never be sure when they're called. But I'm also not a fan of the entire FS shutdown process, as it can get into a mess when you are shutting down to network errors and the filesystems are trying to run through their lists of things to delete. Given that any form of clean termination is a best-effort outcome, I'd argue for writing code which doesn't rely on it. Personally, I'd turn caching back on. You gain from reuse of the worker thread pools and avoid all startup delays, which matters as every worker thread in a spark query is going to be calling fs.get() on init, at the very least. It's a major perf killer. You've just found a different problem > WASB's file delete still throwing Blob not found exception > -- > > Key: HADOOP-13134 > URL: https://issues.apache.org/jira/browse/HADOOP-13134 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.7.1 >Reporter: Lin Chan >Assignee: Thomas Marquardt >Priority: Major > > WASB is still throwing blob not found exception as shown in the following > stack. Need to catch that and convert to Boolean return code in WASB delete. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13134) WASB's file delete still throwing Blob not found exception
[ https://issues.apache.org/jira/browse/HADOOP-13134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326413#comment-16326413 ] Rohan Garg commented on HADOOP-13134: - [~ste...@apache.org] : one of cases where I discovered a similar bug (parent blob for a key suddenly got missing while doing a rename operation) was with a spark 2.1 query. It was an 'insert into' query. spark's code uses 'deleteOnExit' method on staging directory ([https://github.com/apache/spark/blob/branch-2.1/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala#L118]). But from what I saw, spark's code doesn't try to preserve the filesystem object to which it latched the staging directory's deleteOnExit. Also, we were not using file system caching in hadoop. So, whenever the Azure FS object on which deleteOnExit was called got GCed, finalize method for FS object was called which in-turn calls the FileSystem#close(). The file system's close method ended up deleting the staging directory. This posed a problem as we were doing renames in the main thread which failed leading to query failure. > WASB's file delete still throwing Blob not found exception > -- > > Key: HADOOP-13134 > URL: https://issues.apache.org/jira/browse/HADOOP-13134 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.7.1 >Reporter: Lin Chan >Assignee: Dushyanth >Priority: Major > > WASB is still throwing blob not found exception as shown in the following > stack. Need to catch that and convert to Boolean return code in WASB delete. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13134) WASB's file delete still throwing Blob not found exception
[ https://issues.apache.org/jira/browse/HADOOP-13134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297046#comment-16297046 ] Steve Loughran commented on HADOOP-13134: - [~zichensun] thanks for that little hint; does make me thing flat listing is the solution. Probably a performance booster too. might be worth changing to make the default > WASB's file delete still throwing Blob not found exception > -- > > Key: HADOOP-13134 > URL: https://issues.apache.org/jira/browse/HADOOP-13134 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.7.1 >Reporter: Lin Chan >Assignee: Dushyanth > > WASB is still throwing blob not found exception as shown in the following > stack. Need to catch that and convert to Boolean return code in WASB delete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13134) WASB's file delete still throwing Blob not found exception
[
https://issues.apache.org/jira/browse/HADOOP-13134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267968#comment-16267968
]
Zichen Sun commented on HADOOP-13134:
-
We had a similar issue reported a few times when doing delete directories, and
the workaround was to set fs.azure.flatlist.enable to true, introduced by
https://issues.apache.org/jira/browse/HADOOP-13403, which skips the following
if (!enableFlatListing) { // Currently at a depth of one, decrement the
listing depth for // sub-directories. buildUpList(directory,
fileMetadata, maxListingCount,maxListingDepth - 1);}
> WASB's file delete still throwing Blob not found exception
> --
>
> Key: HADOOP-13134
> URL: https://issues.apache.org/jira/browse/HADOOP-13134
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.7.1
>Reporter: Lin Chan
>Assignee: Dushyanth
>
> WASB is still throwing blob not found exception as shown in the following
> stack. Need to catch that and convert to Boolean return code in WASB delete.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13134) WASB's file delete still throwing Blob not found exception
[ https://issues.apache.org/jira/browse/HADOOP-13134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112622#comment-16112622 ] Steve Loughran commented on HADOOP-13134: - Looking into the cause, it's because the parent directory of a deleted object doesn't exist. This may be a race condition in the delete, or some ordering of the delete process, maybe > WASB's file delete still throwing Blob not found exception > -- > > Key: HADOOP-13134 > URL: https://issues.apache.org/jira/browse/HADOOP-13134 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.7.1 >Reporter: Lin Chan >Assignee: Dushyanth > > WASB is still throwing blob not found exception as shown in the following > stack. Need to catch that and convert to Boolean return code in WASB delete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13134) WASB's file delete still throwing Blob not found exception
[
https://issues.apache.org/jira/browse/HADOOP-13134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112613#comment-16112613
]
Steve Loughran commented on HADOOP-13134:
-
WORKAROUND: tell the job committer to ignore failures in cleanup
As discussed in [Spark Cloud
Integration|https://github.com/apache/spark/blob/master/docs/cloud-integration.md],
you can downgrade failures during cleanup to warnings. I recommend this
against object stores for a slightly more robust commit, given that directory
delete is a more complex/brittle operation.& more prone to failures
{code}
spark.hadoop.mapreduce.fileoutputcommitter.cleanup-failures.ignored true
{code}
> WASB's file delete still throwing Blob not found exception
> --
>
> Key: HADOOP-13134
> URL: https://issues.apache.org/jira/browse/HADOOP-13134
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.7.1
>Reporter: Lin Chan
>Assignee: Dushyanth
>
> WASB is still throwing blob not found exception as shown in the following
> stack. Need to catch that and convert to Boolean return code in WASB delete.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13134) WASB's file delete still throwing Blob not found exception
[ https://issues.apache.org/jira/browse/HADOOP-13134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109792#comment-16109792 ] Steve Loughran commented on HADOOP-13134: - there's a set of things which object stores often let you do, and which the blobstore clients often unintentionally let you continue to do, which break our fundamental expectations of an FS * create files under files * create directories under files * rename things under themselves via copy (unless you try hard to detect and stop this) + others most of these surface the stricter the tests we do for FS compliance. They tend not to surface in the wild as much, because people writing code assume that the far end is a filesystem, so their code conforms to the set of shared assumptions. Here's a case of it. Given it's actually happening in the hadoop file output committer, it may surface in production, so I think mertis fixing. The other ones, the ones which only show up if we discover a bug in our own mental model of filesystems and write tests for themn, those I worry less about. > WASB's file delete still throwing Blob not found exception > -- > > Key: HADOOP-13134 > URL: https://issues.apache.org/jira/browse/HADOOP-13134 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.7.1 >Reporter: Lin Chan >Assignee: Dushyanth > > WASB is still throwing blob not found exception as shown in the following > stack. Need to catch that and convert to Boolean return code in WASB delete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13134) WASB's file delete still throwing Blob not found exception
[ https://issues.apache.org/jira/browse/HADOOP-13134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16101131#comment-16101131 ] Shane Mainali commented on HADOOP-13134: [~ste...@apache.org], what do you think about the priority of this one? I first thought it was perhaps more serious given the stack but then thought it's more of a caller issue trying to do this when the parent directory doesn't exist. Also from looking at a few other related cases, including https://issues.apache.org/jira/browse/HADOOP-14583, it seems that this may also be minor? Wanted to see what your take is so we could agree on the priority. > WASB's file delete still throwing Blob not found exception > -- > > Key: HADOOP-13134 > URL: https://issues.apache.org/jira/browse/HADOOP-13134 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.7.1 >Reporter: Lin Chan >Assignee: Dushyanth > > WASB is still throwing blob not found exception as shown in the following > stack. Need to catch that and convert to Boolean return code in WASB delete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13134) WASB's file delete still throwing Blob not found exception
[ https://issues.apache.org/jira/browse/HADOOP-13134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063024#comment-16063024 ] Steve Loughran commented on HADOOP-13134: - I propose here that the client must not assume that the parent dir exists; if it gets an FNFE equivalent there, it must correct by creating the parent entry > WASB's file delete still throwing Blob not found exception > -- > > Key: HADOOP-13134 > URL: https://issues.apache.org/jira/browse/HADOOP-13134 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.7.1 >Reporter: Lin Chan >Assignee: Dushyanth > > WASB is still throwing blob not found exception as shown in the following > stack. Need to catch that and convert to Boolean return code in WASB delete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13134) WASB's file delete still throwing Blob not found exception
[ https://issues.apache.org/jira/browse/HADOOP-13134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063022#comment-16063022 ] Steve Loughran commented on HADOOP-13134: - that stack trace of mine is HADOOP-14583 > WASB's file delete still throwing Blob not found exception > -- > > Key: HADOOP-13134 > URL: https://issues.apache.org/jira/browse/HADOOP-13134 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.7.1 >Reporter: Lin Chan >Assignee: Dushyanth > > WASB is still throwing blob not found exception as shown in the following > stack. Need to catch that and convert to Boolean return code in WASB delete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13134) WASB's file delete still throwing Blob not found exception
[ https://issues.apache.org/jira/browse/HADOOP-13134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15794772#comment-15794772 ] Steve Loughran commented on HADOOP-13134: - The implication is that there is no parent folder here...is that an issue? Should a parent folder be created? > WASB's file delete still throwing Blob not found exception > -- > > Key: HADOOP-13134 > URL: https://issues.apache.org/jira/browse/HADOOP-13134 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 2.7.1 >Reporter: Lin Chan >Assignee: Dushyanth > > WASB is still throwing blob not found exception as shown in the following > stack. Need to catch that and convert to Boolean return code in WASB delete. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13134) WASB's file delete still throwing Blob not found exception
[
https://issues.apache.org/jira/browse/HADOOP-13134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15794767#comment-15794767
]
Steve Loughran commented on HADOOP-13134:
-
{code}
16/05/07 01:24:57 ERROR InsertIntoHadoopFsRelation: Aborting job.
org.apache.hadoop.fs.azure.AzureException:
com.microsoft.azure.storage.StorageException: The specified blob does not exist.
at
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2682)
at
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2693)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.updateParentFolderLastModifiedTime(NativeAzureFileSystem.java:2495)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1860)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1603)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1836)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1603)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1836)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1603)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1836)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1603)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1836)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1603)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1836)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1603)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1836)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1603)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.cleanupJob(FileOutputCommitter.java:510)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJobInternal(FileOutputCommitter.java:403)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:364)
at
org.apache.parquet.hadoop.ParquetOutputCommitter.commitJob(ParquetOutputCommitter.java:46)
at
org.apache.spark.sql.execution.datasources.BaseWriterContainer.commitJob(WriterContainer.scala:230)
at
org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply$mcV$sp(InsertIntoHadoopFsRelation.scala:151)
at
org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:108)
at
org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:108)
at
org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:56)
at
org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation.run(InsertIntoHadoopFsRelation.scala:108)
{code}
> WASB's file delete still throwing Blob not found exception
> --
>
> Key: HADOOP-13134
> URL: https://issues.apache.org/jira/browse/HADOOP-13134
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 2.7.1
>Reporter: Lin Chan
>Assignee: Dushyanth
>
> WASB is still throwing blob not found exception as shown in the following
> stack. Need to catch that and convert to Boolean return code in WASB delete.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
