[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-12678: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) +1 for patch v06. I have committed this to trunk, branch-2 and branch-2.8. [~madhuch-ms], thank you for contributing this patch. > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Fix For: 2.8.0 > > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch, > HADOOP-12678.003.patch, HADOOP-12678.004.patch, HADOOP-12678.005.patch, > HADOOP-12678.006.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Status: Open (was: Patch Available) > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch, > HADOOP-12678.003.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Status: Patch Available (was: Open) > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch, > HADOOP-12678.003.patch, HADOOP-12678.004.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Attachment: HADOOP-12678.004.patch > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch, > HADOOP-12678.003.patch, HADOOP-12678.004.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Status: Open (was: Patch Available) > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch, > HADOOP-12678.003.patch, HADOOP-12678.004.patch, HADOOP-12678.005.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Attachment: HADOOP-12678.005.patch > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch, > HADOOP-12678.003.patch, HADOOP-12678.004.patch, HADOOP-12678.005.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Status: Patch Available (was: Open) > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch, > HADOOP-12678.003.patch, HADOOP-12678.004.patch, HADOOP-12678.005.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Attachment: HADOOP-12678.006.patch > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch, > HADOOP-12678.003.patch, HADOOP-12678.004.patch, HADOOP-12678.005.patch, > HADOOP-12678.006.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Status: Patch Available (was: Open) > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch, > HADOOP-12678.003.patch, HADOOP-12678.004.patch, HADOOP-12678.005.patch, > HADOOP-12678.006.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Status: Open (was: Patch Available) > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch, > HADOOP-12678.003.patch, HADOOP-12678.004.patch, HADOOP-12678.005.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Attachment: (was: HADOOP-12678.006.patch) > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch, > HADOOP-12678.003.patch, HADOOP-12678.004.patch, HADOOP-12678.005.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Attachment: HADOOP-12678.006.patch > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch, > HADOOP-12678.003.patch, HADOOP-12678.004.patch, HADOOP-12678.005.patch, > HADOOP-12678.006.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Status: Patch Available (was: Open) > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch, > HADOOP-12678.003.patch, HADOOP-12678.004.patch, HADOOP-12678.005.patch, > HADOOP-12678.006.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Status: Open (was: Patch Available) > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch, > HADOOP-12678.003.patch, HADOOP-12678.004.patch, HADOOP-12678.005.patch, > HADOOP-12678.006.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Status: Patch Available (was: Open) > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Attachment: HADOOP-12678.001.patch > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Status: Patch Available (was: Open) > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Status: Open (was: Patch Available) > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Attachment: HADOOP-12678.002.patch > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Status: Open (was: Patch Available) > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch, > HADOOP-12678.003.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Status: Patch Available (was: Open) > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch, > HADOOP-12678.003.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Attachment: HADOOP-12678.003.patch > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch, > HADOOP-12678.003.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
