Re: QuotaExceededException as a DoNotRetryIOException?
Thanks Andrew. +1 for the second option, I will create a jira for this change. Huaxiang > On Feb 7, 2018, at 9:25 AM, Huaxiang Sun wrote: > > Hi Mike, > >You are right. For rpc throttling, definitely it is retryable. For storage > quota, I think it will be fail faster (non-retryable). >We probably need to separate these two types of exceptions, I will do some > more research and follow up. > >Thanks, >Huaxiang > >> On Feb 7, 2018, at 9:16 AM, Mike Drob wrote: >> >> I think, philosophically, there can be two kinds of QEE - >> >> For throttling, we can retry. The quota is a temporal quota - you have done >> too many operations this minute, please try again next minute and >> everything will work. >> For storage, we shouldn't retry. The quota is a fixed quote - you have >> exceeded your allotted disk space, please do not try again until you have >> remedied the situation. >> >> Our current usage conflates the two, sometimes it is correct, sometimes not. >> >> On Wed, Feb 7, 2018 at 11:00 AM, Huaxiang Sun wrote: >> >>> Hi Stack, >>> >>> I run into a case that a mapreduce job in hive cannot finish because >>> it runs into a QEE. >>> I need to look into the hive mr task to see if QEE is not handled >>> correctly in hbase code or in hive code. >>> >>> I am thinking that if QEE is a retryable exception, then it should be >>> taken care of by the hbase code. >>> I will check more and report back. >>> >>> Thanks, >>> Huaxiang >>> On Feb 7, 2018, at 8:23 AM, Stack wrote: QEE being a DNRIOE seems right on the face of it. But if throttling, a DNRIOE is inappropriate. Where you seeing a QEE in a throttling scenario Huaxiang? Thanks, S On Tue, Feb 6, 2018 at 4:56 PM, Huaxiang Sun wrote: > Hi HBase devs, > > I found that QuotaExceededException is a DoNotRetryIOException, >>> which > is a bit strange from user’s point of view. > For rpc throttling, the exception is retryable and it tells app to > slow down and retry later. > > Any thoughts? > > Thanks, > Huaxiang >>> >>> >
Re: QuotaExceededException as a DoNotRetryIOException?
We have public class ThrottlingException extends QuotaExceededException public class QuotaExceededException extends DoNotRetryIOException Let the storage quota limits throw QuotaExceededException directly (based on DNRIOE). That seems fine. However, ThrottlingException is thrown as a result of a temporal quota, so it is inappropriate for this to inherit from DNRIOE, it should inherit IOException instead so the client is allowed to retry until successful, or until the retry policy is exhausted. We are in a bit of a pickle because we've released with this inheritance hierarchy, so to change it we will need a new minor, or we will want to deprecate ThrottlingException and use a new exception class instead, one which does not inherit from DNRIOE. On Wed, Feb 7, 2018 at 8:23 AM, Stack wrote: > QEE being a DNRIOE seems right on the face of it. > > But if throttling, a DNRIOE is inappropriate. Where you seeing a QEE in a > throttling scenario Huaxiang? > > Thanks, > S > > > On Tue, Feb 6, 2018 at 4:56 PM, Huaxiang Sun wrote: > > > Hi HBase devs, > > > > I found that QuotaExceededException is a DoNotRetryIOException, > which > > is a bit strange from user’s point of view. > > For rpc throttling, the exception is retryable and it tells app to > > slow down and retry later. > > > > Any thoughts? > > > > Thanks, > > Huaxiang > -- Best regards, Andrew Words like orphans lost among the crosstalk, meaning torn from truth's decrepit hands - A23, Crosstalk
Re: QuotaExceededException as a DoNotRetryIOException?
Hi Mike, You are right. For rpc throttling, definitely it is retryable. For storage quota, I think it will be fail faster (non-retryable). We probably need to separate these two types of exceptions, I will do some more research and follow up. Thanks, Huaxiang > On Feb 7, 2018, at 9:16 AM, Mike Drob wrote: > > I think, philosophically, there can be two kinds of QEE - > > For throttling, we can retry. The quota is a temporal quota - you have done > too many operations this minute, please try again next minute and > everything will work. > For storage, we shouldn't retry. The quota is a fixed quote - you have > exceeded your allotted disk space, please do not try again until you have > remedied the situation. > > Our current usage conflates the two, sometimes it is correct, sometimes not. > > On Wed, Feb 7, 2018 at 11:00 AM, Huaxiang Sun wrote: > >> Hi Stack, >> >>I run into a case that a mapreduce job in hive cannot finish because >> it runs into a QEE. >> I need to look into the hive mr task to see if QEE is not handled >> correctly in hbase code or in hive code. >> >> I am thinking that if QEE is a retryable exception, then it should be >> taken care of by the hbase code. >> I will check more and report back. >> >> Thanks, >> Huaxiang >> >>> On Feb 7, 2018, at 8:23 AM, Stack wrote: >>> >>> QEE being a DNRIOE seems right on the face of it. >>> >>> But if throttling, a DNRIOE is inappropriate. Where you seeing a QEE in a >>> throttling scenario Huaxiang? >>> >>> Thanks, >>> S >>> >>> >>> On Tue, Feb 6, 2018 at 4:56 PM, Huaxiang Sun wrote: >>> Hi HBase devs, I found that QuotaExceededException is a DoNotRetryIOException, >> which is a bit strange from user’s point of view. For rpc throttling, the exception is retryable and it tells app to slow down and retry later. Any thoughts? Thanks, Huaxiang >> >>
Re: QuotaExceededException as a DoNotRetryIOException?
I think, philosophically, there can be two kinds of QEE - For throttling, we can retry. The quota is a temporal quota - you have done too many operations this minute, please try again next minute and everything will work. For storage, we shouldn't retry. The quota is a fixed quote - you have exceeded your allotted disk space, please do not try again until you have remedied the situation. Our current usage conflates the two, sometimes it is correct, sometimes not. On Wed, Feb 7, 2018 at 11:00 AM, Huaxiang Sun wrote: > Hi Stack, > > I run into a case that a mapreduce job in hive cannot finish because > it runs into a QEE. > I need to look into the hive mr task to see if QEE is not handled > correctly in hbase code or in hive code. > >I am thinking that if QEE is a retryable exception, then it should be > taken care of by the hbase code. >I will check more and report back. > >Thanks, >Huaxiang > > > On Feb 7, 2018, at 8:23 AM, Stack wrote: > > > > QEE being a DNRIOE seems right on the face of it. > > > > But if throttling, a DNRIOE is inappropriate. Where you seeing a QEE in a > > throttling scenario Huaxiang? > > > > Thanks, > > S > > > > > > On Tue, Feb 6, 2018 at 4:56 PM, Huaxiang Sun wrote: > > > >> Hi HBase devs, > >> > >>I found that QuotaExceededException is a DoNotRetryIOException, > which > >> is a bit strange from user’s point of view. > >>For rpc throttling, the exception is retryable and it tells app to > >> slow down and retry later. > >> > >>Any thoughts? > >> > >>Thanks, > >>Huaxiang > >
Re: QuotaExceededException as a DoNotRetryIOException?
Hi Stack, I run into a case that a mapreduce job in hive cannot finish because it runs into a QEE. I need to look into the hive mr task to see if QEE is not handled correctly in hbase code or in hive code. I am thinking that if QEE is a retryable exception, then it should be taken care of by the hbase code. I will check more and report back. Thanks, Huaxiang > On Feb 7, 2018, at 8:23 AM, Stack wrote: > > QEE being a DNRIOE seems right on the face of it. > > But if throttling, a DNRIOE is inappropriate. Where you seeing a QEE in a > throttling scenario Huaxiang? > > Thanks, > S > > > On Tue, Feb 6, 2018 at 4:56 PM, Huaxiang Sun wrote: > >> Hi HBase devs, >> >>I found that QuotaExceededException is a DoNotRetryIOException, which >> is a bit strange from user’s point of view. >>For rpc throttling, the exception is retryable and it tells app to >> slow down and retry later. >> >>Any thoughts? >> >>Thanks, >>Huaxiang
Re: QuotaExceededException as a DoNotRetryIOException?
QEE being a DNRIOE seems right on the face of it. But if throttling, a DNRIOE is inappropriate. Where you seeing a QEE in a throttling scenario Huaxiang? Thanks, S On Tue, Feb 6, 2018 at 4:56 PM, Huaxiang Sun wrote: > Hi HBase devs, > > I found that QuotaExceededException is a DoNotRetryIOException, which > is a bit strange from user’s point of view. > For rpc throttling, the exception is retryable and it tells app to > slow down and retry later. > > Any thoughts? > > Thanks, > Huaxiang
