karanmehta93 commented on issue #419: PHOENIX-4009 Run UPDATE STATISTICS command by using MR integration on… URL: https://github.com/apache/phoenix/pull/419#issuecomment-452927962 > this is a common case - we always have some jobs failing after they finish most of work (updated > 90% regions) but fail due to some reason, the retry job should just do minimal work instead of update stats of the whole table again. This should not be common case. Common case should be that few mappers can fail due to some reason. We have retries for that. The job should NOT fail as a whole. Even MR framework doesn't persist data between jobs and can get really tricky depending on use cases. Better way is to make the job idempotent so that re-run doesn't affect it. Also it is hard in this case since we don't know what changed between retries. If snapshot name changed, that can potentially affect region boundaries. We would need another level of orchestration that persists stats MR job information in some table and we look it up before running the current job. These cases would be difficult to handle. I would prefer that we try to avoid that complexity here.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
