[
https://issues.apache.org/jira/browse/HADOOP-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12603826#action_12603826
]
Runping Qi commented on HADOOP-3442:
------------------------------------
bq. The 2*log(n) heuristic was intended to bail out of a worst case, not to
protect against the StackOverflowError. The recursion on the system stack is
already limited to log(n), but quicksort will always have O(n^2) cases that
this needs to handle. I'm happy to discuss an appropriate value for k, but
k*log(n) seems appropriate to this purpose.
I guess it is safe to choose a larger k. For example, with k = 8, the stack
depth is bounded by 250 in practice, but a lot fewer cases of input data
patterns will need to invoke the heapsort.
Even with all these fixes, I still believe it is good to add randomization to
pivotal selection (or randomize the input data upfront).
> QuickSort may get into unbounded recursion
> ------------------------------------------
>
> Key: HADOOP-3442
> URL: https://issues.apache.org/jira/browse/HADOOP-3442
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Runping Qi
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.17.1, 0.18.0
>
> Attachments: 3442-0.patch, 3442-0v17.patch, 3442-1.patch,
> 3442-2.patch, 3442-3.patch, CheckSortBuffer.java, HADOOP-3442.patch,
> overflow.zip, spillbuffers.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
