[ 
https://issues.apache.org/jira/browse/PHOENIX-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15364909#comment-15364909
 ] 

ASF GitHub Bot commented on PHOENIX-2405:
-----------------------------------------

Github user RCheungIT commented on a diff in the pull request:

    https://github.com/apache/phoenix/pull/175#discussion_r69792221
  
    --- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/SpoolingByteBufferSegmentQueue.java
 ---
    @@ -0,0 +1,357 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + * http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.phoenix.iterate;
    +
    +import org.apache.commons.io.input.CountingInputStream;
    +import org.apache.commons.io.output.DeferredFileOutputStream;
    +
    +import org.apache.phoenix.memory.MemoryManager;
    +import org.apache.phoenix.memory.MemoryManager.MemoryChunk;
    +import org.apache.phoenix.query.QueryServices;
    +import org.apache.phoenix.query.QueryServicesOptions;
    +
    +
    +import java.io.*;
    +import java.util.AbstractQueue;
    +import java.util.Iterator;
    +
    +import java.util.concurrent.atomic.AtomicInteger;
    +
    +import static org.apache.phoenix.monitoring.GlobalClientMetrics.*;
    +
    +
    +public abstract class SpoolingByteBufferSegmentQueue<T>  extends 
AbstractQueue<T> {
    --- End diff --
    
    Hi @maryannxue, it seems that I get your idea now. Do you mean I should not 
change the sort logic in memory?  What I only need to do is modify the flush 
logic, flush the sorted result to SpoolingResultIterator. Is that correct? 
Thanks.


> Improve performance and stability of server side sort for ORDER BY
> ------------------------------------------------------------------
>
>                 Key: PHOENIX-2405
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-2405
>             Project: Phoenix
>          Issue Type: Bug
>            Reporter: James Taylor
>            Assignee: Haoran Zhang
>              Labels: gsoc2016
>             Fix For: 4.8.0
>
>
> We currently use memory mapped files to buffer data as it's being sorted in 
> an ORDER BY (see MappedByteBufferQueue). The following types of exceptions 
> have been seen to occur:
> {code}
> Caused by: java.lang.OutOfMemoryError: Map failed
>         at sun.nio.ch.FileChannelImpl.map0(Native Method)
>         at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:904)
> {code}
> [~apurtell] has read that memory mapped files are not cleaned up after very 
> well in Java:
> {quote}
> "Map failed" means the JVM ran out of virtual address space. If you search 
> around stack overflow for suggestions on what to do when your app (in this 
> case Phoenix) encounters this issue when using mapped buffers, the answers 
> tend toward manually cleaning up the mapped buffers or explicitly triggering 
> a full GC. See 
> http://stackoverflow.com/questions/8553158/prevent-outofmemory-when-using-java-nio-mappedbytebuffer
>  for example. There are apparently long standing JVM/JRE problems with 
> reclamation of mapped buffers. I think we may want to explore in Phoenix a 
> different way to achieve what the current code is doing.
> {quote}
> Instead of using memory mapped files, we could use heap memory, or perhaps 
> there are other mechanisms too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to