Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/229#discussion_r83208961
  
    --- Diff: 
processing/src/main/java/org/apache/carbondata/processing/newflow/DataLoadProcessorStep.java
 ---
    @@ -0,0 +1,40 @@
    +package org.apache.carbondata.processing.newflow;
    +
    +import java.util.Iterator;
    +
    +import 
org.apache.carbondata.processing.newflow.exception.CarbonDataLoadingException;
    +
    +/**
    + * This base interface for data loading. It can do transformation jobs as 
per the implementation.
    + *
    + */
    +public interface DataLoadProcessorStep {
    +
    +  /**
    +   * The output meta for this step. The data returns from this step is as 
per this meta.
    +   * @return
    +   */
    +  DataField[] getOutput();
    +
    +  /**
    +   * Intialization process for this step.
    +   * @param configuration
    +   * @param child
    +   * @throws CarbonDataLoadingException
    +   */
    +  void intialize(CarbonDataLoadConfiguration configuration, 
DataLoadProcessorStep child) throws
    +      CarbonDataLoadingException;
    +
    +  /**
    +   * Tranform the data as per the implemetation.
    +   * @return Iterator of data
    +   * @throws CarbonDataLoadingException
    +   */
    +  Iterator<Object[]> execute() throws CarbonDataLoadingException;
    --- End diff --
    
    I thought the SortStep is a singleton object within the executor, and if 
there are only one executor in one datanode, then the SortStep is sorting the 
data within datanode-scope, which is what we want. Synchronization means 
SortStep is thread-safe, so that multiple task can insert row into it. 
    Does your desing look like this? Otherwise how you ensure data is sorting 
within datanode?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Reply via email to