[jira] [Created] (TAJO-1144) INSERT INTO with output rotation may overwrite previous written output file.

[Hyunsik Choi (JIRA)]

Hyunsik Choi created TAJO-1144:
----------------------------------

             Summary: INSERT INTO with output rotation may overwrite previous 
written output file.
                 Key: TAJO-1144
                 URL: https://issues.apache.org/jira/browse/TAJO-1144
             Project: Tajo
          Issue Type: Improvement
          Components: physical operator
            Reporter: Hyunsik Choi
            Assignee: Hyunsik Choi
            Priority: Critical
             Fix For: 0.9.1


Tajo can write multiple output files according to a given maximum file size. We 
can this feature 'output file rotation'.

BTW, INSERT with output file rotation may overwrite the previous written output 
file. See the source code in StoreTableExec.

{noformat}
public void openNewFile(int suffixId) throws IOException {
    String prevFile = null;

    lastFileName = context.getOutputPath();
    if (suffixId > 0) {
      prevFile = lastFileName.toString();

      lastFileName = new Path(lastFileName + "_" + suffixId);
    }

    if (plan instanceof InsertNode) {
      InsertNode createTableNode = (InsertNode) plan;
      appender = 
StorageManager.getStorageManager(context.getConf()).getAppender(meta,
          createTableNode.getTableSchema(), context.getOutputPath());
    } else {
      appender = 
StorageManager.getStorageManager(context.getConf()).getAppender(meta, 
outSchema, lastFileName);
    }

    appender.enableStats();
    appender.init();

    if (suffixId > 0) {
      LOG.info(prevFile + " exceeds " + 
SessionVars.MAX_OUTPUT_FILE_SIZE.keyname() + " (" + maxPerFileSize + " MB), " +
          "The remain output will be written into " + lastFileName.toString());
    }
  }
{noformat}

Especially, {{context.getOutputPath\(\)}} should be lastFileName in the below 
part.
{noformat}
    if (plan instanceof InsertNode) {
      InsertNode createTableNode = (InsertNode) plan;
      appender = 
StorageManager.getStorageManager(context.getConf()).getAppender(meta,
          createTableNode.getTableSchema(), context.getOutputPath());
    } else {
      appender = 
StorageManager.getStorageManager(context.getConf()).getAppender(meta, 
outSchema, lastFileName);
    }
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)