[
https://issues.apache.org/jira/browse/TAJO-1144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jaehwa Jung reassigned TAJO-1144:
---------------------------------
Assignee: Jaehwa Jung (was: Hyunsik Choi)
> INSERT INTO with output rotation may overwrite previous written output file.
> ----------------------------------------------------------------------------
>
> Key: TAJO-1144
> URL: https://issues.apache.org/jira/browse/TAJO-1144
> Project: Tajo
> Issue Type: Improvement
> Components: physical operator
> Reporter: Hyunsik Choi
> Assignee: Jaehwa Jung
> Priority: Critical
> Fix For: 0.10
>
>
> Tajo can write multiple output files according to a given maximum file size.
> We can this feature 'output file rotation'.
> BTW, INSERT with output file rotation may overwrite the previous written
> output file. See the source code in StoreTableExec.
> {noformat}
> public void openNewFile(int suffixId) throws IOException {
> String prevFile = null;
> lastFileName = context.getOutputPath();
> if (suffixId > 0) {
> prevFile = lastFileName.toString();
> lastFileName = new Path(lastFileName + "_" + suffixId);
> }
> if (plan instanceof InsertNode) {
> InsertNode createTableNode = (InsertNode) plan;
> appender =
> StorageManager.getStorageManager(context.getConf()).getAppender(meta,
> createTableNode.getTableSchema(), context.getOutputPath());
> } else {
> appender =
> StorageManager.getStorageManager(context.getConf()).getAppender(meta,
> outSchema, lastFileName);
> }
> appender.enableStats();
> appender.init();
> if (suffixId > 0) {
> LOG.info(prevFile + " exceeds " +
> SessionVars.MAX_OUTPUT_FILE_SIZE.keyname() + " (" + maxPerFileSize + " MB), "
> +
> "The remain output will be written into " +
> lastFileName.toString());
> }
> }
> {noformat}
> Especially, {{context.getOutputPath\(\)}} should be lastFileName in the below
> part.
> {noformat}
> if (plan instanceof InsertNode) {
> InsertNode createTableNode = (InsertNode) plan;
> appender =
> StorageManager.getStorageManager(context.getConf()).getAppender(meta,
> createTableNode.getTableSchema(), context.getOutputPath());
> } else {
> appender =
> StorageManager.getStorageManager(context.getConf()).getAppender(meta,
> outSchema, lastFileName);
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
