Peter Vary created HIVE-23883: --------------------------------- Summary: Streaming does not flush the side file Key: HIVE-23883 URL: https://issues.apache.org/jira/browse/HIVE-23883 Project: Hive Issue Type: Bug Components: Streaming, Transactions Reporter: Peter Vary
When a streaming write commits a mid-batch write with {{connection.commitTransaction()}} then it tries to flush the sideFile with {{OrcInputFormat.SHIMS.hflush(flushLengths)}}. This uses FSOutputSummer.flush, which does not flush the buffer data to the disk so the actual data is not written. Had to remove the check from the end of the streaming tests in {{TestCrudCompactorOnTez.java}} {code:java} CompactorTestUtilities.checkAcidVersion(fs.listFiles(new Path(table.getSd().getLocation()), true), fs, conf.getBoolVar(HiveConf.ConfVars.HIVE_WRITE_ACID_VERSION_FILE), new String[] { AcidUtils.DELTA_PREFIX }); {code} These checks verifies the {{_flush_length}} files, and they would fail otherwise. -- This message was sent by Atlassian Jira (v8.3.4#803005)