xushiyan commented on a change in pull request #2873:
URL: https://github.com/apache/hudi/pull/2873#discussion_r619698467
##########
File path:
hudi-hadoop-mr/src/test/java/org/apache/hudi/hadoop/realtime/TestHoodieRealtimeRecordReader.java
##########
@@ -237,13 +231,13 @@ public void testUnMergedReader() throws Exception {
// create a split with baseFile (parquet file written earlier) and new log
file(s)
String logFilePath = writer.getLogFile().getPath().toString();
HoodieRealtimeFileSplit split = new HoodieRealtimeFileSplit(
- new FileSplit(new Path(partitionDir + "/fileid0_1-0-1_" + instantTime
+ ".parquet"), 0, 1, jobConf),
- basePath.toString(), Collections.singletonList(logFilePath),
newCommitTime);
+ new FileSplit(new Path(partitionDir + "/fileid0_1-0-1_" + instantTime
+ ".parquet"), 0, 1, baseJobConf),
+ basePath.toUri().toString(), Collections.singletonList(logFilePath),
newCommitTime);
// create a RecordReader to be used by HoodieRealtimeRecordReader
RecordReader<NullWritable, ArrayWritable> reader = new
MapredParquetInputFormat().getRecordReader(
- new FileSplit(split.getPath(), 0, fs.getLength(split.getPath()),
(String[]) null), jobConf, null);
- JobConf jobConf = new JobConf();
+ new FileSplit(split.getPath(), 0, fs.getLength(split.getPath()),
(String[]) null), baseJobConf, null);
+ JobConf jobConf = new JobConf(baseJobConf);
Review comment:
@vinothchandar this `jobConf` is needed for setting some hive col
properties via `setHiveColumnNameProps()`
##########
File path:
hudi-hadoop-mr/src/test/java/org/apache/hudi/hadoop/realtime/TestHoodieRealtimeRecordReader.java
##########
@@ -237,13 +231,13 @@ public void testUnMergedReader() throws Exception {
// create a split with baseFile (parquet file written earlier) and new log
file(s)
String logFilePath = writer.getLogFile().getPath().toString();
HoodieRealtimeFileSplit split = new HoodieRealtimeFileSplit(
- new FileSplit(new Path(partitionDir + "/fileid0_1-0-1_" + instantTime
+ ".parquet"), 0, 1, jobConf),
- basePath.toString(), Collections.singletonList(logFilePath),
newCommitTime);
+ new FileSplit(new Path(partitionDir + "/fileid0_1-0-1_" + instantTime
+ ".parquet"), 0, 1, baseJobConf),
+ basePath.toUri().toString(), Collections.singletonList(logFilePath),
newCommitTime);
Review comment:
@vinothchandar Tests are passing now. the issue is isolated with this
test case. do you concern about this change masking the issue with Azure
environment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
