yihua commented on code in PR #10957:
URL: https://github.com/apache/hudi/pull/10957#discussion_r1569752704
##########
hudi-common/src/main/java/org/apache/hudi/avro/AvroSchemaUtils.java:
##########
@@ -231,7 +231,13 @@ private static Option<Schema.Field> findNestedField(Schema
schema, String[] fiel
if (!nestedPart.isPresent()) {
return Option.empty();
}
- return nestedPart;
+ boolean isUnion = false;
Review Comment:
Could you explain this change? This affects the logic of all callers, not
just schema evolution.
##########
hudi-io/src/main/java/org/apache/hudi/common/util/ValidationUtils.java:
##########
@@ -76,4 +76,10 @@ public static void checkState(final boolean expression,
String errorMessage) {
throw new IllegalStateException(errorMessage);
}
}
+
+ public static void checkNotNull(Object o) {
+ if (o == null) {
+ throw new IllegalStateException();
Review Comment:
Add an error message to the exception.
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieHadoopFsRelationFactory.scala:
##########
@@ -64,7 +65,7 @@ abstract class HoodieBaseHadoopFsRelationFactory(val
sqlContext: SQLContext,
val schemaSpec:
Option[StructType]
) extends SparkAdapterSupport
with HoodieHadoopFsRelationFactory {
protected lazy val sparkSession: SparkSession = sqlContext.sparkSession
- protected lazy val optParams: Map[String, String] = options
+ protected var optParams: Map[String, String] = options
Review Comment:
Can we keep this immutable and modify the input `options`?
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieReaderContext.java:
##########
@@ -211,6 +221,10 @@ public Map<String, Object>
updateSchemaAndResetOrderingValInMetadata(Map<String,
*/
public abstract UnaryOperator<T> projectRecord(Schema from, Schema to);
+ public UnaryOperator<T> projectRecordUnsafe(Schema from, Schema to,
Map<String, String> renamedColumns) {
Review Comment:
Can this be unified with `projectRecord` instead of adding a new one?
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieReaderContext.java:
##########
@@ -48,6 +49,12 @@
* and {@code RowData} in Flink.
*/
public abstract class HoodieReaderContext<T> {
+ protected HoodieFileGroupReaderState<T> readerState = new
HoodieFileGroupReaderState<>();
+
+ public HoodieFileGroupReaderState<T> getReaderState() {
+ return readerState;
+ }
+
Review Comment:
Can this be passed in and immutable? It's bad idea to modify this just for
tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
