A possible root cause is the filed of record is null.
public static String getNestedFieldValAsString(GenericRecord record,
String fieldName) {
String[] parts = fieldName.split("\\.");
GenericRecord valueNode = record;
int i = 0;
for (;i < parts.length; i++) {
String part = parts[i];
Object val = valueNode.get(part);
if (val == null) {
break;
}
// return, if last part of name
if (i == parts.length - 1) {
return val.toString();
} else {
// VC: Need a test here
if (!(val instanceof GenericRecord)) {
throw new HoodieException("Cannot find a record at part value
:" + part);
}
valueNode = (GenericRecord) val;
}
}
throw new HoodieException(fieldName + "(Part -" + parts[i] + ")
field not found in record. "
+ "Acceptable fields were :" + valueNode.getSchema().getFields()
.stream().map(Field::name).collect(Collectors.toList()));
}
Vinoth Chandar <[email protected]> 于2019年3月10日周日 下午2:11写道：
> Hmmm. Thats interesting. I can see that the parsing works, since the
> exception said "Part - review_date". There are definitely users who have
> done this before.
> So not sure what's going on.
>
> Can you paste the generated Avro schema? following is the corresponding
> code line
> log.info(s"Registered avro schema : ${schema.toString(true)}")
>
> May be create a gist (gist.github.com), for easier sharing of
> code/stacktrace?
> Thanks
> Vinoth
>
> On Sat, Mar 9, 2019 at 1:33 PM Umesh Kacha <[email protected]> wrote:
>
> > Hi Vinoth thanks I have already did and checked that please see red
> column
> > highlighted below.
> >
> > root |-- marketplace: string (nullable = true) |-- customer_id: string
> > (nullable = true) |-- review_id: string (nullable = true) |-- product_id:
> > string (nullable = true) |-- product_parent: string (nullable = true) |--
> > product_title: string (nullable = true) |-- product_category: string
> > (nullable = true) |-- star_rating: string (nullable = true) |--
> > helpful_votes: string (nullable = true) |-- total_votes: string
> (nullable =
> > true) |-- vine: string (nullable = true) |-- verified_purchase: string
> > (nullable = true) |-- review_headline: string (nullable = true) |--
> > review_body: string (nullable = true) |-- review_date: string (nullable =
> > true) |-- year: integer (nullable = true)
> >
> > On Sun, Mar 10, 2019 at 2:27 AM Vinoth Chandar <[email protected]>
> wrote:
> >
> > > Hi,
> > >
> > > >>review_date(Part
> > > -review_date) field not found in record
> > >
> > > Seems like the precombine field is not in the input DF? Can you try
> doing
> > > df1.printSchema and check that once?
> > >
> > > On Sat, Mar 9, 2019 at 11:52 AM Umesh Kacha <[email protected]>
> > wrote:
> > >
> > > > Hi I have the following code using which I am trying to bulk insert
> > huge
> > > > csv file loaded into Spark DataFrame but it fails saying column
> > > review_date
> > > > not found but that column is definitely there in dataframe. Please
> > guide.
> > > >
> > > > df1.write
> > > > .format("com.uber.hoodie")
> > > > .option(DataSourceWriteOptions.STORAGE_TYPE_OPT_KEY,
> > > > HoodieTableType.COPY_ON_WRITE.name())
> > > > .option(DataSourceWriteOptions.OPERATION_OPT_KEY,
> > > > DataSourceWriteOptions.BULK_INSERT_OPERATION_OPT_VAL) // insert
> > > > .option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY,
> > > > "customer_id")
> > > > .option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY,
> > "year")
> > > > .option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY,
> > > > "review_date")
> > > > .option(HoodieWriteConfig.TABLE_NAME, "hoodie_test_table")
> > > > .mode(SaveMode.Overwrite)
> > > > .save("/tmp/hoodie/test_hoodie")
> > > >
> > > >
> > > > Caused by: com.uber.hoodie.exception.HoodieException:
> review_date(Part
> > > > -review_date) field not found in record. Acceptable fields were
> > > > :[marketplace, customer_id, review_id, product_id, product_parent,
> > > > product_title, product_category, star_rating, helpful_votes,
> > total_votes,
> > > > vine, verified_purchase, review_headline, review_body, review_date,
> > year]
> > > > at
> > > >
> > > >
> > >
> >
> com.uber.hoodie.DataSourceUtils.getNestedFieldValAsString(DataSourceUtils.java:79)
> > > > at
> > > >
> > > >
> > >
> >
> com.uber.hoodie.HoodieSparkSqlWriter$$anonfun$1.apply(HoodieSparkSqlWriter.scala:93)
> > > > at
> > > >
> > > >
> > >
> >
> com.uber.hoodie.HoodieSparkSqlWriter$$anonfun$1.apply(HoodieSparkSqlWriter.scala:92)
> > > > at scala.collection.Iterator$$anon$11.next(Iterator.scala:409) at
> > > > scala.collection.Iterator$$anon$11.next(Iterator.scala:409) at
> > > >
> > > >
> > >
> >
> org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:193)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:62)
> > > > at
> > > >
> > >
> >
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
> > > > at
> > > >
> > >
> >
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)
> > > > at org.apache.spark.scheduler.Task.doRunTask(Task.scala:139) at
> > > > org.apache.spark.scheduler.Task.run(Task.scala:112) at
> > > >
> > > >
> > >
> >
> org.apache.spark.executor.Executor$TaskRunner$$anonfun$13.apply(Executor.scala:497)
> > > > at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1432)
> at
> > > > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:503)
> > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > at java.lang.Thread.run(Thread.java:748)
> > > > Command took 4.45 seconds -- by [email protected] at 3/10/2019,
> > > > 1:17:42
> > > > AM on Spark_Hudi
> > > > Shift+Enter to run shortcuts
> > > >
> > >
> >
>
