KnightChess opened a new issue, #6679:
URL: https://github.com/apache/hudi/issues/6679
**Describe the problem you faced**
```java
@Override
public void write(HoodieRecord record, Option<IndexedRecord> avroRecord) {
...
fileWriter.writeAvro(record.getRecordKey(),
rewriteRecordWithMetadata((GenericRecord) avroRecord.get(),
path.getName()));
} else {
fileWriter.writeAvroWithMetadata(record.getKey(),
rewriteRecord((GenericRecord) avroRecord.get()));
}
...
} catch (Throwable t) {
// Not throwing exception from here, since we don't want to fail the
entire job
// for a single record
writeStatus.markFailure(record, t, recordMetadata);
LOG.error("Error writing record " + record, t);
}
}
```
```java
private def commitAndPerformPostOperations(spark: SparkSession,
...
): (Boolean,
common.util.Option[java.lang.String], common.util.Option[java.lang.String]) = {
if (writeResult.getWriteStatuses.rdd.filter(ws => ws.hasErrors).count()
== 0) {
log.info("Proceeding to commit the write.")
...
(commitSuccess && metaSyncSuccess, compactionInstant,
clusteringInstant)
} else {
log.error(s"${tableInstantInfo.operation} failed with errors")
if (log.isTraceEnabled) {
log.trace("Printing out the top 100 errors")
writeResult.getWriteStatuses.rdd.filter(ws => ws.hasErrors)
.take(100)
.foreach(ws => {
log.trace("Global error :", ws.getGlobalError)
if (ws.getErrors.size() > 0) {
ws.getErrors.foreach(kt =>
log.trace(s"Error for key: ${kt._1}", kt._2))
}
})
}
(false, common.util.Option.empty(), common.util.Option.empty())
}
}
```
I have a `SchemaCompatibilityException` in method `rewriteRecord`, but it
had be catch, only print log in driver: `ERROR HoodieSparkSqlWriter$: UPSERT
failed with errors`, and the spark job status is `SUCCEEDED`
some question need help:
- Is it for streaming scenes?
- in batch model, will use job status as trigger condition often. Please
help me how to resolve this scene in current version, thank you very much
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
