xushiyan commented on code in PR #8290:
URL: https://github.com/apache/hudi/pull/8290#discussion_r1154118452
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/CloudObjectsSelectorCommon.java:
##########
@@ -115,4 +139,41 @@ private static boolean checkIfFileExists(String
storageUrlSchemePrefix, String b
throw new HoodieIOException(errMsg, ioe);
}
}
+
+ public static Option<Dataset<Row>> loadAsDataset(SparkSession spark,
List<CloudObject> cloudObjects, TypedProperties props, String fileFormat) {
+ LOG.debug("Extracted distinct files " + cloudObjects.size()
+ + " and some samples " +
cloudObjects.stream().map(CloudObject::getPath).limit(10).collect(Collectors.toList()));
+
+ if (isNullOrEmpty(cloudObjects)) {
+ return Option.empty();
+ }
+ DataFrameReader reader = spark.read().format(fileFormat);
+ String datasourceOpts = props.getString(SPARK_DATASOURCE_OPTIONS, null);
+ if (StringUtils.isNullOrEmpty(datasourceOpts)) {
+ // fall back to legacy config for BWC. TODO consolidate in HUDI-5780
+ datasourceOpts =
props.getString(S3EventsHoodieIncrSource.Config.SPARK_DATASOURCE_OPTIONS, null);
+ }
+ if (StringUtils.nonEmpty(datasourceOpts)) {
+ final ObjectMapper mapper = new ObjectMapper();
+ Map<String, String> sparkOptionsMap = null;
+ try {
+ sparkOptionsMap = mapper.readValue(datasourceOpts, Map.class);
+ } catch (IOException e) {
+ throw new HoodieException(String.format("Failed to parse sparkOptions:
%s", datasourceOpts), e);
+ }
+ LOG.info(String.format("sparkOptions loaded: %s", sparkOptionsMap));
+ reader = reader.options(sparkOptionsMap);
+ }
+ List<String> paths = new ArrayList<>();
+ long totalSize = 0;
+ for (CloudObject o: cloudObjects) {
+ paths.add(o.getPath());
+ totalSize += o.getSize();
+ }
+ // inflate 10% for potential hoodie meta fields
+ totalSize *= 1.1;
+ long parquetMaxFileSize = props.getLong(PARQUET_MAX_FILE_SIZE.key(),
Long.parseLong(PARQUET_MAX_FILE_SIZE.defaultValue()));
+ int numPartitions = (int) Math.max(totalSize / parquetMaxFileSize, 1);
+ return Option.of(reader.load(paths.toArray(new
String[cloudObjects.size()])).coalesce(numPartitions));
Review Comment:
as discussed, coalesce won't make partition number go up. the main purpose
is to avoid small files / too many partitions case, so coalesce is meant for
it. besides, it avoid full shuffle like what repartition does, so it's
preferred.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
