Prashant Wason created HUDI-1303:
------------------------------------
Summary: Some improvements for the HUDI Test Suite
Key: HUDI-1303
URL: https://issues.apache.org/jira/browse/HUDI-1303
Project: Apache Hudi
Issue Type: Improvement
Reporter: Prashant Wason
Assignee: Prashant Wason
# Use the DAG Node's label from the yaml as its name instead of UUID names
which are not descriptive when debugging issues from logs.
# Fix CleanNode constructor which is not correctly implemented
# When generating upsets, allows more granualar control over the number of
inserts and upserts - zero or more inserts and upserts can be specified instead
of always requiring both inserts and upserts.
# Fixed generation of records of specific size
** The current code was using a class variable "shouldAddMore" which was reset
to false after the first record generation causing subsequent records to be of
minimum size.
** In this change, we pre-calculate the extra size of the complex fields. When
generating records, for complex fields we read the field size from this map.
# Refresh the timeline of the DeltaSync service before calling readFromSource.
This ensures that only the newest generated data is read and data generated in
the older Dag Nodes is ignored (as their AVRO files will have an older
timestamp).
# Making --workload-generator-classname an optional parameter as most probably
the default will be used
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
