This is an automated email from the ASF dual-hosted git repository.
vbalaji pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-hudi.git
The following commit(s) were added to refs/heads/master by this push:
new 14dd649 [MINOR] Remove release notes and move confluent repository to
hoodie parent pom
14dd649 is described below
commit 14dd649d06009e99f0d687edd0763c27e3df0389
Author: Balaji Varadarajan <[email protected]>
AuthorDate: Mon Oct 21 13:04:22 2019 -0700
[MINOR] Remove release notes and move confluent repository to hoodie parent
pom
---
RELEASE_NOTES.md | 476 -------------------------------------------------
hudi-utilities/pom.xml | 7 -
pom.xml | 4 +
3 files changed, 4 insertions(+), 483 deletions(-)
diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md
deleted file mode 100644
index 0e2cb91..0000000
--- a/RELEASE_NOTES.md
+++ /dev/null
@@ -1,476 +0,0 @@
-<!--
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-
-Release 0.5.0-incubating
-------------------------------------
-
-### Highlights
- * First Apache Incubating Release
- * Package Renames from com.uber.hoodie to org.apache.hudi (See
https://cwiki.apache.org/confluence/display/HUDI/Migration+Guide+From+com.uber.hoodie+to+org.apache.hudi)
- * Complete Redo of Hudi Jar bundling
- * Bug fixes in query side integration, hive-sync, deltaStreamer, compaction,
rollbacks, restore
-
-### Full PR List
- * **Balaji Varadarajan** [HUDI-121] Fix licensing issues found during RC
voting by general incubator group
- * **Mehrotra** Shade and relocate Avro dependency in hadoop-mr-bundle
- * **Wenning Ding** [HUDI-301] fix path error when update a non-partition MOR
table
- * **Udit Mehrotra** [HUDI-298] Fix issue with incorrect column mapping
casusing bad data, during on-the-fly merge of Real Time tables (#956)
- * **Anurag870** [Docs] Update README.md (#955)
- * **leesf** [HUDI-292] Avoid consuming more entries from kafka than
specified sourceLimit. (#947)
- * **leesf** [HUDI-296] Explore use of spotless to auto fix formatting errors
(#945)
- * **Balaji Varadarajan** [MINOR] Add incubating to NOTICE and README.md
- * **vinoth chandar** [HOTFIX] Move to openjdk to get travis passing (#944)
- * **leesf** [HUDI-232] Implement sealing/unsealing for HoodieRecord class
(#938)
- * **Balaji Varadarajan** [HUDI-293] Remove KEYS file from github repository
- * **Balaji Varadarajan** [HUDI-121] Prepare for 0.5.0-incubating-rc5
- * **Balaji Varadarajan** [HUDI-121] Remove leftover notice file and replace
com.uber.hoodie with org.apache.hudi in log4j properties
- * **leesf** [HUDI-285] Implement HoodieStorageWriter based on actual file
type
- * **leesf** [HUDI-265] Failed to delete tmp dirs created in unit tests
- * **Balaji Varadarajan** [HUDI-287] Address comments during review of
release candidate. Remove LICENSE and NOTICE files in hoodie child modules.
- * **Balaji Varadarajan** [HUDI-121] Fix bugs in Release Scripts found during
RC creation
- * **Balaji Varadarajan** [HUDI-121] : Address comments during RC2 voting
- * **Bhavani Sudha Saktheeswaran** [HUDI-271] Create QuickstartUtils for
simplifying quickstart guide
- * **vinoyang** [HUDI-247] Unify the re-initialization of
HoodieTableMetaClient in test for hoodie-client module (#930)
- * **Balaji Varadarajan** [HUDI-279] Fix regression in Schema Evolution due
to PR-755
- * **Xing Pan** [HUDI-269] Limit sync frequency (#921)
- * **vinoth chandar** [MINOR] : Further updates to .mailmap for vinoth (#919)
- * **PanXing** [MINOR] support reading cfg file in another s3 bucket (#914)
- * **vinoyang** HUDI-267 Refactor bad method name
HoodieTestUtils#initTableType and
HoodieTableMetaClient#initializePathAsHoodieDataset (#916)
- * **Nishith Agarwal** HUDI-180 : Adding support for hive registration using
metastore along with JDBC
- * **Balaji Varadarajan** Fix mailmap to allow mapping of defunct emailIds
to active emailIds
- * **Balaji Varadarajan** Adding mailmap to allow mapping of defunct emailIds
to active emailIds
- * **Balaji Varadarajan** Updating release notes and preparing for
0.5.0-incubating-rc2 release
- * **Vinoth Chandar** [HUDI-254]: Bundle and shade databricks/avro with spark
bundle
- * **Balaji Varadarajan** [HUDI-257] Fix Bloom Index unit-test failures
- * **Balaji Varadarajan** [HUDI-252] Add Disclaimer and cleanup NOTICE and
LICENSE files in hudi. Identify packages which are under non-apache license in
LICENSE file
- * **Taher Koitwala** [HUDI-62] Index Lookup Timer added to HoodieWriteClient
- * **Balaji Varadarajan** [HUDI-249] Update Release-notes. Add sign-artifacts
to POM and release related scripts. Add missing license headers and update
NOTICE.txt files
- * **Vinoth Chandar** [HUDI-244] : Hive sync should escape partition field
name - now supports field names beginning with '_' for e.g
- * **Balaji Varadarajan** [HUDI-250] Ensure Hudi CLI wrapper works with non
snapshot jars too
- * **Nishith Agarwal** [HUDI-235] Fix scheduled compaction rollback in
restore command
- * **yanghua** [HUDI-217] Provide a unified resource management class to
standardize the resource allocation and release for hudi client test cases
- * **Bhavani Sudha Saktheeswaran** [HUDI-164] Fixes incorrect
averageBytesPerRecord
- * **Balaji Varadarajan** [HUDI-243] Rename HoodieInputFormat and
HoodieRealtimeInputFormat to HoodieParquetInputFormat and
HoodieParquetRealtimeInputFormat
- * **Vinoth Chandar** [HUDI-143] Excluding javax.* from utilities and spark
bundles
- * **vinoth chandar** [HUDI-159] Redesigning bundles for lighter-weight
integrations
- * **Mehrotra** Fix logging in HoodieSparkSqlWriter
- * **leesf** [hotfix] fix typo
- * **leesf** [HUDI-236] Failed to close stream
- * **Alex Filipchik** Support nested ordering fields
- * **leesf** [HUDI-230] Add missing Apache License in some files
- * **Balaji Varadarajan** [HUDI-227] : DeltaStreamer Improvements : Commit
empty input batch with progressing checkpoints and allow users to override
configs through properties. Original PR : PR-805 and PR-806 (#863)
- * **Balaji Varadarajan** HUDI-170 Updating hoodie record before inserting it
into ExternalSpillableMap (#866)
- * **leesf** [HUDI-229] Fix mvn notice:generate issue in windows
- * **leesf** [HUDI-225] Create Hudi Timeline Server Fat Jar
- * **Balaji Varadarajan** HUDI-124 : Exclude jdk.tools from hadoop-common and
update Notice files (#858)
- * **leesf** [hotfix] change hoodie-timeline-*.jar to hudi-timeline-*.jar
- * **leesf** [HUDI-222] Rename main class path to
org.apache.hudi.timeline.service.TimelineService in run_server.sh
- * **Alex Filipchik** Fixed unit test
- * **Alex Filipchik** Addressing comments
- * **Alex Filipchik** Ignore duplicate of a compaction file
- * **Alexander Filipchik** [HUDI-223] Adding a way to infer target schema
from the dataset after the transformation (#854)
- * **Vinoth Chandar** [HUDI-159] Precursor cleanup to reduce build warnings
- * **Balaji Varadarajan** HUDI-128 Preparing POM for release and snapshot
builds (#851)
- * **vinoth chandar** [HUDI-159]: Pom cleanup and removal of
com.twitter.parquet
- * **vinoth chandar** [HUDI-68] Pom cleanup & demo automation (#846)
- * **Bhavani Sudha Saktheeswaran** [HUDI-82] Adds Presto integration in
Docker demo (#847)
- * **leesf** HUDI-212: Specify Charset to UTF-8 for IOUtils.toString (#837)
- * **vinoyang** [HUDI-205] Let checkstyle ban Java and Guava Optional instead
of using Option provided by Hudi (#834)
- * **Balaji Varadarajan** HUDI-204 : Make MOR rollback idempotent and disable
using rolling stats for small file selection (#833)
- * **Nishith Agarwal** Adding GPG Keys
- * **Balaji Varadarajan** HUDI-123 Rename code packages/constants to
org.apache.hudi (#830)
- * **yanghua** [HUDI-153] Use com.uber.hoodie.common.util.Option instead of
Java and Guava Optional
- * **garyli1019** HUDI-171 delete tmp file in addShutDownHook
- * **Balaji Varadarajan** HUDI-149 - Remove platform dependencies and update
NOTICE plugin
- * **n3nash** - Adding HoodieCombineHiveInputFormat for COW tables (#811)
- * **n3nash** - Fix realtime queries by removing COLUMN_ID and COLUMN_NAME
cache in inputformat (#814)
- * **venkatr** Cache RDD to avoid recomputing data ingestion. Return result
RDD after updating index so that this step is not skipped by chained actions on
the same RDD
- * **Balaji Varadarajan** HUDI-197 Hive Sync and othe CLIs using bundle
picking sources jar instead of binary jar
- * **vinothchandar** HUDI-178 : Add keys for vinoth to KEYS file
- * **Anbu Cheeralan** Update Keys with [email protected]
- * **Luke Zhu** Fix typo in hoodie-presto-bundle (#818)
- * **Balaji Varadarajan** Generate Source Jars for bundle packages (#810)
- * **Vinoth Chandar** HUDI-92 : Making deltastreamer with
DistributedTestSource also run locally
- * **vinoyang** [HUDI-181] Fix the Bold markdown grammar issue of README file
(#808)
- * **eisig** HUDI-175 - add an option to manually override the
DeltaStreamer checkpoint (#798)
- * **Balaji Varadarajan** Add balaji gpg key to KEYS file
- * **Balaji Varadarajan** Allow HoodieWrapperFileSystem to wrap other proxy
file-system implementations with no getScheme implementation (#793)
- * **Balaji Varadarajan** HUDI-140 : GCS: Log File Reading not working due to
difference in seek() behavior for EOF
- * **eisig** add jssc.stop() (#797)
- * **n3nash** Fixing default value for avro 1.7 which assumes NULL value
instead of a jsonnode that is null (#792)
- * **Balaji Varadarajan** HUDI-168 Ensure getFileStatus calls for files
getting written is done after close() is called (#788)
- * **eisig** fix HoodieLogFileReader (#787)
- * **Balaji Varadarajan** HUDI-162 : File System view must be built with
correct timeline actions
- * **Balaji Varadarajan** HUDI-138 - Meta Files handling also need to support
consistency guard
- * **Yihua Guo** [HUDI-161] Remove --key-generator-class CLI arg in
HoodieDeltaStreamer and use key generator class specified in datasource
properties. (#781)
- * **Ho Tien Vu** Fixed TableNotFoundException when write with structured
streaming (#778)
- * **Thinking Chen** when column type is decimal, should add precision and
scale (#753)
- * **Balaji Varadarajan** Fixing bugs found during running hoodie demo (#760)
- * **Ho Tien Vu** Added preemptive check for 'spark.scheduler.mode'
- * **Jaimin Shah** adding support for complex keys (#728)
- * **Ron Barabash** Adding support for optional skipping single archiving
failures
- * **Balaji Varadarajan** Reword confusing message and reducing the severity
level
- * **Balaji Varadarajan** Add maprfs to storage schemes
- * **Balaji Varadarajan** Ensure TableMetaClient and FileSystem instances
have exclusive copy of Configuration
- * **Balaji Varadarajan** HUDI-70 : Making DeltaStreamer run in continuous
mode with concurrent compaction
- * **Balaji Varadarajan** Disable Notice Plugin
- * **Balaji Varadarajan** HUDI-148 Small File selection logic for MOR must
skip fileIds selected for pending compaction correctly
- * **vinoth chandar** Update README.md
- * **Nishith Agarwal** Reading baseCommitTime from the latest file slice as
opposed to the tagged record value
- * **Nishith Agarwal** - Ugrading to Hive 2.x - Eliminating in-memory
deltaRecordsMap - Use writerSchema to generate generic record needed by custom
payloads - changes to make tests work with hive 2.x
- * **Balaji Varadarajan** All Opened hoodie clients in tests needs to be
closed TestMergeOnReadTable must use embedded timeline server
- * **Balaji Varadarajan** TestMergeOnReadTable must use embedded timeline
server
- * **Balaji Varadarajan** Turn on embedded server for all client tests
- * **Balaji Varadarajan** Ensure log files are consistently ordered when
scanning
- * **Vinoth Chandar** Introduce HoodieReadHandle abstraction into index
- * **Balaji Varadarajan** Close Hoodie Clients which are opened to properly
shutdown embedded timeline service
- * **Balaji Varadarajan** HUDI-147 Compaction Inflight Rollback not deleting
Marker directory
- * **Balaji Varadarajan** HUDI-125 : Change License for all source files and
update RAT configurations
- * **Balaji Varadarajan** Changes related to Licensing work 1. Go through
dependencies list one round to ensure compliance. Generated current NOTICE list
in all submodules (other apache projects like flink does this). To be on
conservative side regarding licensing, NOTICE.txt lists all dependencies
including transitive. Pending Compliance questions reported in
https://issues.apache.org/jira/browse/LEGAL-461 2. Automate generating
NOTICE.txt files to allow future package compliance is [...]
- * **guanjianhui** exlude fasterxml and parquet from presto bundle
- * **guanjianhui** set codehaus.jackson modules to the same version 1.9.13
- * **Balaji Varadarajan** Auto generated Slack Channel Notifications setup
- * **Balaji Varadarajan** Replace Non-Compliant dnl.utils package with Apache
2.0 licensed alternative
- * **Balaji Varadarajan** LogFile comparator must handle log file names
without write token for backwards compatibility
- * **Thinking** fix spark-shell add jar problem
- * **Vinoth Chandar** Move depedency repos to https urls
- * **Vinoth Chandar** Create hoodie-utilities-bundle to host the shaded jar
- * **Vinoth Chandar** Turn off noisy test
- * **Vinoth Chandar** Add support for maven deploy plugin to make snapshot
releases
- * **guanjianhui** fix maven pom
- * **Balaji Varadarajan** HUDI-139 Compaction running twice due to duplicate
"map" transformation while finalizing compaction
-
-Release 0.4.7
-------------------------------------
-
-### Highlights
- * Major releases with fundamental changes to filesystem listing & write
failure handling
- * Introduced the first version of HoodieTimelineServer that runs embedded on
the driver
- * With all executors fetching filesystem listing via RPC to timeline server,
drastically reduced filesystem listing!
- * Failing concurrent write tasks are now handled differently to be robust
against spark stage retries
- * Bug fixes/clean up around indexing, compaction
-
-### Full PR List
- * **@bvaradar** - HUDI-135 - Skip Meta folder when looking for partitions #698
- * **@bvaradar** - HUDI-136 - Only inflight commit timeline
(.commit/.deltacommit) must be used when checking for sanity during compaction
scheduling #699
- * **@bvaradar** - HUDI-134 - Disable inline compaction for Hoodie Demo #696
- * **@v3nkatesh** - default implementation for HBase index qps allocator #685
- * **@bvaradar** - SparkUtil#initLauncher shoudn't raise when
spark-defaults.conf doesn't exist #670HUDI-131 Zero File Listing in Compactor
run #693
- * **@vinothchandar** - Fixed HUDI-116 : Handle duplicate record keys across
partitions #687
- * **@leilinen** - HUDI-105 : Fix up offsets not available on leader exception
#650
- * **@bvaradar** - Allow users to set hoodie configs figs for Compactor,
Cleaner and HDFSParquetImporter utility scripts #691
- * **@bvaradar** - Spark Stage retry handling #651
- * **@pseudomoto** - HUDI-113: Use Pair over # delimited string #672
- * **@bvaradar** - Support nested types for recordKey, partitionPath and
combineKey #684
- * **@vinothchandar** - Downgrading fasterxml jackson to 2.6.7 to be spark
compatible #686
- * **@bvaradar** - Timeline Service with Incremental View Syncing support #600
-
-Release 0.4.6
-------------------------------------
-
-### Highlights
- * Index performance! Interval trees + bucketized checking speed up index
lookup upto 10x!
- * Faster writing due to cached avro encoder/decoders, lighter memory usage,
lesser data shuffled.
- * Support for spark jobs using > 1 cores per executor
- * DeltaStreamer bug fixes (inline compaction, hive sync, error record
handling)
- * Empty Record payload to support deletes out-of-box easily
- * Fixes to hive/spark bundles around dependencies, versioning, shading
-
-### Full PR List
-
- * **@bvaradar** - Minor CLI documentation change in delta-streamer #679
- * **@n3nash** - converting map task memory from mb to bytes #678
- * **@bvaradar** - Fix various errors found by long running delta-streamer
tests #675
- * **@vinothchandar** - Bucketized Bloom Filter checking #671
- * **@pseudomuto** - SparkUtil#initLauncher shoudn't raise when
spark-defaults.conf doesn't exist #670
- * **@abhioncbr** - HUDI-101: added exclusion filters for signature files. #669
- * **@ovj** - migrating kryo's dependency from twitter chill to plain kryo
library #649
- * **@bvaradar** - Revert "HUDI-101: added mevn-shade plugin with filters."
#665
- * **@abhioncbr** - HUDI-101: added mevn-shade plugin with filters. #659
- * **@bvaradar** - Rollback inflights when using Spark [Streaming] write #660
- * **@vinothchandar** - Making DataSource/DeltaStreamer use defaults for
combining #634
- * **@vinothchandar** - Fixes HUDI-85 : Interval tree based pruning for Bloom
Index #653
- * **@takezoe** - Fix to enable hoodie.datasource.read.incr.filters #655
- * **@n3nash** - Removing OLD MAGIC header #648
- * **@bvaradar** - Revert "Read and apply schema for each log block from the
metadata header instead of the latest schema" #647
- * **@lyogev** - Add empty payload class to support deletes via apache spark
#635
- * **@bvaradar** - Move to apachehudi dockerhub repository & use openjdk
docker containers #644
- * **@bvaradar** - Fix Hive RT query failure in hoodie demo #645
- * **@ovj** - Revert - Replacing Apache commons-lang3 object serializer with
Kryo #642
- * **@n3nash** - Read and apply schema for each log block from the metadata
header instead of the latest schema #640
- * **@bhasudha** - FIXES HUDI-98: Fix multiple issues when using
build_local_docker_images for demo setup #636
- * **@n3nash** - Performing commit archiving in batches to avoid keeping a
huge chunk in memory #631
- * **@bvaradar** - Essential Hive packages missing in hoodie spark bundle #633
- * **@n3nash** - 1. Minor changes to fix compaction 2. Adding 2 compaction
policies 3. Adding a Hbase index property #629
- * **@milantracy** - [HUDI-66] FSUtils.getRelativePartitionPath does not
handle repeated f… #627
- * **@vinothchandar** - Fixing small file handling, inline compaction defaults
#599
- * **@vinothchandar** - Follow up HUDI-27 : Call super.close() in
HoodieWraperFileSystem::close() #621
- * **@vinothchandar** - Fix HUDI-27 : Support num_cores > 1 for writing
through spark #620
- * **@vinothchandar** - Fixes HUDI-38: Reduce memory overhead of WriteStatus
#616
- * **@vinothchandar** - Fixed HUDI-87 : Remove schemastr from BaseAvroPayload
#619
- * **@vinothchandar** - Fixes HUDI-9 : Check precondition minInstantsToKeep >
cleanerCommitsR… #617
- * **@n3nash** - Fixing source schema and writer schema distinction in
payloads #612
- * **@ambition119** - [HUDI-63] Removed unused BucketedIndex code #608
- * **@bvaradar** - run_hive_sync tool must be able to handle case where there
are multiple standalone jdbc jars in hive installation dir #609
- * **@milantracy** - add a script that shuts down demo cluster gracefully #606
- * **@n3nash** - Enable multi rollbacks for MOR table type #546
- * **@ovj** - Replacing Apache commons-lang3 object serializer with Kryo
serializer #583
- * **@kaka11chen** - Add compression codec configurations for
HoodieParquetWriter. #604
- * **@smarthi** - HUDI-75: Add KEYS #601
- * **@vinothchandar** - Removing docs folder from master branch #602
- * **@bvaradar** - Fix hive sync and deltastreamer issue in demo #593
- * **@bhasudha** - Fix quickstart documentation for querying via Presto #598
- * **@ovj** - Handling duplicate record update for single partition
(duplicates in single or different parquet files) #584
- * **@kaka11chen** - Fix avro doesn't have short and byte type. #595
- * **@bvaradar** - FIleSystem View to handle same fileIds across partitions
correctly #572
- * **@vinothchandar** - Upgrade various jar, gem versions for maintenance #575
-
-Release 0.4.5
-------------------------------------
-
-### Highlights
- * Dockerized demo with support for different Hive versions
- * Smoother handling of append log on cloud stores
- * Introducing a global bloom index, that enforces unique constraint across
partitions
- * CLI commands to analyze workloads, manage compactions
- * Migration guide for folks wanting to move datasets to Hudi
- * Added Spark Structured Streaming support, with a Hudi sink
- * In-built support for filtering duplicates in DeltaStreamer
- * Support for plugging in custom transformation in DeltaStreamer
- * Better support for non-partitioned Hive tables
- * Support hard deletes for Merge on Read storage
- * New slack url & site urls
- * Added presto bundle for easier integration
- * Tons of bug fixes, reliability improvements
-
-
-### Full PR List
-
- * **@bhasudha** - Create hoodie-presto bundle jar. fixes #567 #571
- * **@bhasudha** - Close FSDataInputStream for meta file open in
HoodiePartitionMetadata . Fixes issue #573 #574
- * **@yaoqinn** - handle no such element exception in HoodieSparkSqlWriter
#576
- * **@vinothchandar** - Update site url in README
- * **@yaooqinn** - typo: bundle jar with unrecognized variables #570
- * **@bvaradar** - Table rollback for inflight compactions MUST not delete
instant files at any time to avoid race conditions #565
- * **@bvaradar** - Fix Hoodie Record Reader to work with non-partitioned
dataset ( ISSUE-561) #569
- * **@bvaradar** - Hoodie Delta Streamer Features : Transformation and Hoodie
Incremental Source with Hive integration #485
- * **@vinothchandar** - Updating new slack signup link #566
- * **@yaooqinn** - Using immutable map instead of mutables to generate
parameters #559
- * **@n3nash** - Fixing behavior of buffering in Create/Merge handles for
invalid/wrong schema records #558
- * **@n3nash** - cleaner should now use commit timeline and not include
deltacommits #539
- * **@n3nash** - Adding compaction to HoodieClient example #551
- * **@n3nash** - Filtering partition paths before performing a list status on
all partitions #541
- * **@n3nash** - Passing a path filter to avoid including folders under
.hoodie directory as partition paths #548
- * **@n3nash** - Enabling hard deletes for MergeOnRead table type #538
- * **@msridhar** - Add .m2 directory to Travis cache #534
- * **@artem0** - General enhancements #520
- * **@bvaradar** - Ensure Hoodie works for non-partitioned Hive table #515
- * **@xubo245** - fix some spell errorin Hudi #530
- * **@leletan** - feat(SparkDataSource): add structured streaming sink #486
- * **@n3nash** - Serializing the complete payload object instead of
serializing just the GenericRecord in HoodieRecordConverter #495
- * **@n3nash** - Returning empty Statues for an empty spark partition caused
due to incorrect bin packing #510
- * **@bvaradar** - Avoid WriteStatus collect() call when committing batch to
prevent Driver side OOM errors #512
- * **@vinothchandar** - Explicitly handle lack of append() support during
LogWriting #511
- * **@n3nash** - Fixing number of insert buckets to be generated by rounding
off to the closest greater integer #500
- * **@vinothchandar** - Enabling auto tuning of insert splits by default #496
- * **@bvaradar** - Useful Hudi CLI commands to debug/analyze production
workloads #477
- * **@bvaradar** - Compaction validate, unschedule and repair #481
- * **@shangxinli** - Fix addMetadataFields() to carry over 'props' #484
- * **@n3nash** - Adding documentation for migration guide and COW vs MOR
tradeoffs #470
- * **@leletan** - Add additional feature to drop later arriving dups #468
- * **@bvaradar** - Fix regression bug which broke HoodieInputFormat handling
of non-hoodie datasets #482
- * **@vinothchandar** - Add --filter-dupes to DeltaStreamer #478
- * **@bvaradar** - A quickstart demo to showcase Hudi functionalities using
docker along with support for integration-tests #455
- * **@bvaradar** - Ensure Hoodie metadata folder and files are filtered out
when constructing Parquet Data Source #473
- * **@leletan** - Adds HoodieGlobalBloomIndex #438
-
-
-Release 0.4.4
-------------------------------------
-
-### Highlights
- * Dependencies are now decoupled from CDH and based on apache versions!
- * Support for Hive 2 is here!! Use -Dhive11 to build for older hive versions
- * Deltastreamer tool reworked to make configs simpler, hardended tests, added
Confluent Kafka support
- * Provide strong consistency for S3 datasets
- * Removed dependency on commons lang3, to ease use with different
hadoop/spark versions
- * Better CLI support and docs for managing async compactions
- * New CLI commands to manage datasets
-
-### Full PR List
-
- * **@vinothchandar** - Perform consistency checks during write finalize #464
- * **@bvaradar** - Travis CI tests needs to be run in quieter mode (WARN log
level) to avoid max log-size errors #465
- * **@lys0716** - Fix the name of avro schema file in Test #467
- * **@bvaradar** - Hive Sync handling must work for datasets with
multi-partition keys #460
- * **@bvaradar** - Explicitly release resources in LogFileReader and
TestHoodieClientBase. Fixes Memory allocation errors #463
- * **@bvaradar** - [Release Blocking] Ensure packaging modules create
sources/javadoc jars #461
- * **@vinothchandar** - Fix bug with incrementally pulling older data #458
- * **@saravsars** - Updated jcommander version to fix NPE in
HoodieDeltaStreamer tool #443
- * **@n3nash** - Removing dependency on apache-commons lang 3, adding
necessary classes as needed #444
- * **@n3nash** - Small file size handling for inserts into log files. #413
- * **@vinothchandar** - Update Gemfile.lock with higher ffi version
- * **@bvaradar** - Simplify and fix CLI to schedule and run compactions #447
- * **@n3nash** - Fix a failing test case intermittenly in TestMergeOnRead due
to incorrect prev commit time #448
- * **@bvaradar**- CLI to create and desc hoodie table #446
- * **@vinothchandar**- Reworking the deltastreamer tool #449
- * **@bvaradar**- Docs for describing async compaction and how to operate it
#445
- * **@n3nash**- Adding check for rolling stats not present in existing
timeline to handle backwards compatibility #451
- * **@bvaradar** **@vinothchandar** - Moving all dependencies off cdh and to
apache #420
- * **@bvaradar**- Reduce minimum delta-commits required for compaction #452
- * **@bvaradar**- Use spark Master from environment if set #454
-
-
-Release 0.4.3
-------------------------------------
-
-### Highlights
- * Ability to run compactions asynchrously & in-parallel to ingestion/write
added!!!
- * Day based compaction does not respect IO budgets i.e agnostic of them
- * Adds ability to throttle writes to HBase via the HBaseIndex
- * (Merge on read) Inserts are sent to log files, if they are indexable.
-
-### Full PR List
-
- * **@n3nash** - Adding ability for inserts to be written to log files #400
- * **@n3nash** - Fixing bug introducted in rollback for MOR table type with
inserts into log files #417
- * **@n3nash** - Changing Day based compaction strategy to be IO agnostic #398
- * **@ovj** - Changing access level to protected so that subclasses can access
it #421
- * **@n3nash** - Fixing missing hoodie record location in HoodieRecord when
record is read from disk after being spilled #419
- * **@bvaradar** - Async compaction - Single Consolidated PR #404
- * **@bvaradar** - BUGFIX - Use Guava Optional (which is Serializable) in
CompactionOperation to avoid NoSerializableException #435
- * **@n3nash** - Adding another metric to HoodieWriteStat #434
- * **@n3nash** - Fixing Null pointer exception in finally block #440
- * **@kaushikd49** - Throttling to limit QPS from HbaseIndex #427
-
-Release 0.4.2
-------------------------------------
-
-### Highlights
- * Parallelize Parquet writing & input record read resulting in upto 2x
performance improvement
- * Better out-of-box configs to support upto 500GB upserts, improved
ROPathFilter performance
- * Added a union mode for RT View, that supports near-real time event
ingestion without update semantics
- * Added a tuning guide with suggestions for oft-encountered problems
- * New configs for configs for compression ratio, index storage levels
-
-### Full PR List
-
- * **@jianxu** - Use hadoopConf in HoodieTableMetaClient and related tests #343
- * **@jianxu** - Add more options in HoodieWriteConfig #341
- * **@n3nash** - Adding a tool to read/inspect a HoodieLogFile #328
- * **@ovj** - Parallelizing parquet write and spark's external read operation.
#294
- * **@n3nash** - Fixing memory leak due to HoodieLogFileReader holding on to a
logblock #346
- * **@kaushikd49** - DeduplicateRecords based on recordKey if global index is
used #345
- * **@jianxu** - Checking storage level before persisting preppedRecords #358
- * **@n3nash** - Adding config for parquet compression ratio #366
- * **@xjodoin** - Replace deprecated jackson version #367
- * **@n3nash** - Making ExternalSpillableMap generic for any datatype #350
- * **@bvaradar** - CodeStyle formatting to conform to basic Checkstyle rules.
#360
- * **@vinothchandar** - Update release notes for 0.4.1 (post) #371
- * **@bvaradar** - Issue-329 : Refactoring
TestHoodieClientOnCopyOnWriteStorage and adding test-cases #372
- * **@n3nash** - Parallelized read-write operations in Hoodie Merge phase #370
- * **@n3nash** - Using BufferedFsInputStream to wrap FSInputStream for
FSDataInputStream #373
- * **@suniluber** - Fix for updating duplicate records in same/different
files in same pa… #380
- * **@bvaradar** - Fixit : Add Support for ordering and limiting results in
CLI show commands #383
- * **@n3nash** - Adding metrics for MOR and COW #365
- * **@n3nash** - Adding a fix/workaround when fs.append() unable to return a
valid outputstream #388
- * **@n3nash** - Minor fixes for MergeOnRead MVP release readiness #387
- * **@bvaradar** - Issue-257: Support union mode in HoodieRealtimeRecordReader
for pure insert workloads #379
- * **@n3nash** - Enabling global index for MOR #389
- * **@suniluber** - Added a new filter function to filter by record keys when
reading parquet file #395
- * **@vinothchandar** - Improving out of box experience for data source #295
- * **@xjodoin** - Fix wrong use of TemporaryFolder junit rule #411
-
-Release 0.4.1
-------------------------------------
-
-### Highlights
-
- * Good enhancements for merge-on-read write path : spillable map for merging,
evolvable log format, rollback support
- * Cloud file systems should now work out-of-box for copy-on-write tables,
with configs picked up from SparkContext
- * Compaction action is no more, multiple delta commits now lead to a commit
upon compaction
- * API level changes include : compaction api, new prepped APIs for higher
plugability for advanced clients
-
-
-### Full PR List
-
- * **@n3nash** - Separated rollback as a table operation, implement rollback
for MOR #247
- * **@n3nash** - Implementing custom payload/merge hooks abstractions for
application #275
- * **@vinothchandar** - Reformat project & tighten code style guidelines #280
- * **@n3nash** - Separating out compaction() API #282
- * **@n3nash** - Enable hive sync even if there is no compaction commit #286
- * **@n3nash** - Partition compaction strategy #281
- * **@n3nash** - Removing compaction action type and associated compaction
timeline operations, replace with commit action type #288
- * **@vinothchandar** - Multi/Cloud FS Support for Copy-On-Write tables #293
- * **@vinothchandar** - Update Gemfile.lock #298
- * **@n3nash** - Reducing memory footprint required in HoodieAvroDataBlock and
HoodieAppendHandle #290
- * **@jianxu** - Add FinalizeWrite in HoodieCreateHandle for COW tables #285
- * **@n3nash** - Adding global indexing to HbaseIndex implementation #318
- * **@n3nash** - Small File Size correction handling for MOR table type #299
- * **@jianxu** - Use FastDateFormat for thread safety #320
- * **@vinothchandar** - Fix formatting in HoodieWriteClient #322
- * **@n3nash** - Write smaller sized multiple blocks to log file instead of a
large one #317
- * **@n3nash** - Added support for Disk Spillable Compaction to prevent OOM
issues #289
- * **@jianxu** - Add new APIs in HoodieReadClient and HoodieWriteClient #327
- * **@jianxu** - Handle inflight clean instants during Hoodie instants
archiving #332
- * **@n3nash** - Introducing HoodieLogFormat V2 with versioning support #331
- * **@n3nash** - Re-factoring Compaction as first level API in WriteClient
similar to upsert/insert #330
-
-
-Release 0.4.0
-------------------------------------
-
-### Highlights
-
- * [Spark datasource
API](https://uber.github.io/hoodie/quickstart.html#datasource-api) now
supported for Copy-On-Write datasets, across all views
- * BloomIndex can now [prune based on key
ranges](https://uber.github.io/hoodie/configurations.html#bloomIndexPruneByRanges)
& cut down index tagging time dramatically, for time-prefixed/ordered record
keys
- * Hive sync tool registers RO and RT tables now.
- * Client application can now specify the partitioner to be used by
bulkInsert(), useful for low-level control over initial record placement
- * Framework for metadata tracking inside IO handles, to implement Spark
accumulator-style counters, that are consistent with the timeline
- * Bug fixes around cleaning, savepoints & upsert's partitioner.
-
-### Full PR List
-
- * **@gekath** - Writes relative paths to .commit files #184
- * **@kaushikd49** - Correct clean bug that causes exception when
partitionPaths are empty #202
- * **@vinothchandar** - Refactor HoodieTableFileSystemView using FileGroups &
FileSlices #201
- * **@prazanna** - Savepoint should not create a hole in the commit timeline
#207
- * **@jianxu** - Fix TimestampBasedKeyGenerator in HoodieDeltaStreamer when
DATE_STRING is used #211
- * **@n3nash** - Sync Tool registers 2 tables, RO and RT Tables #210
- * **@n3nash** - Using FsUtils instead of Files API to extract file extension
#213
- * **@vinothchandar** - Edits to documentation #219
- * **@n3nash** - Enabled deletes in merge_on_read #218
- * **@n3nash** - Use HoodieLogFormat for the commit archived log #205
- * **@n3nash** - fix for cleaning log files in master branch (mor) #228
- * **@vinothchandar** - Adding range based pruning to bloom index #232
- * **@n3nash** - Use CompletedFileSystemView instead of CompactedView
considering deltacommits too #229
- * **@n3nash** - suppressing logs (under 4MB) for jenkins #240
- * **@jianxu** - Add nested fields support for MOR tables #234
- * **@n3nash** - adding new config to separate shuffle and write parallelism
#230
- * **@n3nash** - adding ability to read archived files written in log format
#252
- * **@ovj** - Removing randomization from UpsertPartitioner #253
- * **@ovj** - Replacing SortBy with custom partitioner #245
- * **@esmioley** - Update deprecated hash function #259
- * **@vinothchandar** - Adding canIndexLogFiles(), isImplicitWithStorage(),
isGlobal() to HoodieIndex #268
- * **@kaushikd49** - Hoodie Event callbacks #251
- * **@vinothchandar** - Spark Data Source (finally) #266
-
-
-Previous Releases
-------------
-* Refer to [github](https://github.com/uber/hoodie/releases)
diff --git a/hudi-utilities/pom.xml b/hudi-utilities/pom.xml
index 612aa2f..ba59e22 100644
--- a/hudi-utilities/pom.xml
+++ b/hudi-utilities/pom.xml
@@ -68,13 +68,6 @@
</resources>
</build>
- <repositories>
- <repository>
- <id>confluent</id>
- <url>https://packages.confluent.io/maven/</url>
- </repository>
- </repositories>
-
<dependencies>
<!-- Jetty -->
diff --git a/pom.xml b/pom.xml
index 6e97333..91434df 100644
--- a/pom.xml
+++ b/pom.xml
@@ -840,6 +840,10 @@
<enabled>false</enabled>
</snapshots>
</repository>
+ <repository>
+ <id>confluent</id>
+ <url>https://packages.confluent.io/maven/</url>
+ </repository>
</repositories>
<profiles>
