commits  

Messages by Thread

• [gobblin] branch master updated: Update CHANGELOG to reflect changes in 0.17.0 abti
• [gobblin] branch master updated: Reserving 0.18.0 version for next release abti
• [gobblin] branch 0.17.0 created (now a00b57cda) abti
• [gobblin] branch master updated: [GOBBLIN-1836] Ensuring Task Reliability: Handling Job Cancellation and Graceful Exits for Error-Free Completion (#3699) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1805] Check watermark for the most recent hour for quiet topics (#3698) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1825]Hive retention job should fail if deleting underlying files fail (#3687) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1823] Improving Container Calculation and Allocation Methodology (#3692) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1830] Improving Container Transition Tracking in Streaming Data Ingestion (#3693) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1833]Emit Completeness watermark information in snapshotCommitEvent (#3696) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1832] Emit warning instead of failing job for retention of Hive Table Views (#3695) wlo
• [gobblin] branch master updated: [GOBBLIN-1831] Use flowexecutionid in kafka monitor and jobnames (#3694) wlo
• [gobblin] branch master updated: [GOBBLIN-1824]Improving the Efficiency of Work Planning in Manifest-Based DistCp Jobs (#3686) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1829] Fixes bug where the wrong workunit event was being tracked for keepin… (#3691) wlo
• [gobblin] branch master updated: [GOBBLIN-1828] Implement Timeout for Creating Writer Functionality (#3690) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1827] Add check that if nested field is optional and has a non-null default… (#3689) wlo
• [gobblin] branch master updated: [GOBBLIN-1826] Change isAssignableFrom() to isSuperTypeOf() per Guava 20 javadocs to… (#3688) wlo
• [gobblin] branch master updated: [GOBBLIN-1822]Logging Abnormal Helix Task States (#3685) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1819] Log helix workflow information and timeout information during submission wait / polling (#3681) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1821] Let flow execution ID propagate to the Job ID if it exists (#3684) wlo
• [gobblin] branch master updated: [GOBBLIN-1810] Support general iceberg catalog (support configurable behavior for metadata retention policy) (#3680) zihanli58
• [gobblin] branch master updated: Add null default value to observability events that are additionally added (#3682) wlo
• [gobblin] branch master updated: [GOBBLIN-1816] Add job properties and GaaS instance ID to observability event (#3676) wlo
• [gobblin] branch master updated: [GOBBLIN-1785] add MR_JARS_BASE_DIR and logic to delete old mr jar dirs (#3642) wlo
• [gobblin] branch master updated: initiliaze yarn clients in yarn app launcher so that a child class can override the yarn client creation logic (#3679) wlo
• [gobblin] branch master updated (ebdb03fa4 -> 27dea4a6d) wlo
• [gobblin] branch master updated: [GOBBLIN-1817] change some deprecated code and fix minor codestyle (#3678) wlo
• [gobblin] branch master updated: [GOBBLIN-1812] Mockito should only be test compile (#3674) wlo
• [gobblin] branch master updated: [GOBBLIN-1813] Helix workflows submission timeouts are configurable (#3677) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1810] Support general iceberg catalog in icebergMetadataWriter (#3672) zihanli58
• [gobblin] branch master updated (234067109 -> 80c45fb88) wlo
• [gobblin] branch master updated: [GOBBLIN-1808] Bump Guava version from 15.0 to 20.0 (#3669) wlo
• [gobblin] branch master updated (7389a193b -> 2002c99a7) wlo
• [gobblin] branch master updated: [GOBBLIN-1814] Add `MRJobLauncher` configurability for any failing mapper to be fatal to the MR job (#3675) wlo
• [gobblin] branch master updated: Add new lookback version finder for use with iceberg retention (#3670) wlo
• [gobblin] branch master updated: [GOBBLIN-1807] Replaces conjars.org with conjars.wensel.net (#3668) wlo
• [gobblin] branch master updated: [GOBBLIN-1802]Register iceberg table metadata update with destination side catalog (#3663) wlo
• [gobblin] branch master updated: Add matching of non-transient exceptions that will avoid failing the container in GMIP (#3662) wlo
• [gobblin] branch master updated: [GOBBLIN-1804] Merge similar logic between `FlowConfig{,V2}ResourceLocalHandler.update` into single base class impl. (#3665) wlo
• [gobblin] branch master updated: [GOBBLIN-1804] Reject flow config updates that would fail compilation by returning service error (#3664) zihanli58
• [gobblin] branch master updated: Add completion watermark to State (#3659) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1795] Make Manifest based copy to support facl (#3654) zihanli58
• [gobblin] branch master updated: fix add spec and actual number flows scheduled count (#3660) zihanli58
• [gobblin] branch master updated (8e8e3df03 -> 8b1ca21b4) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1779] Ability to filter datasets that contain non optional unions (#3648) wlo
• [gobblin] branch master updated (ab62e7283 -> d2d0c6389) wlo
• [gobblin] branch master updated: [GOBBLIN-1796] Log startup command when container fails to startup (#3655) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1794] Add defaults to newly added fields in observability events (#3653) wlo
• [gobblin] branch master updated: [GOBBLIN-1793] Add metrics to measure and isolate bottleneck for init… (#3652) wlo
• [gobblin] branch master updated: [GOBBLIN-1786] Support Other Catalog Types for Iceberg Distcp (#3643) wlo
• [gobblin] branch master updated: [GOBBLIN-1792] Upgrade Mockito to 4.* (#3651) wlo
• [gobblin] branch master updated: [GOBBLIN-1790] Add and change appropriate job status fields for observability events (#3649) wlo
• [gobblin] branch master updated: [GOBBLIN-1791] Prevent the adding of flowspec compilation errors to the scheduler's … (#3650) wlo
• [gobblin] branch master updated: [GOBBLIN-1783] Initialize scheduler with batch gets instead of individual get per flow (#3640) wlo
• [gobblin] branch master updated: [GOBBLIN-1789] Create Generic Iceberg Data Node to Support Different Types of Catalogs (#3646) wlo
• [gobblin] branch master updated: [GOBBLIN-1787] Ability to delete multiple watermarks in a state store (#3645) wlo
• [gobblin] branch master updated: Make GobblinMCEWriter constructor public (#3647) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1784] Only clean dags from the dag manager if a flow event is received (#3641) wlo
• [gobblin] branch master updated: [GOBBLIN-1749] Add dependency for handling xz-compressed Avro file (#3609) wlo
• [gobblin] branch master updated: [GOBBLIN-1611] Fix a wrong value for writer.codec.type in the document (#3470) wlo
• [gobblin] branch master updated: Log current stacktrace on every `MysqlStateStore.newDataSource` invocation, to help catch possible resource leak (#3644) wlo
• [gobblin] branch master updated: [GOBBLIN-1781] Helix offline instance purging is not thread safe in the yarn service (#3638) wlo
• [gobblin] branch master updated (9bc9df52a -> cc8076bfb) wlo
• [gobblin] branch master updated: [GOBBLIN-1774] Util for detecting non optional uniontypes Hive tables (#3632) wlo
• [gobblin] branch master updated: [GOBBLIN-1773] Fix bugs in quota manager (#3636) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1782] Fix Merge State for Flow Pending Resume statuses (#3639) wlo
• [gobblin] branch master updated (69f3f9c33 -> fd8f6e332) wlo
• [gobblin] branch master updated: [GOBBLIN-1780] Refactor/rename YarnServiceIT to YarnServiceTest (#3637) wlo
• [gobblin] branch master updated: [GOBBLIN-1778] Add house keeping thread in DagManager to periodically sync in memory state with mysql table (#3635) zihanli58
• [gobblin] branch master updated: Register gauge metrics for change monitors (#3634) wlo
• [gobblin] branch master updated: [GOBBLIN-1771] Clean up logs for dataset commit and file cleanup (#3631) wlo
• [gobblin] branch master updated: [GOBBLIN-1770] Allow null values for fields in GaaSObservabilityEvent.Issue fields which are optional wlo
• [gobblin] branch master updated: [GOBBLIN-1769] Change a noisy log that indicates that the queue capacity is almost full to debug (#3629) wlo
• [gobblin] branch master updated: [GOBBLIN-1766] Define metric to measure lag from producing to consume… (#3625) zihanli58
• [gobblin] branch master updated (3dd0657e9 -> efee9f0ae) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1768] Fix constructor in KafkaJobStatusMonitorFactory so that it can be injected (#3628) wlo
• [gobblin] branch master updated: Specifically name each Hikari connection pool created, for traceability (#3627) wlo
• [gobblin] branch master updated: [GOBBLIN-1759] Add error reporting when attempting to resolve flow configs (#3614) wlo
• [gobblin] branch master updated (8702d9691 -> a2fcb8646) wlo
• [gobblin] branch master updated: [GOBBLIN-1764] Emit observability event (#3623) wlo
• [gobblin] branch master updated (85cd1f987 -> b0e0c108d) wlo
• [gobblin] branch master updated: upgrade hadoop version (#3621) wlo
• [gobblin] branch master updated: [GOBBLIN-1744] Improve handling of null value edge cases when querying Helix (#3603) zihanli58
• [gobblin] branch master updated: Handle d2 markup case when not only announcing leader (#3622) wlo
• [gobblin] branch master updated: Update Gobblin OSS Slack channel link to a never-expire link (#3620) wlo
• [gobblin] branch master updated: [GOBBLIN-1750] Add schemas for observability events in GaaS (#3610) wlo
• [gobblin] branch master updated: [GOBBLIN-1757]Refactor manifest, add reader/writer and iterator for efficient reading (#3618) zihanli58
• [gobblin] branch master updated: Disable flaky HiveMaterializerTest on CI/CD (#3619) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1754] Fixes for mysql store change monitors (#3615) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1756] Fix the issue that we skipping flows for multihop jobs (#3617) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1752] Fix race condition where FSTemplateCatalog would update at the same t… (#3612) wlo
• [gobblin] branch master updated: [GOBBLIN-1753] Migrate DB connection pool from o.a.commons.dbcp/dbcp2 to HikariCP (#3613) wlo
• [gobblin] branch master updated: [GOBBLIN-1748] Add logs to debug multi-hop flows creation, progression, and cleanup (#3608) wlo
• [gobblin] branch master updated: [GOBBLIN-1741] Create manifest based dataset finder (#3598) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1747] add job.name and job.id to kafka and compaction workunits (#3607) wlo
• [gobblin] branch master updated (94c40dc95 -> 12ba4035b) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1743] Ensure GobblinTaskRunner works without Yarn use (#3602) wlo
• [gobblin] branch master updated (deec15e46 -> e5f4047b7) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1739]Define Datanodes and Dataset Descriptor for Iceberg (#3596) wlo
• [gobblin] branch master updated: do not close DestinationDatasetHandlerService prematurely (#3601) wlo
• [gobblin] branch master updated: [GOBBLIN-1720]Add ancestors owner permissions preservations for iceberg distcp (#3577) wlo
• [gobblin] branch master updated: [HOTFIX] Fix checkstyleMain (#3600) wlo
• [gobblin] branch revert-3593-GOBBLIN-1736 created (now 4f9932b2c) zihanli58
  • [gobblin] 01/01: Revert "[GOBBLIN-1736] Add metrics for change stream monitor and mysql quota manager (#3593)" zihanli58
• [gobblin] branch master updated: [GOBBLIN-1736] Add metrics for change stream monitor and mysql quota manager (#3593) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1737] Fix bug when using mysql user quota manager (#3595) zihanli58
• [gobblin] branch master updated: Correct a log line and GTE with currect number of total task count (#3591) wlo
• [gobblin] branch master updated: move dataset handler code before cleaning up staging data (#3594) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1730] Include flow execution id when try to cancel/submit job using SimpleKafkaSpecProducer (#3588) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1734] make DestinationDatasetHandler work on streaming sources (#3592) wlo
• [gobblin] branch master updated: give option to cancel helix workflow through Delete API (#3580) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1728] Fix YarnService incorrect container allocation behavior (#3586) zihanli58
• [gobblin] branch master updated: Support multiple node types in shared flowgraph, fix logs (#3590) wlo
• [gobblin] branch master updated: Search for dummy file in writer directory (#3589) wlo
• [gobblin] branch master updated: Use root cause for checking if exception is transient (#3585) wlo
• [gobblin] branch master updated: [GOBBLIN-1724] Support a shared flowgraph layout in GaaS (#3583) wlo
• [gobblin] branch master updated: [GOBBLIN-1731] Enable HiveMetadataWriter to override table schema lit… (#3587) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1726] Avro 1.9 upgrade of Gobblin OSS (#3581) wlo
• [gobblin] branch master updated: [GOBBLIN-1725] Fix bugs in gaas warm standby mode (#3582) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1718] Define DagActionStoreMonitor to listen for kill/resume… (#3572) zihanli58
• [gobblin] branch master updated: Add log line for committing/retrieving watermarks in streaming (#3578) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1707] Enhance `IcebergDataset` to detect when files already at dest then proceed with only delta (#3575) zihanli58
• [gobblin] branch master updated: Ignore AlreadyExistsException in hive writer (#3579) wlo
• [gobblin] branch master updated: Fail GMIP container for known transient exceptions to avoid data loss (#3576) wlo
• [gobblin] branch master updated: GOBBLIN-1715: Support vectorized row batch pooling (#3574) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1696] Implement file based flowgraph that detects changes to the underlying… (#3548) wlo
• [gobblin] branch master updated: GOBBLIN-1719 Replace moveToTrash with moveToAppropriateTrash for hadoop trash (#3573) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1703] avoid double quota increase for adhoc flows (#3550) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1716] refactor HighLevelConsumer to make consumer initializa… (#3570) zihanli58
• [gobblin] branch master updated: Correct semantics of `IcebergDatasetTest` and streamline both impl and test code (#3571) zihanli58
• [gobblin] branch master updated (e385ea46b -> ae2ea1965) wlo
• [gobblin] branch master updated: [GOBBLIN-1707] Update `IcebergDataset` to incorporate all snapshots, not only the current one (#3569) zihanli58
• [gobblin] branch master updated (31f7ec5b7 -> 95c217da8) wlo
• [gobblin] branch master updated (bbd18f33c -> 31f7ec5b7) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1706] Add DagActionStore to store the action to kill/resume one flow execution (#3558) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1704] Purge offline helix instances during startup (#3561) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1708] Improve TimeAwareRecursiveCopyableDataset to lookback only into datefolders that match range (#3563) wlo
• [gobblin] branch master updated (18ec55bcd -> 80b9866b0) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1709] Create Iceberg Datasets Finder, Iceberg Dataset and FileSet to generate Copy Entities to support Distcp for Iceberg (#3560) wlo
• [gobblin] branch master updated: [GOBBLIN-1697]Have a separate resource handler to rely on CDC stream to do message forwarding (#3549) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1711] Replace Jcenter with maven central (#3566) wlo
• [gobblin] branch master updated: [GOBBLIN-1710] Codecov should be optional in CI and not fail Github Actions (#3562) wlo
• [gobblin] branch master updated: Define basics for collecting Iceberg metadata for the current snapshot (#3559) wlo
• [gobblin] branch master updated: [GOBBLIN-1698] Fast fail during work unit generation based on config. (#3542) wlo
• [gobblin] branch master updated: [GOBBLIN-1673][GOBBLIN-1683] Skeleton code for handling messages between task runner / application master for Dynamic work unit allocation (#3539) wlo
• [gobblin] branch master updated: [GOBBLIN-1699] Log progress of reducer task for visibility with slow compaction jobs #3552 wlo
• [gobblin] branch master updated: fix helix job wait completion bug when job goes to STOPPING state (#3556) zihanli58
• [gobblin] branch master updated (f36bd1bc7 -> 6f367839c) wlo
• [gobblin] branch master updated (0c234e46f -> f36bd1bc7) wlo
• [gobblin] branch master updated: [GOBBLIN-1700] Remove unused coveralls-gradle-plugin dependency wlo
• [gobblin] branch master updated: add MysqlUserQuotaManager (#3545) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1689] Decouple compiler from scheduler in warm standby mode (#3544) zihanli58
• [gobblin] branch master updated: Add GMCE topic explicitly to hive commit event (#3547) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1678] Refactor git flowgraph component to be extensible (#3536) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1690] Added logging to ORC writer abti
• [gobblin] branch master updated: Allow all iceberg exceptions to be fault tolerant (#3541) zihanli58
• [gobblin] branch master updated: Guard against exists fs call as well (#3538) zihanli58
• [gobblin] branch master updated: Add error handling for timeaware finder to handle scenarios where fil… (#3537) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1675] Add pagination for GaaS on server side (#3533) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1672] Refactor metrics from DagManager into its own class, add metrics per … (#3532) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1677] Fix timezone property to read from key correctly (#3535) zihanli58
• [gobblin] branch master updated: [Gobblin-931] Fix typo in gobblin CLI usage (#3530) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1671] : Fix gobblin.sh script to add external jars as colon separated to HADOOP_CLASSPATH (#3531) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1656] Return a http status 503 on GaaS when quota is exceeded for user or flowgroup (#3516) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1669] Clean up TimeAwareRecursiveCopyableDataset to support seconds in time… (#3528) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1670] Remove rat tasks and unneeded checkstyles blocking build pipeline (#3529) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1668] Add audit counts for iceberg registration (#3527) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1667] Create new predicate - ExistingPartitionSkipPredicate (#3526) zihanli58
• [gobblin] branch master updated: Calculate requested container count based on adding allocated count and outstanding ContainerRequests in Yarn (#3524) zihanli58
• [gobblin] branch master updated: make the requestedContainerCountMap correctly update the container count (#3523) zihanli58
• [gobblin] branch master updated: Fix running counts for retried flows (#3520) zihanli58
• [gobblin] branch master updated: Allow table to flush after write failure (#3522) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1652]Add more log in the KafkaJobStatusMonitor in case it fails to process one GobblinTrackingEvent (#3513) zihanli58
• [gobblin] branch master updated: Make Yarn container and helix instance allocation group by tag (#3519) suvasude
• [gobblin] branch master updated: [GOBBLIN-1657] Update completion watermark on change_property in IcebergMetadataWriter (#3517) suvasude
• [gobblin] branch master updated: [GOBBLIN-1654] Add capacity floor to avoid aggressively requesting resource and small files. (#3515) suvasude
• [gobblin] branch master updated: [GOBBLIN-1653] Shorten job name length if it exceeds 255 characters (#3514) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1650] Implement flowGroup quotas for the DagManager (#3511) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1648] Complete use of JDBC `DataSource` 'read-only' validation query by incorporating where previously omitted (#3509) zihanli58
• [gobblin] branch master updated: Add config to set close timeout in HiveRegister (#3512) zihanli58
• [gobblin] branch master updated: add an API in AbstractBaseKafkaConsumerClient to list selected topics (#3501) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1649] Revert gobblin-1633 (#3510) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1639] Prevent metrics reporting if configured, clean up workunit count metric (#3500) zihanli58
• [gobblin] branch master updated (af33445a2 -> d7104e3cc) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1633] Fix compaction actions on job failure not retried if compaction succeeds (#3494) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1646] Revert yarn container / helix tag group changes (#3507) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1641] Add meter for sla exceeded flows (#3502) zihanli58
• [gobblin] branch master updated: GOBBLIN-1644 (#3506) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1645]Change the prefix of dagManager heartbeat to make it consistent with other metrics (#3505) zihanli58
• [gobblin] branch master updated (ffcab02df -> dd6d910a7) suvasude
• [gobblin] branch master updated: [GOBBLIN-1637] Add writer, operation, and partition info to failed metadata writer events (#3498) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1638] Fix unbalanced running count metrics due to Azkaban failures (#3499) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1634] Add retries on flow sla kills (#3495) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1620]Make yarn container allocation group by helix tag (#3487) suvasude
• [gobblin] branch master updated: [GOBBLIN-1636] Close DatasetCleaner after clean task (#3497) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1635] Avoid loading env configuration when using config store to improve the performance (#3496) zihanli58
• [gobblin] branch master updated: use user supplied props to create FileSystem in DatasetCleanerTask (#3483) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1619] WriterUtils.mkdirsWithRecursivePermission contains race condition and puts unnecessary load on filesystem (#3477) zihanli58
• [gobblin] branch master updated: use data node aliases to figure out data node names before using DMAS (#3493) zihanli58
• [gobblin] branch master updated (ffcd2d9dc -> 172d8793d) zihanli58
• [gobblin] branch master updated: [GOBBLIN-1631]Emit heartbeat for dagManagerThread (#3492) zihanli58

• Earlier messages
• Later messages