bhor-sanket opened a new pull request, #791:
URL: https://github.com/apache/atlas/pull/791
## What changes were proposed in this pull request?
### Background
Apache Atlas supports multiple audit repository backends (HBase, RDBMS,
InMemory, Cassandra, Noop). The REST `API GET
/api/atlas/v2/entity/bulk/headers?tagUpdateStartTime={epoch_ms}` enables
consumers to retrieve entities that had classification (tag) changes within a
given time window. This API is used by `ClassificationAssociator.Retriever `to
synchronize classification state across systems.
#### Problem Statement
- **Existing behavior**: The API always returns an empty `guidHeaderMap`
when Atlas is configured with the RDBMS audit backend
(`atlas.EntityAuditRepository.impl=org.apache.atlas.repository.audit.rdbms.RdbmsBasedAuditRepository`).
- **Expected behavior**: The API should return entity GUIDs that had
classification changes (`CLASSIFICATION_ADD, CLASSIFICATION_DELETE,
CLASSIFICATION_UPDATE`) within the specified time range, matching the HBase
backend behavior.
- **Root cause**:
`RdbmsBasedAuditRepository.getEntitiesWithTagChanges(fromTimestamp,
toTimestamp)` was a // TODO: stub returning Collections.emptySet().
- I**mpact**: Classification sync via the bulk headers API is completely
non-functional on RDBMS-backed Atlas deployments.
#### How the patch resolves it
The implementation mirrors the HBase backend's behavior using JPA/JPQL
against the `atlas_entity_audit` table.
1. **New JPA Named Query** (janus-jpa_named_queries.xml)
- `DbEntityAudit.getDistinctEntityIdsByActionPrefixAndTimeRange` — queries
`atlas_entity_audit` for distinct entity IDs where `operation LIKE
'CLASSIFICATION_%'` within a half-open time range (`fromTimestamp,
toTimestamp`)
- Uses `SELECT DISTINCT obj.entityId` to avoid materializing full
DbEntityAudit objects
- `LIKE 'CLASSIFICATION_%'` matches `CLASSIFICATION_ADD,
CLASSIFICATION_DELETE, CLASSIFICATION_UPDATE` but **excludes**
`PROPAGATED_CLASSIFICATION_*` — exact parity with HBase's
BinaryPrefixComparator("CLASSIFICATION_")
- Half-open interval [from, to) matches HBase's Scan.setTimeRange()
semantics
2. **New DAO Method** (DbEntityAuditDao.java)
- `getDistinctEntityIdsByActionPrefixAndTimeRange(String actionPrefix,
long fromTimestamp, long toTimestamp)` — executes the named query with
parameterized action prefix and time range
- Returns List<String> (entity GUIDs), follows existing error-handling
convention
3. **Repository Implementation (RdbmsBasedAuditRepository.java)**
- Replaced the // TODO: stub with a working implementation using
`RdbmsTransaction + DbEntityAuditDao` pattern (consistent with existing
listEventsV2 method)
- Passes "CLASSIFICATION_%" as the action prefix
- Wraps result in HashSet<> to satisfy Set<String> return type
## How was this patch tested?
### Setup :
- Atlas 3.0.0-SNAPSHOT running with RDBMS (PostgreSQL) backend via Docker
compose
- API endpoint: `GET
/api/atlas/v2/entity/bulk/headers?tagUpdateStartTime={epoch_ms}`
### Use-cases validation :
- **Scenario 1: Classification Add** — Added classification to an entity,
called API with timestamp captured before the add. Entity GUID returned with
classification in response. **Passed**.
- **Scenario 2: Classification Delete** — Removed classification from an
entity, called API with timestamp captured before the delete. Entity GUID
returned in response (reflecting current state without the
classification). **Passed**.
- **Scenario 3: Classification Update** — Updated classification attributes,
called API with timestamp before the update. Entity GUID returned with updated
classification attributes. **Passed**.
- **Scenario 4: No classification changes (empty result)** — Performed a
non-classification attribute update, called API. Response returned empty
guidHeaderMap. **Passed**.
- **Scenario 5: Time range boundary** — Added classification, then called
API with a timestamp after the add event. Entity excluded from results as
expected. **Passed**.
- **Scenario 6: Multiple entities** — Added/removed classifications on 3
different entities. All 3 GUIDs returned in a single API response. **Passed**.
- **Scenario 7: Deduplication** — Multiple classification events (add +
update) on the same entity. Entity GUID appeared exactly once. **Passed**.
- **Scenario 8: Propagated classification exclusion** — Added classification
with propagate=true to a parent entity linked via hive_process lineage. Parent
entity (direct CLASSIFICATION_ADD) returned. Child
entity (PROPAGATED_CLASSIFICATION_ADD) excluded — matches HBase
behavior. **Passed**.
### Unit testing
- **Test class:** AuditRepositoryTestBase.java (executed via
InMemoryAuditRepositoryTest)
- **Tests added:**
- **testGetEntitiesWithTagChanges** — verifies CLASSIFICATION_ADD,
CLASSIFICATION_DELETE, CLASSIFICATION_UPDATE events within the time range
return their entity GUIDs
- **testGetEntitiesWithTagChanges_emptyForNoMatchingTimeRange** —
verifies events outside the queried time window are excluded
- **testGetEntitiesWithTagChanges_deduplicatesEntities** — verifies
multiple classification events for the same entity produce only one GUID in the
result set
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
