tpalfy commented on a change in pull request #3979: NIFI-7009: Atlas reporting
task retrieves only the active flow compon…
URL: https://github.com/apache/nifi/pull/3979#discussion_r366877708
##########
File path:
nifi-nar-bundles/nifi-atlas-bundle/nifi-atlas-reporting-task/src/main/java/org/apache/nifi/atlas/NiFiAtlasClient.java
##########
@@ -230,6 +231,35 @@ public NiFiFlow fetchNiFiFlow(String rootProcessGroupId,
String clusterName) thr
return nifiFlow;
}
+ /**
+ * Retrieves the flow components of type {@code componentType} from Atlas
server.
+ * Deleted components will be filtered out before calling Atlas.
+ * Atlas object ids will be initialized with all the attributes (guid,
type, unique attributes) in order to be able
+ * to match ids retrieved from Atlas (having guid) and ids created by the
reporting task (not having guid yet).
+ *
+ * @param componentType Atlas type of the flow component (nifi_flow_path,
nifi_queue, nifi_input_port, nifi_output_port)
+ * @param referredEntities referred entities of the flow entity (returned
when the flow fetched) containing the basic data (id, status) of the flow
components
+ * @return flow component entities mapped to their object ids
+ */
+ private Map<AtlasObjectId, AtlasEntity> fetchFlowComponents(String
componentType, Map<String, AtlasEntity> referredEntities) {
+ return referredEntities.values().stream()
+ .filter(referredEntity ->
referredEntity.getTypeName().equals(componentType))
+ .filter(referredEntity -> referredEntity.getStatus() ==
AtlasEntity.Status.ACTIVE)
+ .map(referredEntity -> {
+ final Map<String, Object> uniqueAttributes =
Collections.singletonMap(ATTR_QUALIFIED_NAME,
referredEntity.getAttribute(ATTR_QUALIFIED_NAME));
+ final AtlasObjectId id = new
AtlasObjectId(referredEntity.getGuid(), componentType, uniqueAttributes);
+ try {
+ final AtlasEntity.AtlasEntityWithExtInfo
fetchedEntityExt = searchEntityDef(id);
+ return new Tuple<>(id, fetchedEntityExt.getEntity());
+ } catch (AtlasServiceException e) {
+ logger.warn("Failed to search entity by id {}, due to
{}", id, e);
+ return null;
Review comment:
We only WARN if a component cannot be retrieved from Atlas then move on. We
don't end up with an incomplete flow because we will try to recreate the (only
seemingly) missing components?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
