----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71568/#review218124 -----------------------------------------------------------
addons/models/1000-Hadoop/1110-ml_model.json Lines 1 (patched) <https://reviews.apache.org/r/71568/#comment305699> I think this should follow the conventions stated in this blog post https://blog.cloudera.com/creating-an-open-standard-machine-learning-governance-using-apache-atlas/ Basically strip out the idea of "ml_model" and directly go from project to model build. Here is my implementation which is based on the standards proposed in the blog post https://reviews.apache.org/r/71567/diff/1#21 The reason I am pushing for the standard is because it becomes difficult to update schemas (especially removing types). addons/models/1000-Hadoop/1110-ml_model.json Lines 15 (patched) <https://reviews.apache.org/r/71568/#comment305707> All types should have a guid or uuid. We can call it "resourceName" or "resourceIdentified" or "qualifiedName" or whatever. But there should be an identifier associated which is unique and identifiable. For example, this could be arn (amazon resource name) or a crn (cloudera resource name) addons/models/1000-Hadoop/1110-ml_model.json Lines 165 (patched) <https://reviews.apache.org/r/71568/#comment305702> Anything related to kubernetes needs to be optional. We are designing a generic type system and this might not be a kubernetes deployement. addons/models/1000-Hadoop/1110-ml_model.json Lines 182 (patched) <https://reviews.apache.org/r/71568/#comment305703> Why is cpuMillicores optional and the other memory related things not optional? I think this should be optional as well. addons/models/1000-Hadoop/1110-ml_model.json Lines 204 (patched) <https://reviews.apache.org/r/71568/#comment305704> We should remove this type completely (as per a previous comment) on removing things related to training until we have a good understanding of how to attach training data. addons/models/1000-Hadoop/1110-ml_model.json Lines 284 (patched) <https://reviews.apache.org/r/71568/#comment305705> I think this should be an optional field. addons/models/1000-Hadoop/1110-ml_model.json Lines 318 (patched) <https://reviews.apache.org/r/71568/#comment305706> What if the build is not an s2i build? This should be optional. Actually, most of the fields should be optional except for: 1. user name 2. createdAt (timestamps) 3. uniqueName addons/models/1000-Hadoop/1110-ml_model.json Lines 420 (patched) <https://reviews.apache.org/r/71568/#comment305700> Do we need this relationship definition? The model and model deployment are related through lineage, no? You can follow the lineage from model to model deployment. Do we need another layer of relationship duplicated? addons/models/1000-Hadoop/1110-ml_model.json Lines 439 (patched) <https://reviews.apache.org/r/71568/#comment305701> Same question as before. Why duplicate this relationship when the lineage can define the project -> model relationship. - Karthik Manamcheri On Oct. 4, 2019, 8:21 p.m., Na Li wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71568/ > ----------------------------------------------------------- > > (Updated Oct. 4, 2019, 8:21 p.m.) > > > Review request for atlas, Austin Nobis, Ashutosh Mestry, and Sarath > Subramanian. > > > Repository: atlas > > > Description > ------- > > add integration for ML > > > Diffs > ----- > > addons/hive-bridge/pom.xml 13384cb > addons/ml-bridge-shim/pom.xml PRE-CREATION > > addons/ml-bridge-shim/src/main/java/org/apache/atlas/ml/governance/hooks/GovernanceLineageHook.java > PRE-CREATION > addons/ml-bridge/README.md PRE-CREATION > addons/ml-bridge/pom.xml PRE-CREATION > > addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/HookToolConfiguration.java > PRE-CREATION > > addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/MlGovernanceHookTool.java > PRE-CREATION > > addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/hooks/AtlasMlGovernanceProcessContext.java > PRE-CREATION > > addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/hooks/GovernanceLineageHook.java > PRE-CREATION > > addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/hooks/events/BaseMlGovernanceEvent.java > PRE-CREATION > > addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/hooks/events/CreateMlGovernanceLineage.java > PRE-CREATION > > addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlDataBase.java > PRE-CREATION > > addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlDataType.java > PRE-CREATION > > addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlLifecycleInfo.java > PRE-CREATION > > addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlModel.java > PRE-CREATION > > addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlModelBuild.java > PRE-CREATION > > addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlModelDeployment.java > PRE-CREATION > > addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlModelRef.java > PRE-CREATION > > addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlProject.java > PRE-CREATION > > addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlReplicaStatus.java > PRE-CREATION > > addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlReplicationPolicy.java > PRE-CREATION > > addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlReplicationStatus.java > PRE-CREATION > > addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlResources.java > PRE-CREATION > > addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlTrainingData.java > PRE-CREATION > > addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlTrainingDataEntry.java > PRE-CREATION > > addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlUser.java > PRE-CREATION > addons/ml-bridge/src/main/resources/atlas-log4j.xml PRE-CREATION > addons/ml-bridge/src/main/resources/governance_hook.sh PRE-CREATION > > addons/ml-bridge/src/test/java/org/apache/atlas/ml/governance/GovernanceHookIT.java > PRE-CREATION > > addons/ml-bridge/src/test/java/org/apache/atlas/ml/governance/GovernanceHookITBase.java > PRE-CREATION > addons/ml-bridge/src/test/resources/atlas-application.properties > PRE-CREATION > addons/ml-bridge/src/test/resources/atlas-log4j.xml PRE-CREATION > addons/ml-bridge/src/test/resources/mlModelDeployments.json PRE-CREATION > addons/ml-bridge/src/test/resources/mlModelDeploymentsTrainingData.json > PRE-CREATION > addons/ml-bridge/src/test/resources/mlModels.json PRE-CREATION > addons/ml-bridge/src/test/resources/mlModelsTrainingData.json PRE-CREATION > addons/ml-bridge/src/test/resources/users-credentials.properties > PRE-CREATION > addons/ml-hook-api/pom.xml PRE-CREATION > > addons/ml-hook-api/src/main/java/org/apache/mlops/governance/hooks/Entity.java > PRE-CREATION > > addons/ml-hook-api/src/main/java/org/apache/mlops/governance/hooks/EntityType.java > PRE-CREATION > > addons/ml-hook-api/src/main/java/org/apache/mlops/governance/hooks/GovernanceHook.java > PRE-CREATION > > addons/ml-hook-api/src/main/java/org/apache/mlops/governance/hooks/HookContext.java > PRE-CREATION > > addons/ml-hook-api/src/main/java/org/apache/mlops/governance/hooks/MethodType.java > PRE-CREATION > addons/models/1000-Hadoop/1110-ml_model.json PRE-CREATION > pom.xml ce7d6fb > > > Diff: https://reviews.apache.org/r/71568/diff/3/ > > > Testing > ------- > > add integration tests > > > Thanks, > > Na Li > >
