Re: Review Request 53724: ATLAS-1116 Performance monitoring of backend methods in API requests
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53724/
---
(Updated Nov. 18, 2016, 10:45 a.m.)
Review request for atlas.
Changes
---
Aspectj plugin disabled by default. Added instructions on how to enable
instrumentation at compile time and to disable metrics at runtime when atlas is
already instrumented.
Bugs: ATLAS-1116
https://issues.apache.org/jira/browse/ATLAS-1116
Repository: atlas
Description
---
Exposes annotations @Monitored and @Loggable on methods.
@Monitored records how many times that method is called and total execution
time in that method. These metrics are aggregated at API level and are recorded
in metric.log(one log line per API call)
@Loggable logs at the beginning of the method and ending of the method with
arguments and return values
These annotations are injected using aspectj plugin at compile time, there is
no runtime overhead except for the method invocation
Example metrics collected:
entity submit - 2016-11-14 16:34:06,079 {GraphHelper.findVertex=[count=3,
totalTimeMSec=60], GraphHelper.setProperty=[count=15, totalTimeMSec=16],
GraphHelper.createVertexWithoutIdentity=[count=1, totalTimeMSec=2],
TypedInstanceToGraphMapper.createVerticesAndDiscoverInstances=[count=1,
totalTimeMSec=62],
TypedInstanceToGraphMapper.addOrUpdateAttributesAndTraits=[count=1,
totalTimeMSec=10], GraphHelper.getSingleValuedProperty=[count=33,
totalTimeMSec=3], GraphHelper.getProperty=[count=3, totalTimeMSec=0],
GraphToTypedInstanceMapper.mapVertexToInstance=[count=3, totalTimeMSec=2],
GraphToTyped
InstanceMapper.mapVertexToInstanceTraits=[count=3, totalTimeMSec=0],
GraphToTypedInstanceMapper.mapGraphToTypedInstance=[count=3, totalTimeMSec=6],
FullTextMapper.mapRecursive=[count=1, totalTimeMSec=3],
TypedInstanceToGraphMapper.mapTypedInstanceToGraph=[count=1, totalTimeMSec=81],
EntityR
esource.submit=[count=1, totalTimeMSec=892]}
search - 2016-11-14 16:34:18,817
{GraphHelper.getSingleValuedProperty=[count=544, totalTimeMSec=20],
GraphHelper.getProperty=[count=47, totalTimeMSec=0],
GraphHelper.getAdjacentEdgesByLabel=[count=18, totalTimeMSec=3],
GraphHelper.getEdgeForLabel=[count=18, totalTimeMSec=3],
GraphToTypedInstanceMapper.mapVertexToInstance=[count=70, totalTimeMSec=33],
GraphToTypedInstanceMapper.mapVertexToInstanceTraits=[count=47,
totalTimeMSec=5], GraphToTypedInstanceMapper.mapGraphToTypedInstance=[count=47,
totalTimeMSec=39], GraphHelper.getArrayElementsProperty=[count=10,
totalTimeMSec=0], MetadataDiscoveryResource.searchUsingQueryDSL=[count=1,
totalTimeMSec=249], MetadataDiscoveryResource.search=[count=1,
totalTimeMSec=249]}
Diffs (updated)
-
distro/src/conf/atlas-log4j.xml 400cd3a
docs/src/site/twiki/Configuration.twiki 7f24f88
docs/src/site/twiki/InstallationSteps.twiki 4dae7a3
pom.xml 1b3975f
repository/pom.xml 949118b
repository/src/main/java/org/apache/atlas/repository/graph/FullTextMapper.java
053e8ac
repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java
7e47d30
repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java
ceb6011
repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
47ae5e1
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasGraphUtilsV1.java
cb389a9
server-api/pom.xml c183468
server-api/src/main/java/org/apache/atlas/RequestContext.java ec38c11
server-api/src/main/java/org/apache/atlas/aspect/AtlasAspect.java
PRE-CREATION
server-api/src/main/java/org/apache/atlas/aspect/Loggable.java PRE-CREATION
server-api/src/main/java/org/apache/atlas/aspect/Monitored.java PRE-CREATION
server-api/src/main/java/org/apache/atlas/metrics/Metrics.java PRE-CREATION
server-api/src/test/aspect/org/apache/atlas/MonitoredAspectTest.java
PRE-CREATION
typesystem/src/main/resources/atlas-log4j.xml 8312657
webapp/pom.xml 2bef274
webapp/src/main/java/org/apache/atlas/web/filters/AuditFilter.java 79b5be4
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
4b45927
webapp/src/main/java/org/apache/atlas/web/resources/DataSetLineageResource.java
a11c0cf
webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java
230265b
webapp/src/main/java/org/apache/atlas/web/resources/LineageResource.java
811c486
webapp/src/main/java/org/apache/atlas/web/resources/MetadataDiscoveryResource.java
ac89869
webapp/src/main/java/org/apache/atlas/web/resources/TypesResource.java
e899fcf
Diff: https://reviews.apache.org/r/53724/diff/
Testing
---
UT added and tested with atlas deployment
Thanks,
Shwetha GS
Re: Review Request 53724: ATLAS-1116 Performance monitoring of backend methods in API requests
> On Nov. 17, 2016, 1:35 a.m., Suma Shivaprasad wrote:
> > Is there a way to disable/enable this if required in production. Can this
> > be made configurable and disabled if required?
Do you mean metrics logging or method inspection? Method inspection is done at
compile time. Metric logging is controlled using log4j. So, log4j.xml can be
modified to not log the metrics
- Shwetha
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53724/#review156163
---
On Nov. 16, 2016, 9:33 a.m., Shwetha GS wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53724/
> ---
>
> (Updated Nov. 16, 2016, 9:33 a.m.)
>
>
> Review request for atlas.
>
>
> Bugs: ATLAS-1116
> https://issues.apache.org/jira/browse/ATLAS-1116
>
>
> Repository: atlas
>
>
> Description
> ---
>
> Exposes annotations @Monitored and @Loggable on methods.
>
> @Monitored records how many times that method is called and total execution
> time in that method. These metrics are aggregated at API level and are
> recorded in metric.log(one log line per API call)
> @Loggable logs at the beginning of the method and ending of the method with
> arguments and return values
>
> These annotations are injected using aspectj plugin at compile time, there is
> no runtime overhead except for the method invocation
>
> Example metrics collected:
> entity submit - 2016-11-14 16:34:06,079 {GraphHelper.findVertex=[count=3,
> totalTimeMSec=60], GraphHelper.setProperty=[count=15, totalTimeMSec=16],
> GraphHelper.createVertexWithoutIdentity=[count=1, totalTimeMSec=2],
> TypedInstanceToGraphMapper.createVerticesAndDiscoverInstances=[count=1,
> totalTimeMSec=62],
> TypedInstanceToGraphMapper.addOrUpdateAttributesAndTraits=[count=1,
> totalTimeMSec=10], GraphHelper.getSingleValuedProperty=[count=33,
> totalTimeMSec=3], GraphHelper.getProperty=[count=3, totalTimeMSec=0],
> GraphToTypedInstanceMapper.mapVertexToInstance=[count=3, totalTimeMSec=2],
> GraphToTyped
> InstanceMapper.mapVertexToInstanceTraits=[count=3, totalTimeMSec=0],
> GraphToTypedInstanceMapper.mapGraphToTypedInstance=[count=3,
> totalTimeMSec=6], FullTextMapper.mapRecursive=[count=1, totalTimeMSec=3],
> TypedInstanceToGraphMapper.mapTypedInstanceToGraph=[count=1,
> totalTimeMSec=81], EntityR
> esource.submit=[count=1, totalTimeMSec=892]}
>
> search - 2016-11-14 16:34:18,817
> {GraphHelper.getSingleValuedProperty=[count=544, totalTimeMSec=20],
> GraphHelper.getProperty=[count=47, totalTimeMSec=0],
> GraphHelper.getAdjacentEdgesByLabel=[count=18, totalTimeMSec=3],
> GraphHelper.getEdgeForLabel=[count=18, totalTimeMSec=3],
> GraphToTypedInstanceMapper.mapVertexToInstance=[count=70, totalTimeMSec=33],
> GraphToTypedInstanceMapper.mapVertexToInstanceTraits=[count=47,
> totalTimeMSec=5],
> GraphToTypedInstanceMapper.mapGraphToTypedInstance=[count=47,
> totalTimeMSec=39], GraphHelper.getArrayElementsProperty=[count=10,
> totalTimeMSec=0], MetadataDiscoveryResource.searchUsingQueryDSL=[count=1,
> totalTimeMSec=249], MetadataDiscoveryResource.search=[count=1,
> totalTimeMSec=249]}
>
>
> Diffs
> -
>
> distro/src/conf/atlas-log4j.xml 400cd3a
> pom.xml 1b3975f
> repository/pom.xml 949118b
>
> repository/src/main/java/org/apache/atlas/repository/graph/FullTextMapper.java
> 053e8ac
> repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java
> 7e47d30
>
> repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java
> ceb6011
>
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
> 47ae5e1
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasGraphUtilsV1.java
> cb389a9
> server-api/pom.xml c183468
> server-api/src/main/aspect/org/apache/atlas/AtlasAspect.java PRE-CREATION
> server-api/src/main/aspect/org/apache/atlas/Loggable.java PRE-CREATION
> server-api/src/main/aspect/org/apache/atlas/Monitored.java PRE-CREATION
> server-api/src/main/java/org/apache/atlas/RequestContext.java ec38c11
> server-api/src/main/java/org/apache/atlas/metrics/Metrics.java PRE-CREATION
> server-api/src/test/aspect/org/apache/atlas/MonitoredAspectTest.java
> PRE-CREATION
> typesystem/src/main/resources/atlas-log4j.xml 8312657
> webapp/pom.xml 2bef274
> webapp/src/main/java/org/apache/atlas/web/
Re: Review Request 52257: Return system attributes in get entity definition
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52257/#review156176 --- repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java (line 529) <https://reviews.apache.org/r/52257/#comment226310> why can't the default be null, if not set? Same for other properties - Shwetha GS On Nov. 16, 2016, 1:33 p.m., Vimal Sharma wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/52257/ > --- > > (Updated Nov. 16, 2016, 1:33 p.m.) > > > Review request for atlas. > > > Bugs: ATLAS-916 > https://issues.apache.org/jira/browse/ATLAS-916 > > > Repository: atlas > > > Description > --- > > Atlas should maintain system attributes created time, last modified time, > created by user and last modified by user for every entity. This information > should be returned in get entity definition > > > Diffs > - > > catalog/src/main/java/org/apache/atlas/catalog/DefaultPropertyMapper.java > 2f52b3b > common/src/main/java/org/apache/atlas/repository/Constants.java cc184a5 > > repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java > fb014f2 > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java > 691a12c > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java > aea54fa > repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java > 7e47d30 > > repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java > ceb6011 > > repository/src/main/java/org/apache/atlas/repository/graph/SoftDeleteHandler.java > 92e43cb > > repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java > 47ae5e1 > > repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java > 645fef1 > > repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java > 725b9a6 > > typesystem/src/main/java/org/apache/atlas/typesystem/IReferenceableInstance.java > 9285014 > typesystem/src/main/java/org/apache/atlas/typesystem/Referenceable.java > 5b8e157 > > typesystem/src/main/java/org/apache/atlas/typesystem/persistence/AtlasSystemAttributes.java > PRE-CREATION > typesystem/src/main/java/org/apache/atlas/typesystem/persistence/Id.java > 42280d0 > > typesystem/src/main/java/org/apache/atlas/typesystem/persistence/ReferenceableInstance.java > 561cb62 > typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java > 6b530a8 > > typesystem/src/main/scala/org/apache/atlas/typesystem/json/InstanceSerialization.scala > 73b3526 > > typesystem/src/main/scala/org/apache/atlas/typesystem/json/Serialization.scala > 68c47ec > > Diff: https://reviews.apache.org/r/52257/diff/ > > > Testing > --- > > Testing done with Hive table creation and updation > > > Thanks, > > Vimal Sharma > >
Re: Review Request 53724: ATLAS-1116 Performance monitoring of backend methods in API requests
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53724/
---
(Updated Nov. 16, 2016, 9:33 a.m.)
Review request for atlas.
Changes
---
Re-based patch
Bugs: ATLAS-1116
https://issues.apache.org/jira/browse/ATLAS-1116
Repository: atlas
Description
---
Exposes annotations @Monitored and @Loggable on methods.
@Monitored records how many times that method is called and total execution
time in that method. These metrics are aggregated at API level and are recorded
in metric.log(one log line per API call)
@Loggable logs at the beginning of the method and ending of the method with
arguments and return values
These annotations are injected using aspectj plugin at compile time, there is
no runtime overhead except for the method invocation
Example metrics collected:
entity submit - 2016-11-14 16:34:06,079 {GraphHelper.findVertex=[count=3,
totalTimeMSec=60], GraphHelper.setProperty=[count=15, totalTimeMSec=16],
GraphHelper.createVertexWithoutIdentity=[count=1, totalTimeMSec=2],
TypedInstanceToGraphMapper.createVerticesAndDiscoverInstances=[count=1,
totalTimeMSec=62],
TypedInstanceToGraphMapper.addOrUpdateAttributesAndTraits=[count=1,
totalTimeMSec=10], GraphHelper.getSingleValuedProperty=[count=33,
totalTimeMSec=3], GraphHelper.getProperty=[count=3, totalTimeMSec=0],
GraphToTypedInstanceMapper.mapVertexToInstance=[count=3, totalTimeMSec=2],
GraphToTyped
InstanceMapper.mapVertexToInstanceTraits=[count=3, totalTimeMSec=0],
GraphToTypedInstanceMapper.mapGraphToTypedInstance=[count=3, totalTimeMSec=6],
FullTextMapper.mapRecursive=[count=1, totalTimeMSec=3],
TypedInstanceToGraphMapper.mapTypedInstanceToGraph=[count=1, totalTimeMSec=81],
EntityR
esource.submit=[count=1, totalTimeMSec=892]}
search - 2016-11-14 16:34:18,817
{GraphHelper.getSingleValuedProperty=[count=544, totalTimeMSec=20],
GraphHelper.getProperty=[count=47, totalTimeMSec=0],
GraphHelper.getAdjacentEdgesByLabel=[count=18, totalTimeMSec=3],
GraphHelper.getEdgeForLabel=[count=18, totalTimeMSec=3],
GraphToTypedInstanceMapper.mapVertexToInstance=[count=70, totalTimeMSec=33],
GraphToTypedInstanceMapper.mapVertexToInstanceTraits=[count=47,
totalTimeMSec=5], GraphToTypedInstanceMapper.mapGraphToTypedInstance=[count=47,
totalTimeMSec=39], GraphHelper.getArrayElementsProperty=[count=10,
totalTimeMSec=0], MetadataDiscoveryResource.searchUsingQueryDSL=[count=1,
totalTimeMSec=249], MetadataDiscoveryResource.search=[count=1,
totalTimeMSec=249]}
Diffs (updated)
-
distro/src/conf/atlas-log4j.xml 400cd3a
pom.xml 1b3975f
repository/pom.xml 949118b
repository/src/main/java/org/apache/atlas/repository/graph/FullTextMapper.java
053e8ac
repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java
7e47d30
repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java
ceb6011
repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
47ae5e1
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasGraphUtilsV1.java
cb389a9
server-api/pom.xml c183468
server-api/src/main/aspect/org/apache/atlas/AtlasAspect.java PRE-CREATION
server-api/src/main/aspect/org/apache/atlas/Loggable.java PRE-CREATION
server-api/src/main/aspect/org/apache/atlas/Monitored.java PRE-CREATION
server-api/src/main/java/org/apache/atlas/RequestContext.java ec38c11
server-api/src/main/java/org/apache/atlas/metrics/Metrics.java PRE-CREATION
server-api/src/test/aspect/org/apache/atlas/MonitoredAspectTest.java
PRE-CREATION
typesystem/src/main/resources/atlas-log4j.xml 8312657
webapp/pom.xml 2bef274
webapp/src/main/java/org/apache/atlas/web/filters/AuditFilter.java 79b5be4
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
4b45927
webapp/src/main/java/org/apache/atlas/web/resources/DataSetLineageResource.java
a11c0cf
webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java
230265b
webapp/src/main/java/org/apache/atlas/web/resources/LineageResource.java
811c486
webapp/src/main/java/org/apache/atlas/web/resources/MetadataDiscoveryResource.java
ac89869
webapp/src/main/java/org/apache/atlas/web/resources/TypesResource.java
1eafc7d
Diff: https://reviews.apache.org/r/53724/diff/
Testing
---
UT added and tested with atlas deployment
Thanks,
Shwetha GS
Re: Review Request 52257: Return system attributes in get entity definition
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52257/#review156026 --- Fix it, then Ship it! repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java (line 78) <https://reviews.apache.org/r/52257/#comment226126> Use TypeSystem.getInstance().getDateFormat typesystem/src/main/java/org/apache/atlas/typesystem/persistence/AtlasSystemAttributes.java (line 87) <https://reviews.apache.org/r/52257/#comment226125> Should return Date object to avoid the confusion of the date format used in string - Shwetha GS On Nov. 15, 2016, 4:33 p.m., Vimal Sharma wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/52257/ > --- > > (Updated Nov. 15, 2016, 4:33 p.m.) > > > Review request for atlas. > > > Bugs: ATLAS-916 > https://issues.apache.org/jira/browse/ATLAS-916 > > > Repository: atlas > > > Description > --- > > Atlas should maintain system attributes created time, last modified time, > created by user and last modified by user for every entity. This information > should be returned in get entity definition > > > Diffs > - > > catalog/src/main/java/org/apache/atlas/catalog/DefaultPropertyMapper.java > 2f52b3b > common/src/main/java/org/apache/atlas/repository/Constants.java cc184a5 > > repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java > fb014f2 > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java > 691a12c > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java > aea54fa > repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java > 7e47d30 > > repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java > ceb6011 > > repository/src/main/java/org/apache/atlas/repository/graph/SoftDeleteHandler.java > 92e43cb > > repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java > 47ae5e1 > > repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java > 645fef1 > > repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java > 725b9a6 > > typesystem/src/main/java/org/apache/atlas/typesystem/IReferenceableInstance.java > 9285014 > typesystem/src/main/java/org/apache/atlas/typesystem/Referenceable.java > 5b8e157 > > typesystem/src/main/java/org/apache/atlas/typesystem/persistence/AtlasSystemAttributes.java > PRE-CREATION > typesystem/src/main/java/org/apache/atlas/typesystem/persistence/Id.java > 42280d0 > > typesystem/src/main/java/org/apache/atlas/typesystem/persistence/ReferenceableInstance.java > 561cb62 > typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java > 6b530a8 > > typesystem/src/main/scala/org/apache/atlas/typesystem/json/InstanceSerialization.scala > 73b3526 > > typesystem/src/main/scala/org/apache/atlas/typesystem/json/Serialization.scala > 68c47ec > > Diff: https://reviews.apache.org/r/52257/diff/ > > > Testing > --- > > Testing done with Hive table creation and updation > > > Thanks, > > Vimal Sharma > >
Re: Review Request 53724: ATLAS-1116 Performance monitoring of backend methods in API requests
> On Nov. 15, 2016, 5:05 p.m., Apoorv Naik wrote:
> > server-api/pom.xml, line 65
> > <https://reviews.apache.org/r/53724/diff/1/?file=1563010#file1563010line65>
> >
> > Why not use the existing MethodInterceptor mechanism that we're using
> > for GraphTransaction ? Seems like we're overloading our project with lot of
> > dependencies and compilation configurations. The builds would get very
> > finicky in my opinion.
Guice method interceptor will work only if the class is created through guice
injection. Most of the classes like GraphToTypedInstanceMapper,
TypedInstanceToGraphMapper, GraphHelper which need to be measured are not
injected through guice. So, Guice method interceptor didn't work. The code
changes to create these classes through guice were hacky and cumbersome.
Moreover, guice method interceptor is done at runtime vs aspectj compile time
injection
- Shwetha
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53724/#review155932
-------
On Nov. 14, 2016, 11:06 a.m., Shwetha GS wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53724/
> ---
>
> (Updated Nov. 14, 2016, 11:06 a.m.)
>
>
> Review request for atlas.
>
>
> Bugs: ATLAS-1116
> https://issues.apache.org/jira/browse/ATLAS-1116
>
>
> Repository: atlas
>
>
> Description
> ---
>
> Exposes annotations @Monitored and @Loggable on methods.
>
> @Monitored records how many times that method is called and total execution
> time in that method. These metrics are aggregated at API level and are
> recorded in metric.log(one log line per API call)
> @Loggable logs at the beginning of the method and ending of the method with
> arguments and return values
>
> These annotations are injected using aspectj plugin at compile time, there is
> no runtime overhead except for the method invocation
>
> Example metrics collected:
> entity submit - 2016-11-14 16:34:06,079 {GraphHelper.findVertex=[count=3,
> totalTimeMSec=60], GraphHelper.setProperty=[count=15, totalTimeMSec=16],
> GraphHelper.createVertexWithoutIdentity=[count=1, totalTimeMSec=2],
> TypedInstanceToGraphMapper.createVerticesAndDiscoverInstances=[count=1,
> totalTimeMSec=62],
> TypedInstanceToGraphMapper.addOrUpdateAttributesAndTraits=[count=1,
> totalTimeMSec=10], GraphHelper.getSingleValuedProperty=[count=33,
> totalTimeMSec=3], GraphHelper.getProperty=[count=3, totalTimeMSec=0],
> GraphToTypedInstanceMapper.mapVertexToInstance=[count=3, totalTimeMSec=2],
> GraphToTyped
> InstanceMapper.mapVertexToInstanceTraits=[count=3, totalTimeMSec=0],
> GraphToTypedInstanceMapper.mapGraphToTypedInstance=[count=3,
> totalTimeMSec=6], FullTextMapper.mapRecursive=[count=1, totalTimeMSec=3],
> TypedInstanceToGraphMapper.mapTypedInstanceToGraph=[count=1,
> totalTimeMSec=81], EntityR
> esource.submit=[count=1, totalTimeMSec=892]}
>
> search - 2016-11-14 16:34:18,817
> {GraphHelper.getSingleValuedProperty=[count=544, totalTimeMSec=20],
> GraphHelper.getProperty=[count=47, totalTimeMSec=0],
> GraphHelper.getAdjacentEdgesByLabel=[count=18, totalTimeMSec=3],
> GraphHelper.getEdgeForLabel=[count=18, totalTimeMSec=3],
> GraphToTypedInstanceMapper.mapVertexToInstance=[count=70, totalTimeMSec=33],
> GraphToTypedInstanceMapper.mapVertexToInstanceTraits=[count=47,
> totalTimeMSec=5],
> GraphToTypedInstanceMapper.mapGraphToTypedInstance=[count=47,
> totalTimeMSec=39], GraphHelper.getArrayElementsProperty=[count=10,
> totalTimeMSec=0], MetadataDiscoveryResource.searchUsingQueryDSL=[count=1,
> totalTimeMSec=249], MetadataDiscoveryResource.search=[count=1,
> totalTimeMSec=249]}
>
>
> Diffs
> -
>
> distro/src/conf/atlas-log4j.xml 400cd3a
> pom.xml 1b3975f
> repository/pom.xml 949118b
>
> repository/src/main/java/org/apache/atlas/repository/graph/FullTextMapper.java
> 053e8ac
> repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java
> 7e47d30
>
> repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java
> ceb6011
>
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
> 47ae5e1
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasGraphUtilsV1.java
> cb389a9
> server-api/pom.xml c183468
> server-api/src/main/aspect/org/apache/atlas/AtlasAspect.java PRE-CREATION
> serve
Re: Review Request 53525: ATLAS-1246 Upgrade versions of dependencies
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/53525/ --- (Updated Nov. 15, 2016, 8:17 a.m.) Review request for atlas. Changes --- Addressed review comment Bugs: ATLAS-1246 https://issues.apache.org/jira/browse/ATLAS-1246 Repository: atlas Description --- 1. Changed scala version to 2.11 2. Changed guava version to 18.0. hbase and titan doesn't work with guava 18. In general, when hbase and titan versions are changed using properties, the guava versions that hbase, titan and atlas require may be incompatible and will fail at runtime. So, shaded the guava dependencies in hbase and titan Diffs (updated) - addons/falcon-bridge/pom.xml 96743e7 addons/hive-bridge/pom.xml 7ab47f1 addons/sqoop-bridge/pom.xml e71d64a addons/storm-bridge/pom.xml 6be55b4 authorization/src/main/java/org/apache/atlas/authorize/simple/PolicyParser.java fe81598 common/pom.xml e3b6465 graphdb/common/pom.xml 43d51f6 graphdb/titan0/pom.xml ca8c0fb intg/pom.xml b29a0c2 notification/pom.xml 90d84ec pom.xml 2044bfd repository/pom.xml af5d4a1 repository/src/main/java/org/apache/atlas/discovery/DataSetLineageService.java c3fd72b shaded/hbase-client-shaded/pom.xml PRE-CREATION shaded/hbase-server-shaded/pom.xml PRE-CREATION typesystem/pom.xml 28c077d webapp/pom.xml b9e Diff: https://reviews.apache.org/r/53525/diff/ Testing --- UTs and end to end test external hbase and solr Thanks, Shwetha GS
Re: Review Request 53653: ATLAS-1284 Fixed issues with order by clause while sorting with alias and on super type attributes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/53653/#review155887 --- Will query 'hive_table select owner as o orderby o' work as well? - Shwetha GS On Nov. 11, 2016, 2:34 p.m., Neeru Gupta wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/53653/ > --- > > (Updated Nov. 11, 2016, 2:34 p.m.) > > > Review request for atlas and Jeff Hagelberg. > > > Repository: atlas > > > Description > --- > > Merge branch 'master' into oms-750 > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/gremlin/Gremlin2ExpressionFactory.java > a901b5127b3dcd2939fec2c642475d7aad869d8e > > repository/src/main/java/org/apache/atlas/gremlin/Gremlin3ExpressionFactory.java > 32189bc6dab4cc25748e7b6136dc0a89476012c7 > repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala > ee58eee7b4b2c41cc8f6dc1e4952b83ed0c833df > repository/src/main/scala/org/apache/atlas/query/Resolver.scala > cff92afe98a6215499a70a9ef00a937804abe73e > repository/src/test/java/org/apache/atlas/BaseRepositoryTest.java > 8851b793b0ca032efb57bd76ec9198822e1520c2 > > repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java > 645fef1fb3abf8cd2d4abf4429b00c32837ef0c0 > > Diff: https://reviews.apache.org/r/53653/diff/ > > > Testing > --- > > Executed all existing order by tests successfully. Added new tests to order > by with alias > > > Thanks, > > Neeru Gupta > >
Re: Review Request 53749: [ATLAS-1293] Add query text information for hive_process entities in lineage
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/53749/#review155886 --- repository/src/main/java/org/apache/atlas/discovery/EntityLineageService.java (line 199) <https://reviews.apache.org/r/53749/#comment225983> For hdfs path, qualified name contains the full path including namenode endpoint which is not relevant while viewing the lineage(Only part of the value is shown in UI and only the namenode endpoint is visible, which is not a good user experience). Instead of hardcoding these attributes on server side, we should make this part of model(type definition), just like attributes marked for search results display - Shwetha GS On Nov. 15, 2016, 1:44 a.m., Sarath Subramanian wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/53749/ > --- > > (Updated Nov. 15, 2016, 1:44 a.m.) > > > Review request for atlas, Apoorv Naik, keval bhatt, Madhan Neethiraj, Shwetha > GS, and Suma Shivaprasad. > > > Bugs: ATLAS-1293 > https://issues.apache.org/jira/browse/ATLAS-1293 > > > Repository: atlas > > > Description > --- > > When UI renders lineage for an entity using v2 lineage APIs, query text needs > to be displayed for hive_process types. > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/discovery/EntityLineageService.java > 14bf143 > > Diff: https://reviews.apache.org/r/53749/diff/ > > > Testing > --- > > verified using POST man rest client and ran all unit tests. > > > Thanks, > > Sarath Subramanian > >
Review Request 53724: ATLAS-1116 Performance monitoring of backend methods in API requests
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53724/
---
Review request for atlas.
Bugs: ATLAS-1116
https://issues.apache.org/jira/browse/ATLAS-1116
Repository: atlas
Description
---
Exposes annotations @Monitored and @Loggable on methods.
@Monitored records how many times that method is called and total execution
time in that method. These metrics are aggregated at API level and are recorded
in metric.log(one log line per API call)
@Loggable logs at the beginning of the method and ending of the method with
arguments and return values
These annotations are injected using aspectj plugin at compile time, there is
no runtime overhead except for the method invocation
Example metrics collected:
entity submit - 2016-11-14 16:34:06,079 {GraphHelper.findVertex=[count=3,
totalTimeMSec=60], GraphHelper.setProperty=[count=15, totalTimeMSec=16],
GraphHelper.createVertexWithoutIdentity=[count=1, totalTimeMSec=2],
TypedInstanceToGraphMapper.createVerticesAndDiscoverInstances=[count=1,
totalTimeMSec=62],
TypedInstanceToGraphMapper.addOrUpdateAttributesAndTraits=[count=1,
totalTimeMSec=10], GraphHelper.getSingleValuedProperty=[count=33,
totalTimeMSec=3], GraphHelper.getProperty=[count=3, totalTimeMSec=0],
GraphToTypedInstanceMapper.mapVertexToInstance=[count=3, totalTimeMSec=2],
GraphToTyped
InstanceMapper.mapVertexToInstanceTraits=[count=3, totalTimeMSec=0],
GraphToTypedInstanceMapper.mapGraphToTypedInstance=[count=3, totalTimeMSec=6],
FullTextMapper.mapRecursive=[count=1, totalTimeMSec=3],
TypedInstanceToGraphMapper.mapTypedInstanceToGraph=[count=1, totalTimeMSec=81],
EntityR
esource.submit=[count=1, totalTimeMSec=892]}
search - 2016-11-14 16:34:18,817
{GraphHelper.getSingleValuedProperty=[count=544, totalTimeMSec=20],
GraphHelper.getProperty=[count=47, totalTimeMSec=0],
GraphHelper.getAdjacentEdgesByLabel=[count=18, totalTimeMSec=3],
GraphHelper.getEdgeForLabel=[count=18, totalTimeMSec=3],
GraphToTypedInstanceMapper.mapVertexToInstance=[count=70, totalTimeMSec=33],
GraphToTypedInstanceMapper.mapVertexToInstanceTraits=[count=47,
totalTimeMSec=5], GraphToTypedInstanceMapper.mapGraphToTypedInstance=[count=47,
totalTimeMSec=39], GraphHelper.getArrayElementsProperty=[count=10,
totalTimeMSec=0], MetadataDiscoveryResource.searchUsingQueryDSL=[count=1,
totalTimeMSec=249], MetadataDiscoveryResource.search=[count=1,
totalTimeMSec=249]}
Diffs
-
distro/src/conf/atlas-log4j.xml 400cd3a
pom.xml 1b3975f
repository/pom.xml 949118b
repository/src/main/java/org/apache/atlas/repository/graph/FullTextMapper.java
053e8ac
repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java
7e47d30
repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java
ceb6011
repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
47ae5e1
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasGraphUtilsV1.java
cb389a9
server-api/pom.xml c183468
server-api/src/main/aspect/org/apache/atlas/AtlasAspect.java PRE-CREATION
server-api/src/main/aspect/org/apache/atlas/Loggable.java PRE-CREATION
server-api/src/main/aspect/org/apache/atlas/Monitored.java PRE-CREATION
server-api/src/main/java/org/apache/atlas/RequestContext.java ec38c11
server-api/src/main/java/org/apache/atlas/metrics/Metrics.java PRE-CREATION
server-api/src/test/aspect/org/apache/atlas/MonitoredAspectTest.java
PRE-CREATION
typesystem/src/main/resources/atlas-log4j.xml 8312657
webapp/pom.xml 6d83cc0
webapp/src/main/java/org/apache/atlas/web/filters/AuditFilter.java 79b5be4
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
4b45927
webapp/src/main/java/org/apache/atlas/web/resources/DataSetLineageResource.java
a11c0cf
webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java
bbf01a6
webapp/src/main/java/org/apache/atlas/web/resources/LineageResource.java
811c486
webapp/src/main/java/org/apache/atlas/web/resources/MetadataDiscoveryResource.java
64344da
webapp/src/main/java/org/apache/atlas/web/resources/TypesResource.java
ace0d14
Diff: https://reviews.apache.org/r/53724/diff/
Testing
---
UT added and tested with atlas deployment
Thanks,
Shwetha GS
Re: Review Request 53525: ATLAS-1246 Upgrade versions of dependencies
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/53525/ --- (Updated Nov. 14, 2016, 9:01 a.m.) Review request for atlas. Changes --- mvn 3.3.x includes hbase and titan jars even though they are shaded and used as uber jar. So, excluding them explicitly in maven war plugin. I have verified this on the stack Bugs: ATLAS-1246 https://issues.apache.org/jira/browse/ATLAS-1246 Repository: atlas Description --- 1. Changed scala version to 2.11 2. Changed guava version to 18.0. hbase and titan doesn't work with guava 18. In general, when hbase and titan versions are changed using properties, the guava versions that hbase, titan and atlas require may be incompatible and will fail at runtime. So, shaded the guava dependencies in hbase and titan Diffs (updated) - addons/falcon-bridge/pom.xml 96743e7 addons/hive-bridge/pom.xml 7ab47f1 addons/sqoop-bridge/pom.xml e71d64a addons/storm-bridge/pom.xml 6be55b4 authorization/src/main/java/org/apache/atlas/authorize/simple/PolicyParser.java fe81598 common/pom.xml e3b6465 graphdb/common/pom.xml 43d51f6 graphdb/titan0/pom.xml ca8c0fb intg/pom.xml b29a0c2 notification/pom.xml 90d84ec pom.xml 2044bfd repository/pom.xml af5d4a1 repository/src/main/java/org/apache/atlas/discovery/DataSetLineageService.java c3fd72b shaded/hbase-client-shaded/pom.xml PRE-CREATION shaded/hbase-server-shaded/pom.xml PRE-CREATION typesystem/pom.xml 28c077d webapp/pom.xml b9e Diff: https://reviews.apache.org/r/53525/diff/ Testing --- UTs and end to end test external hbase and solr Thanks, Shwetha GS
Review Request 53525: ATLAS-1246 Upgrade versions of dependencies
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/53525/ --- Review request for atlas. Bugs: ATLAS-1246 https://issues.apache.org/jira/browse/ATLAS-1246 Repository: atlas Description --- 1. Changed scala version to 2.11 2. Changed guava version to 18.0. hbase and titan doesn't work with guava 18. In general, when hbase and titan versions are changed using properties, the guava versions that hbase, titan and atlas require may be incompatible and will fail at runtime. So, shaded the guava dependencies in hbase and titan Diffs - addons/falcon-bridge/pom.xml de3e810 addons/hive-bridge/pom.xml 884791d addons/sqoop-bridge/pom.xml 9ac411e addons/storm-bridge/pom.xml fcb6e01 authorization/src/main/java/org/apache/atlas/authorize/simple/PolicyParser.java fe81598 common/pom.xml e3b6465 graphdb/common/pom.xml 43d51f6 graphdb/titan0/pom.xml ca8c0fb intg/pom.xml b29a0c2 notification/pom.xml 90d84ec pom.xml a6fa7fb repository/pom.xml af5d4a1 repository/src/main/java/org/apache/atlas/discovery/DataSetLineageService.java c3fd72b shaded/hbase-client-shaded/pom.xml PRE-CREATION shaded/hbase-server-shaded/pom.xml PRE-CREATION typesystem/pom.xml 28c077d webapp/pom.xml 6dbd484 Diff: https://reviews.apache.org/r/53525/diff/ Testing --- UTs and end to end test external hbase and solr Thanks, Shwetha GS
Re: Review Request 52914: ATLAS-1226: updated servlet initialization to use params in web.xml
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52914/#review152824 --- Ship it! Ship It! - Shwetha GS On Oct. 15, 2016, 9:11 p.m., Madhan Neethiraj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/52914/ > --- > > (Updated Oct. 15, 2016, 9:11 p.m.) > > > Review request for atlas. > > > Bugs: ATLAS-1226 > https://issues.apache.org/jira/browse/ATLAS-1226 > > > Repository: atlas > > > Description > --- > > Updated servlet initialization to use all init-parameters specified in > web.xml. Replaced jackson.version in pom.xml with codehaus.jackson.version. > > > Diffs > - > > pom.xml 811d9b9 > webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java > 7a8a4f0 > > Diff: https://reviews.apache.org/r/52914/diff/ > > > Testing > --- > > - verified that all unit tests pass successfully > - install the updated Atlas server and verified that UI works > > > Thanks, > > Madhan Neethiraj > >
Re: Review Request 52585: Implementation of the new TypesREST API
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52585/#review151767
---
intg module should contain only the client side classes which is required for
integration from external users. AtlasTypeRegistry on the other hand is the
cache of all types on atlas service. So, AtlasTypeRegistry should be moved out
to a server side module - either existing/new module is your choice.
AtlasTypeRegistry.getAllTypeNames() - eventually for atlas to scale, caching
all types in memory is not possible. GraphBackedTypeStore implements graph
based type store wherein types are loaded on demand and is used by IBM.
getAllTypeNames() can't be supported with GraphBackedTypeStore. In fact, all
get all methods should be removed like getAllEnumDefs() etc. contains() method
can be added instead
@dave kantor, can you review the interfaces to make sure your usecases are
covered?
intg/src/main/java/org/apache/atlas/type/AtlasStructType.java (line 73)
<https://reviews.apache.org/r/52585/#comment220268>
typo - ype to type
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
(line 275)
<https://reviews.apache.org/r/52585/#comment220271>
why change the log formats, log statements at other places use {}. To be
consistent, lets leave it as is?
repository/src/main/java/org/apache/atlas/repository/graph/mapper/BaseTypeDefMapper.java
(line 33)
<https://reviews.apache.org/r/52585/#comment220269>
These are defined in Constants as well
- Shwetha GS
On Oct. 7, 2016, 7:01 a.m., Apoorv Naik wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52585/
> ---
>
> (Updated Oct. 7, 2016, 7:01 a.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Sarath Kumar Subramanian, Shwetha
> GS, and Suma Shivaprasad.
>
>
> Repository: atlas
>
>
> Description
> ---
>
> TypesREST implementation
>
> TODO: Update and delete operations need to be refined
>
>
> Diffs
> -
>
> .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406
> intg/src/main/java/org/apache/atlas/model/typedef/AtlasBaseTypeDef.java
> a3fdbcc025bf5f56e7770015d55616ec83d2be64
> intg/src/main/java/org/apache/atlas/model/typedef/AtlasTypesDef.java
> PRE-CREATION
> intg/src/main/java/org/apache/atlas/type/AtlasStructType.java
> 096f061f16bba3ecb886d00ec66c98ad17c1e243
> repository/pom.xml 7a34331ce2876d9d9e916f96f1141d3efa91e959
>
> repository/src/main/java/org/apache/atlas/repository/graph/FullTextMapper.java
> b342e2700d454b0d6fba595b5cc01cd0e06bbdac
>
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
> 263ea465fda0b445a952943def9a6f7c49834f25
> repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java
> 1ce87c9d306faa43fb9d3fdc491c4bcbdd7b2bdb
>
> repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java
> 5c7cb2e8fa32b540f80beed40fb4f25a89d39c56
>
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
> 2e0414e2cee7ca3d5958650ac6abc8a290473545
>
> repository/src/main/java/org/apache/atlas/repository/graph/mapper/AtlasClassificationDefMapper.java
> PRE-CREATION
>
> repository/src/main/java/org/apache/atlas/repository/graph/mapper/AtlasEntityDefMapper.java
> PRE-CREATION
>
> repository/src/main/java/org/apache/atlas/repository/graph/mapper/AtlasEnumDefMapper.java
> PRE-CREATION
>
> repository/src/main/java/org/apache/atlas/repository/graph/mapper/AtlasStructDefMapper.java
> PRE-CREATION
>
> repository/src/main/java/org/apache/atlas/repository/graph/mapper/BaseTypeDefMapper.java
> PRE-CREATION
>
> repository/src/main/java/org/apache/atlas/repository/graph/mapper/factory/AtlasMapperFactory.java
> PRE-CREATION
>
> repository/src/main/java/org/apache/atlas/repository/graph/mapper/impl/AtlasClassificationDefMapperV1.java
> PRE-CREATION
>
> repository/src/main/java/org/apache/atlas/repository/graph/mapper/impl/AtlasEntityDefMapperV1.java
> PRE-CREATION
>
> repository/src/main/java/org/apache/atlas/repository/graph/mapper/impl/AtlasEnumDefMapperV1.java
> PRE-CREATION
>
> repository/src/main/java/org/apache/atlas/repository/graph/mapper/impl/AtlasStructDefMapperV1.java
> PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/AtlasStore.java
> PRE-CREATION
>
> repository/src/main/
Re: Review Request 52585: Implementation of the new TypesREST API
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52585/#review151761 --- The jira mentioned in this request ATLAS-1098 is a different resolved bug. Can you create a new jira and link this request there? - Shwetha GS On Oct. 6, 2016, 12:45 a.m., Apoorv Naik wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/52585/ > --- > > (Updated Oct. 6, 2016, 12:45 a.m.) > > > Review request for atlas, Madhan Neethiraj, Sarath Kumar Subramanian, Shwetha > GS, and Suma Shivaprasad. > > > Repository: atlas > > > Description > --- > > TypesREST implementation > > > Diffs > - > > .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406 > intg/src/main/java/org/apache/atlas/model/typedef/AtlasBaseTypeDef.java > a3fdbcc025bf5f56e7770015d55616ec83d2be64 > intg/src/main/java/org/apache/atlas/model/typedef/AtlasTypesDef.java > PRE-CREATION > intg/src/main/java/org/apache/atlas/type/AtlasStructType.java > 096f061f16bba3ecb886d00ec66c98ad17c1e243 > repository/pom.xml 7a34331ce2876d9d9e916f96f1141d3efa91e959 > > repository/src/main/java/org/apache/atlas/repository/graph/FullTextMapper.java > b342e2700d454b0d6fba595b5cc01cd0e06bbdac > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java > 263ea465fda0b445a952943def9a6f7c49834f25 > repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java > 1ce87c9d306faa43fb9d3fdc491c4bcbdd7b2bdb > > repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java > 5c7cb2e8fa32b540f80beed40fb4f25a89d39c56 > > repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java > 2e0414e2cee7ca3d5958650ac6abc8a290473545 > > repository/src/main/java/org/apache/atlas/repository/graph/mapper/AtlasClassificationDefMapper.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/graph/mapper/AtlasEntityDefMapper.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/graph/mapper/AtlasEnumDefMapper.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/graph/mapper/AtlasStructDefMapper.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/graph/mapper/BaseTypeDefMapper.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/graph/mapper/factory/AtlasMapperFactory.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/graph/mapper/impl/AtlasClassificationDefMapperV1.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/graph/mapper/impl/AtlasEntityDefMapperV1.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/graph/mapper/impl/AtlasEnumDefMapperV1.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/graph/mapper/impl/AtlasStructDefMapperV1.java > PRE-CREATION > repository/src/main/java/org/apache/atlas/repository/store/AtlasStore.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/instance/AbstractInstanceStore.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/instance/AtlasInstanceStore.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/instance/ClassificationStore.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/instance/EntityStore.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/instance/StructStore.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/typedef/AbstractAtlasStore.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/typedef/AtlasTypeDefStore.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/typedef/ClassificationDefStore.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/typedef/EntityDefStore.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/typedef/EnumDefStore.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/typedef/StructDefStore.java > PRE-CREATION > webapp/pom.xml 8fe4b9bed7efefdcc623df2f8e9cfa6ef84d8
Re: Review Request 52533: patch file to add "position" attribute to hive_column type
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52533/#review151632 --- addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java (line 232) <https://reviews.apache.org/r/52533/#comment220072> This is a minor backward compatible change. We should set it to 1.1 - Shwetha GS On Oct. 4, 2016, 11:17 p.m., Sarath Kumar Subramanian wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/52533/ > --- > > (Updated Oct. 4, 2016, 11:17 p.m.) > > > Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad. > > > Bugs: ATLAS-1210 > https://issues.apache.org/jira/browse/ATLAS-1210 > > > Repository: atlas > > > Description > --- > > for clients who have not updated to latest code, this patch adds the > "position" attribute to hive_column type and updates version to 2.0 > > > Diffs > - > > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java > 28078f4 > distro/src/main/assemblies/standalone-package.xml ff7b784 > > repository/src/main/java/org/apache/atlas/services/ReservedTypesRegistrar.java > c269c3a > src/main/patches/001-hive_column_add_position.json PRE-CREATION > > Diff: https://reviews.apache.org/r/52533/diff/ > > > Testing > --- > > on older builds - updates the type by adding new attribute and updates > version to 2.0 > new builds - ignore the patch since version is already in 2.0 > > > Thanks, > > Sarath Kumar Subramanian > >
Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47810/#review151486 --- graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java (line 65) <https://reviews.apache.org/r/47810/#comment219901> This has to be synchronized - Shwetha GS On Oct. 4, 2016, 1:28 a.m., Jeff Hagelberg wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/47810/ > --- > > (Updated Oct. 4, 2016, 1:28 a.m.) > > > Review request for atlas, David Kantor and Neeru Gupta. > > > Bugs: ATLAS-694 > https://issues.apache.org/jira/browse/ATLAS-694 > > > Repository: atlas > > > Description > --- > > ATLAS-694: Update Atlas to use abstraction layer. All of the Atlas code > (with the exception of the catalog, which was only updated minimally) has > been updated to use the graph database abstraction layer. In addition, there > is now an optional Atlas configuration property that specifies the class of > the abstraction layer database to use. I basically put all of the changes in > here with the exception of the actual Titan 1 implementation of code. This > includes the changes to support Tinkerpop 3 syntax. This is mostly to > expedite getting the changes into Atlas. Originally the TP3 changes were > going to be brought in as part of the Titan 1 implementation task. > > Change Summary: > >- change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc > instead of TitanGraph/Vertex/Edge, etc >- compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, > which was only changed to use AtlasGraphProvider/AtlasGraph) - see > repository\pom.xml, other pom.xmls >- updated DSL translation to generate Gremlin that is compliant with TP3 > when TP3 is being used. See GremlinQuery.scala, > GraphPersistenceStrategies.scala >- TitanGraphProvider replaced by AtlasGraphProvider. Graph database > implementation is determined from a new optional configuration property >- HiveTitanSample is no longer used by tests. It has been replaced by > hive-instances.json (which uses normal Atlas json syntax). The data is saved > with a new JSONImporter class. This was needed because the graphson syntax > used by HiveTitanSample is not compatible with TP3. > > Last rebase: 10/3/2016 > > > Diffs > - > > .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406 > addons/falcon-bridge/pom.xml 1365bd05a388dc92f7a56c7f7427b5b85f97c7da > addons/hdfs-model/pom.xml 492f39cea085c6e69781e17bcbdbc3a231806df3 > > addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java > ac60294e328835ba0340e150799ddfb348ccdb52 > addons/hive-bridge/pom.xml 6993bdb938a6095ca24482e290393eeeb3911bcb > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java > ad7a4a5d09d8542a841701dfe04981f65f767c14 > addons/sqoop-bridge/pom.xml 8c9d278d43b5979ea1743d10845905c13249f8a6 > addons/storm-bridge/pom.xml 12c1208b448d456a923bd7309601174ddb561ba5 > catalog/pom.xml 2f58a8f0748de65ab78eab35df6abd2fe7c336af > catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java > e7bb505075983371ca12d9bc1d8c6eb240c3d134 > distro/src/conf/atlas-application.properties > d334600dc5534840409b586157799ef3abf9abf2 > > graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java > dd4b7e614cdd9bf30f957fb6a839d8c60f3e1701 > > graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java > 1bc0fc38c0802897f32260520770a16795474d04 > > graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java > 995c5457ac7f807172f367cc8e3348b3a98dd6f3 > > graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java > 41194d34f079842db0d95c73a8b099459f76ff2f > > graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java > c8cd2842ca3090b6bbd384c773b4eb45aff149ce > > graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java > 93447495bcf18e9f19df9df68fd1cbe1427fc462 > > graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java > e719d306ffe9f68e3ac6f7406baaf60a12390c34 > > graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java > 315ecddb861e1a1be6e0ab9b36fe4c0a52486ae8 > > graphdb/api/src/main
Re: Review Request 51896: ATLAS-1171: structured, high-level APIs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51896/#review151128 --- Ship it! Ship It! - Shwetha GS On Oct. 2, 2016, 5:55 a.m., Madhan Neethiraj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51896/ > --- > > (Updated Oct. 2, 2016, 5:55 a.m.) > > > Review request for atlas, Abhay Kulkarni and Selvamohan Neethiraj. > > > Bugs: ATLAS-1171 > https://issues.apache.org/jira/browse/ATLAS-1171 > > > Repository: atlas > > > Description > --- > > ATLAS-1171: structured, high-level APIs - phase #1 > - define data-strctures for use in the interface (REST, ..) to describe types > and instances: in package org.apache.atlas.model > - implementation of runtime behaviour of types - create/validate/normalize > instances (i.e. from values in REST to typed-values): in package > org.apache.atlas.type > > I think phase #1 in providing high-level APIs is now ready to review and > commit. Please note this patch only includes new files and there are no > changes to any existing code. Please review. > > Next steps: > - Phase #2 will include the REST API and the modules that deal with > persistence of type and instance objects (defined in phase #1) in Tital Graph > database. > - Phase #3 will replace current REST API implementation to use the new APIs, > while continuing to provide the current REST API. > - GUI updates to use the new API can begin once phase #2 goes in. > > > Diffs > - > > intg/pom.xml PRE-CREATION > intg/src/main/java/org/apache/atlas/api/AtlasApiEntities.java PRE-CREATION > intg/src/main/java/org/apache/atlas/api/AtlasApiTypes.java PRE-CREATION > intg/src/main/java/org/apache/atlas/api/PList.java PRE-CREATION > intg/src/main/java/org/apache/atlas/api/SearchFilter.java PRE-CREATION > intg/src/main/java/org/apache/atlas/exception/AtlasBaseException.java > PRE-CREATION > intg/src/main/java/org/apache/atlas/model/instance/AtlasClassification.java > PRE-CREATION > intg/src/main/java/org/apache/atlas/model/instance/AtlasEntity.java > PRE-CREATION > intg/src/main/java/org/apache/atlas/model/instance/AtlasObjectId.java > PRE-CREATION > intg/src/main/java/org/apache/atlas/model/instance/AtlasStruct.java > PRE-CREATION > intg/src/main/java/org/apache/atlas/model/typedef/AtlasBaseTypeDef.java > PRE-CREATION > > intg/src/main/java/org/apache/atlas/model/typedef/AtlasClassificationDef.java > PRE-CREATION > intg/src/main/java/org/apache/atlas/model/typedef/AtlasEntityDef.java > PRE-CREATION > intg/src/main/java/org/apache/atlas/model/typedef/AtlasEnumDef.java > PRE-CREATION > intg/src/main/java/org/apache/atlas/model/typedef/AtlasStructDef.java > PRE-CREATION > intg/src/main/java/org/apache/atlas/type/AtlasArrayType.java PRE-CREATION > intg/src/main/java/org/apache/atlas/type/AtlasBuiltInTypes.java > PRE-CREATION > intg/src/main/java/org/apache/atlas/type/AtlasClassificationType.java > PRE-CREATION > intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java PRE-CREATION > intg/src/main/java/org/apache/atlas/type/AtlasEnumType.java PRE-CREATION > intg/src/main/java/org/apache/atlas/type/AtlasMapType.java PRE-CREATION > intg/src/main/java/org/apache/atlas/type/AtlasStructType.java PRE-CREATION > intg/src/main/java/org/apache/atlas/type/AtlasType.java PRE-CREATION > intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java > PRE-CREATION > intg/src/test/java/org/apache/atlas/model/ModelTestUtil.java PRE-CREATION > > intg/src/test/java/org/apache/atlas/model/instance/TestAtlasClassification.java > PRE-CREATION > intg/src/test/java/org/apache/atlas/model/instance/TestAtlasEntity.java > PRE-CREATION > > intg/src/test/java/org/apache/atlas/model/typedef/TestAtlasClassificationDef.java > PRE-CREATION > intg/src/test/java/org/apache/atlas/model/typedef/TestAtlasEntityDef.java > PRE-CREATION > intg/src/test/java/org/apache/atlas/model/typedef/TestAtlasEnumDef.java > PRE-CREATION > intg/src/test/java/org/apache/atlas/model/typedef/TestAtlasStructDef.java > PRE-CREATION > intg/src/test/java/org/apache/atlas/type/TestAtlasArrayType.java > PRE-CREATION > intg/src/test/java/org/apache/atlas/type/TestAtlasBigDecimalType.java > PRE-CREATION > intg/src/test/java/org/apache/atlas/type/TestAtlasBigIntegerType.java > PRE-CREATION
Re: Review Request 52077: Column level lineage in Hive
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52077/#review150990 --- Ship it! Ship It! - Shwetha GS On Sept. 29, 2016, 8:15 a.m., Vimal Sharma wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/52077/ > --- > > (Updated Sept. 29, 2016, 8:15 a.m.) > > > Review request for atlas. > > > Bugs: ATLAS-1184 and ATLAS-247 > https://issues.apache.org/jira/browse/ATLAS-1184 > https://issues.apache.org/jira/browse/ATLAS-247 > > > Repository: atlas > > > Description > --- > > After a CTAS query, lineage relationship between source columns and > destination column can be captured. This information can be used to create a > column lineage process. > > > Diffs > - > > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/ColumnLineageUtils.java > PRE-CREATION > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java > a3464a0 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java > 45f0bc9 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataTypes.java > e094cb6 > addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java > a5838b4 > > Diff: https://reviews.apache.org/r/52077/diff/ > > > Testing > --- > > > Thanks, > > Vimal Sharma > >
Re: Review Request 51896: ATLAS-1171: structured, high-level APIs
> On Sept. 30, 2016, 5:21 a.m., Shwetha GS wrote: > > Previously, the common module contained some common code required for atlas > > server. Since the new classes that are added are client side classes, can > > you move them to new module clientv2 so that we have clear dependencies for > > atlas clients. > > > > Also, move the server side rest API to webappv2 and webapp can depend on > > webappv2 > > Madhan Neethiraj wrote: > atlas-common module is being referenced by a number of other modules - > not just Atlas server. Like: typesystem, notification, authorization, > hive-bridge, falcon-bridge, sqoop-bridge, storm-bridge. Hence I have the new > classes in atlas-common module. Type system module, for example has both client and server side code. Since the clients would anyway depend on type system module, it becomes difficult to restrict the clients to not use any server side classes in the type system module, and they will be confused about the right classes to use. Instead, if we separate out the dependencies right, it will go a long way in helping the clients to integrate easily Yes, the server side modules need to depend on this new module. If I understand your intentions right, clientv2 will be the only point of atlas reference for all clients, and will replace all existing dependencies(common, typesystem, notification etc) - Shwetha --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51896/#review150984 --- On Sept. 30, 2016, 12:04 a.m., Madhan Neethiraj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51896/ > --- > > (Updated Sept. 30, 2016, 12:04 a.m.) > > > Review request for atlas, Abhay Kulkarni and Selvamohan Neethiraj. > > > Bugs: ATLAS-1171 > https://issues.apache.org/jira/browse/ATLAS-1171 > > > Repository: atlas > > > Description > --- > > ATLAS-1171: structured, high-level APIs - phase #1 > - define data-strctures for use in the interface (REST, ..) to describe types > and instances: in package org.apache.atlas.model > - implementation of runtime behaviour of types - create/validate/normalize > instances (i.e. from values in REST to typed-values): in package > org.apache.atlas.type > > I think phase #1 in providing high-level APIs is now ready to review and > commit. Please note this patch only includes new files and there are no > changes to any existing code. Please review. > > Next steps: > - Phase #2 will include the REST API and the modules that deal with > persistence of type and instance objects (defined in phase #1) in Tital Graph > database. > - Phase #3 will replace current REST API implementation to use the new APIs, > while continuing to provide the current REST API. > - GUI updates to use the new API can begin once phase #2 goes in. > > > Diffs > - > > common/pom.xml e3b6465 > common/src/main/java/org/apache/atlas/api/AtlasApiEntities.java > PRE-CREATION > common/src/main/java/org/apache/atlas/api/AtlasApiTypes.java PRE-CREATION > common/src/main/java/org/apache/atlas/api/PList.java PRE-CREATION > common/src/main/java/org/apache/atlas/api/SearchFilter.java PRE-CREATION > > common/src/main/java/org/apache/atlas/model/instance/AtlasClassification.java > PRE-CREATION > common/src/main/java/org/apache/atlas/model/instance/AtlasEntity.java > PRE-CREATION > common/src/main/java/org/apache/atlas/model/instance/AtlasObjectId.java > PRE-CREATION > common/src/main/java/org/apache/atlas/model/instance/AtlasStruct.java > PRE-CREATION > common/src/main/java/org/apache/atlas/model/typedef/AtlasBaseTypeDef.java > PRE-CREATION > > common/src/main/java/org/apache/atlas/model/typedef/AtlasClassificationDef.java > PRE-CREATION > common/src/main/java/org/apache/atlas/model/typedef/AtlasEntityDef.java > PRE-CREATION > common/src/main/java/org/apache/atlas/model/typedef/AtlasEnumDef.java > PRE-CREATION > common/src/main/java/org/apache/atlas/model/typedef/AtlasStructDef.java > PRE-CREATION > common/src/main/java/org/apache/atlas/type/AtlasArrayType.java PRE-CREATION > common/src/main/java/org/apache/atlas/type/AtlasBuiltInTypes.java > PRE-CREATION > common/src/main/java/org/apache/atlas/type/AtlasClassificationType.java > PRE-CREATION > common/src/main/java/org/apache/atlas/type/AtlasEntityType.java > PRE-CREATION > common/src/main/java/org/apach
Re: Review Request 51896: ATLAS-1171: structured, high-level APIs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51896/#review150984 --- Previously, the common module contained some common code required for atlas server. Since the new classes that are added are client side classes, can you move them to new module clientv2 so that we have clear dependencies for atlas clients. Also, move the server side rest API to webappv2 and webapp can depend on webappv2 - Shwetha GS On Sept. 30, 2016, 12:04 a.m., Madhan Neethiraj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51896/ > --- > > (Updated Sept. 30, 2016, 12:04 a.m.) > > > Review request for atlas, Abhay Kulkarni and Selvamohan Neethiraj. > > > Bugs: ATLAS-1171 > https://issues.apache.org/jira/browse/ATLAS-1171 > > > Repository: atlas > > > Description > --- > > ATLAS-1171: structured, high-level APIs - phase #1 > - define data-strctures for use in the interface (REST, ..) to describe types > and instances: in package org.apache.atlas.model > - implementation of runtime behaviour of types - create/validate/normalize > instances (i.e. from values in REST to typed-values): in package > org.apache.atlas.type > > I think phase #1 in providing high-level APIs is now ready to review and > commit. Please note this patch only includes new files and there are no > changes to any existing code. Please review. > > Next steps: > - Phase #2 will include the REST API and the modules that deal with > persistence of type and instance objects (defined in phase #1) in Tital Graph > database. > - Phase #3 will replace current REST API implementation to use the new APIs, > while continuing to provide the current REST API. > - GUI updates to use the new API can begin once phase #2 goes in. > > > Diffs > - > > common/pom.xml e3b6465 > common/src/main/java/org/apache/atlas/api/AtlasApiEntities.java > PRE-CREATION > common/src/main/java/org/apache/atlas/api/AtlasApiTypes.java PRE-CREATION > common/src/main/java/org/apache/atlas/api/PList.java PRE-CREATION > common/src/main/java/org/apache/atlas/api/SearchFilter.java PRE-CREATION > > common/src/main/java/org/apache/atlas/model/instance/AtlasClassification.java > PRE-CREATION > common/src/main/java/org/apache/atlas/model/instance/AtlasEntity.java > PRE-CREATION > common/src/main/java/org/apache/atlas/model/instance/AtlasObjectId.java > PRE-CREATION > common/src/main/java/org/apache/atlas/model/instance/AtlasStruct.java > PRE-CREATION > common/src/main/java/org/apache/atlas/model/typedef/AtlasBaseTypeDef.java > PRE-CREATION > > common/src/main/java/org/apache/atlas/model/typedef/AtlasClassificationDef.java > PRE-CREATION > common/src/main/java/org/apache/atlas/model/typedef/AtlasEntityDef.java > PRE-CREATION > common/src/main/java/org/apache/atlas/model/typedef/AtlasEnumDef.java > PRE-CREATION > common/src/main/java/org/apache/atlas/model/typedef/AtlasStructDef.java > PRE-CREATION > common/src/main/java/org/apache/atlas/type/AtlasArrayType.java PRE-CREATION > common/src/main/java/org/apache/atlas/type/AtlasBuiltInTypes.java > PRE-CREATION > common/src/main/java/org/apache/atlas/type/AtlasClassificationType.java > PRE-CREATION > common/src/main/java/org/apache/atlas/type/AtlasEntityType.java > PRE-CREATION > common/src/main/java/org/apache/atlas/type/AtlasEnumType.java PRE-CREATION > common/src/main/java/org/apache/atlas/type/AtlasMapType.java PRE-CREATION > common/src/main/java/org/apache/atlas/type/AtlasStructType.java > PRE-CREATION > common/src/main/java/org/apache/atlas/type/AtlasType.java PRE-CREATION > common/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java > PRE-CREATION > common/src/test/java/org/apache/atlas/model/ModelTestUtil.java PRE-CREATION > > common/src/test/java/org/apache/atlas/model/instance/TestAtlasClassification.java > PRE-CREATION > common/src/test/java/org/apache/atlas/model/instance/TestAtlasEntity.java > PRE-CREATION > > common/src/test/java/org/apache/atlas/model/typedef/TestAtlasClassificationDef.java > PRE-CREATION > common/src/test/java/org/apache/atlas/model/typedef/TestAtlasEntityDef.java > PRE-CREATION > common/src/test/java/org/apache/atlas/model/typedef/TestAtlasEnumDef.java > PRE-CREATION > common/src/test/java/org/apache/atlas/model/typedef/TestAtlasStructDef.java > PRE-CRE
Re: Review Request 52077: Column level lineage in Hive
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52077/#review150821 --- Fix it, then Ship it! addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java (line 634) <https://reviews.apache.org/r/52077/#comment218885> change to warn as we continue even without it addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java (line 1049) <https://reviews.apache.org/r/52077/#comment218887> remove toString addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java (line 1051) <https://reviews.apache.org/r/52077/#comment218886> Change to warn addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java (line 1110) <https://reviews.apache.org/r/52077/#comment218891> Add a comment on why its disabled and when the test can be enabled addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java (line 1163) <https://reviews.apache.org/r/52077/#comment218892> add assert that vertices contains a_guid and b_guid addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java (line 1169) <https://reviews.apache.org/r/52077/#comment218893> Add assert that vertices contains sourceTableGUID - Shwetha GS On Sept. 26, 2016, 1:06 p.m., Vimal Sharma wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/52077/ > --- > > (Updated Sept. 26, 2016, 1:06 p.m.) > > > Review request for atlas. > > > Bugs: ATLAS-1184 and ATLAS-247 > https://issues.apache.org/jira/browse/ATLAS-1184 > https://issues.apache.org/jira/browse/ATLAS-247 > > > Repository: atlas > > > Description > --- > > After a CTAS query, lineage relationship between source columns and > destination column can be captured. This information can be used to create a > column lineage process. > > > Diffs > - > > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/ColumnLineageUtils.java > PRE-CREATION > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java > a3464a0 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java > 45f0bc9 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataTypes.java > e094cb6 > addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java > a5838b4 > > Diff: https://reviews.apache.org/r/52077/diff/ > > > Testing > --- > > > Thanks, > > Vimal Sharma > >
Re: Review Request 52336: ReservedTypesRegistrar checks for existence of 1st class type
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52336/#review150818 --- Ship it! Ship It! - Shwetha GS On Sept. 28, 2016, 9:10 a.m., Vimal Sharma wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/52336/ > --- > > (Updated Sept. 28, 2016, 9:10 a.m.) > > > Review request for atlas. > > > Bugs: ATLAS-1184 > https://issues.apache.org/jira/browse/ATLAS-1184 > > > Repository: atlas > > > Description > --- > > ReservedTypesRegistrar registers the models for supported integrations like > hive, falcon, sqoop and storm. ReservedTypesRegistrar reads the model files > and registers the types defined in the model file. The check thats done to > verify if the model is already registered - checks for existence of first > class type. This will not work if we add more class types to the existing > model or modify the existing type to add optional attributes. > ReservedTypesRegistrar should check for every type in the model file. Also, > ReservedTypesRegistrar should do type update which does create if not exists, > and update if exists > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/services/ReservedTypesRegistrar.java > 41c0155 > repository/src/test/java/org/apache/atlas/TestUtils.java bd9df62 > > repository/src/test/java/org/apache/atlas/services/ReservedTypesRegistrarTest.java > d602bd8 > > Diff: https://reviews.apache.org/r/52336/diff/ > > > Testing > --- > > > Thanks, > > Vimal Sharma > >
Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47810/#review150691 --- repository/src/test/resources/hive-instances.json (line 2) <https://reviews.apache.org/r/47810/#comment218730> Modifying this json because of changes in model, serialisation/deserialisation is tough. Can we just generate the entities from test code? - Shwetha GS On Sept. 23, 2016, 3:29 a.m., Jeff Hagelberg wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/47810/ > --- > > (Updated Sept. 23, 2016, 3:29 a.m.) > > > Review request for atlas, David Kantor and Neeru Gupta. > > > Bugs: ATLAS-694 > https://issues.apache.org/jira/browse/ATLAS-694 > > > Repository: atlas > > > Description > --- > > ATLAS-694: Update Atlas to use abstraction layer. All of the Atlas code > (with the exception of the catalog, which was only updated minimally) has > been updated to use the graph database abstraction layer. In addition, there > is now an optional Atlas configuration property that specifies the class of > the abstraction layer database to use. I basically put all of the changes in > here with the exception of the actual Titan 1 implementation of code. This > includes the changes to support Tinkerpop 3 syntax. This is mostly to > expedite getting the changes into Atlas. Originally the TP3 changes were > going to be brought in as part of the Titan 1 implementation task. > > Change Summary: > >- change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc > instead of TitanGraph/Vertex/Edge, etc >- compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, > which was only changed to use AtlasGraphProvider/AtlasGraph) - see > repository\pom.xml, other pom.xmls >- updated DSL translation to generate Gremlin that is compliant with TP3 > when TP3 is being used. See GremlinQuery.scala, > GraphPersistenceStrategies.scala >- TitanGraphProvider replaced by AtlasGraphProvider. Graph database > implementation is determined from a new optional configuration property >- HiveTitanSample is no longer used by tests. It has been replaced by > hive-instances.json (which uses normal Atlas json syntax). The data is saved > with a new JSONImporter class. This was needed because the graphson syntax > used by HiveTitanSample is not compatible with TP3. > > Last rebase: 9/22/2016 > > > Diffs > - > > .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406 > addons/falcon-bridge/pom.xml 1365bd05a388dc92f7a56c7f7427b5b85f97c7da > addons/hdfs-model/pom.xml 492f39cea085c6e69781e17bcbdbc3a231806df3 > > addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java > ac60294e328835ba0340e150799ddfb348ccdb52 > addons/hive-bridge/pom.xml 6993bdb938a6095ca24482e290393eeeb3911bcb > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java > ad7a4a5d09d8542a841701dfe04981f65f767c14 > addons/sqoop-bridge/pom.xml 8c9d278d43b5979ea1743d10845905c13249f8a6 > addons/storm-bridge/pom.xml 12c1208b448d456a923bd7309601174ddb561ba5 > catalog/pom.xml 2f58a8f0748de65ab78eab35df6abd2fe7c336af > catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java > e7bb505075983371ca12d9bc1d8c6eb240c3d134 > distro/src/conf/atlas-application.properties > d334600dc5534840409b586157799ef3abf9abf2 > > graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java > dd4b7e614cdd9bf30f957fb6a839d8c60f3e1701 > > graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java > 1bc0fc38c0802897f32260520770a16795474d04 > > graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java > 995c5457ac7f807172f367cc8e3348b3a98dd6f3 > > graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java > 41194d34f079842db0d95c73a8b099459f76ff2f > > graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java > c8cd2842ca3090b6bbd384c773b4eb45aff149ce > > graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java > 93447495bcf18e9f19df9df68fd1cbe1427fc462 > > graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java > e719d306ffe9f68e3ac6f7406baaf60a12390c34 > > graphdb/api/src/main/java/org/apache/atlas/repository/graph
Re: Review Request 52336: ReservedTypesRegistrar checks for existence of 1st class type
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52336/#review150690 --- repository/src/main/java/org/apache/atlas/services/ReservedTypesRegistrar.java (line 150) <https://reviews.apache.org/r/52336/#comment218729> We shouldn't do any updates as part of this, as it might override any type updates from user. Let the patch framework take care of updates. Lets just create new types as part of this - Shwetha GS On Sept. 28, 2016, 6:27 a.m., Vimal Sharma wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/52336/ > --- > > (Updated Sept. 28, 2016, 6:27 a.m.) > > > Review request for atlas. > > > Bugs: ATLAS-1184 > https://issues.apache.org/jira/browse/ATLAS-1184 > > > Repository: atlas > > > Description > --- > > ReservedTypesRegistrar registers the models for supported integrations like > hive, falcon, sqoop and storm. ReservedTypesRegistrar reads the model files > and registers the types defined in the model file. The check thats done to > verify if the model is already registered - checks for existence of first > class type. This will not work if we add more class types to the existing > model or modify the existing type to add optional attributes. > ReservedTypesRegistrar should check for every type in the model file. Also, > ReservedTypesRegistrar should do type update which does create if not exists, > and update if exists > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/services/ReservedTypesRegistrar.java > 41c0155 > repository/src/test/java/org/apache/atlas/TestUtils.java bd9df62 > > repository/src/test/java/org/apache/atlas/services/ReservedTypesRegistrarTest.java > d602bd8 > > Diff: https://reviews.apache.org/r/52336/diff/ > > > Testing > --- > > > Thanks, > > Vimal Sharma > >
Re: Review Request 52166: Allow optional attributes to be removed when updating type
> On Sept. 23, 2016, 6:19 a.m., Shwetha GS wrote: > > In case the attribute is added again, the data will be wrong. We can't > > discount that as an edge case, as it causes data inconsistency. Follow up > > jiras will hardly be taken up. Its fine even if we decide that we don't > > allow adding the attribute again(the deleted attribute). But we should > > address the issue completely > > > > Add UTs in all your patches. Just an end to end test for this patch now > > will not ensure that this functionality won't be broken in further commits > > Suma Shivaprasad wrote: > One sugestion to resolve this case would be > > 1. Add a isDeleted flag to the attribute. It may be useful to know that > this atribute was deleted and in case of upgrade/rollbacks , we could > potentially use this flag to resurface it back again and roll back the type > update. > 2. Rename the existing property key - > http://s3.thinkaurelius.com/docs/titan/0.5.1/schema.html#_changing_schema_elements. > Not sure how much time this operation would take. If its an expensive > operation, we would need to run this as an asynchronous thread. We can add the isDeleted flag for now and can cleanup the deleted attributes during maintenance windows like upgrades - Shwetha --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52166/#review150154 --- On Sept. 22, 2016, 6:22 p.m., Sarath Kumar Subramanian wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/52166/ > ------- > > (Updated Sept. 22, 2016, 6:22 p.m.) > > > Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad. > > > Bugs: ATLAS-1175 > https://issues.apache.org/jira/browse/ATLAS-1175 > > > Repository: atlas > > > Description > --- > > we restrict optional attributes to be removed from the type system. This > needs to be allowed > > > Diffs > - > > typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeUtils.java > f5c2ce9 > > Diff: https://reviews.apache.org/r/52166/diff/ > > > Testing > --- > > tested using REST calls to delete optional and required attributes and only > optional attributes were removed. > > > Thanks, > > Sarath Kumar Subramanian > >
Re: Review Request 52077: Column level lineage in Hive
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52077/#review150383 --- 1. Can you add a test with lineage query on column? 2. ReservedTypesRegistrar should change to updateTypes, so that upgrades work 3. Once you make sure the tests work, disable the tests so that tests don't break with apache hive 1.2 addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java (line 334) <https://reviews.apache.org/r/52077/#comment218361> rename to query/command as this class type is also process - Shwetha GS On Sept. 22, 2016, 11:53 a.m., Vimal Sharma wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/52077/ > --- > > (Updated Sept. 22, 2016, 11:53 a.m.) > > > Review request for atlas. > > > Bugs: ATLAS-247 > https://issues.apache.org/jira/browse/ATLAS-247 > > > Repository: atlas > > > Description > --- > > After a CTAS query, lineage relationship between source columns and > destination column can be captured. This information can be used to create a > column lineage process. > > > Diffs > - > > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/ColumnLineageUtils.java > PRE-CREATION > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java > a3464a0 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java > 45f0bc9 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataTypes.java > e094cb6 > addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java > a5838b4 > > Diff: https://reviews.apache.org/r/52077/diff/ > > > Testing > --- > > > Thanks, > > Vimal Sharma > >
Re: Review Request 51939: Framework to apply updates to types in the type-system
On Sept. 21, 2016, 2:46 p.m., Sarath Kumar Subramanian wrote: > > Add tests > > > > 1. Currently, the model files(like hive_model.json) are auto generated from > > model definitions defined in java(like HiveDataModelGenerator). The patch > > files in this case has to be hand coded which is error prone > > 2. For completeness, readability and debuggability, the type update has to > > be done in the corresponding model definitions like HiveDataModelGenerator. > > So, same data will be in two places and the model definitions and the patch > > files can go out of sync > > 3. Since the model definitions(like HiveDataModelGenerator) will be updated > > anyways, if we modify ReservedTypesRegistrar to do type update instead of > > type create, the type updates will automatically be taken care with the > > same model json. So, model update patches are not necessary then > > 3. This jira doesn't implement type versioning - doesn't have support for > > storing multiple versions of the type. But it maintains the version of the > > latest type definition which I think is useful for debugging, to know the > > version of type that the server knows. Can we maintain this info in > > HiveDataModelGenerator itself, and hence will be part of hive_model.json > > Sarath Kumar Subramanian wrote: > 1. patch json genarator can be addressed in a separate jiira > 2. this is only for upgrade scenarios > 3. lets address this in a separate jiira. > 4. we dont want to maintain multiple versions of the same type in the > typesystem, will confuse users on which version to use to create entities. > > Shwetha GS wrote: > What benefit does this patch framework add when the type update can be > done with the existing functionality itself (Type versioning is useful and > can be done without the patch framework). The patch framework adds more > overhead, and I don't see the necessity > > David Radley wrote: > I agree - I think this capability would be better placed in separate > tooling that uses the Atlas REST APIs. > > Madhan Neethiraj wrote: > @Shwetha, @David - the rational for the patch framework is exactly same > as the need for reading contents of "models" directory during startup and > initializing the typesystem. This just makes it easier to deal with > updates/additions to typesystem during software upgrade/patch. > > Sarath Kumar Subramanian wrote: > The existing system to update types (using REST) has these limitations: > > 1. Clients might add new attributes to a type and when we add a set of > new attributes in model.json - the rest api model overwrites customer > changes. > > 2. More granular type updates cannot be applied, if you add a new > attribute in hive_model for eg., During atlas startup it doesn't do a type > by model comparison and apply the diff. This patch can update to a specific > type based on its version - condition based update. > > David Radley wrote: > @Madhan @Sarath It seems to me that the model files - are system types > that we supply to aid integration with Hive and the like. As these do not > change, then I am happy these are in model files. If we put changes in patch > files, then patch files become the master. I think we need the Atlas > repository to be the master of all the types in all cases. It should not be > a slave to patch files. I can see a place for patch files that tooling uses > at a one off upgrade of types to update the repository. I agree that care > needs to be taken on type shape changes, in practice I would expect that > these shape change operations could be restricted to admin roles. Sarath, >>Clients might add new attributes to a type and when we add a set of new >>attributes in model.json - the rest api model overwrites customer changes Thats a valid point. The patch doesn't apply on trunk. Can you re-base the patch and add the tests? David, Registering these models/patches is part of set-up tool required for hooks to work. These use internal methods directly which we should change to make use of APIs. This set-up is required to be run before any hooks can register the entities, hence its run as part of service start itself. But I agree that we should move this out of repository. - Shwetha --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51939/#review149811 --- On Sept. 23, 2016, 3:02 a.m., Sarath Kumar Subramanian wrote: > > ------- > This is an automatically generated e-mail. To
Re: Review Request 52166: Allow optional attributes to be removed when updating type
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52166/#review150154 --- In case the attribute is added again, the data will be wrong. We can't discount that as an edge case, as it causes data inconsistency. Follow up jiras will hardly be taken up. Its fine even if we decide that we don't allow adding the attribute again(the deleted attribute). But we should address the issue completely Add UTs in all your patches. Just an end to end test for this patch now will not ensure that this functionality won't be broken in further commits - Shwetha GS On Sept. 22, 2016, 6:22 p.m., Sarath Kumar Subramanian wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/52166/ > --- > > (Updated Sept. 22, 2016, 6:22 p.m.) > > > Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad. > > > Bugs: ATLAS-1175 > https://issues.apache.org/jira/browse/ATLAS-1175 > > > Repository: atlas > > > Description > --- > > we restrict optional attributes to be removed from the type system. This > needs to be allowed > > > Diffs > - > > typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeUtils.java > f5c2ce9 > > Diff: https://reviews.apache.org/r/52166/diff/ > > > Testing > --- > > tested using REST calls to delete optional and required attributes and only > optional attributes were removed. > > > Thanks, > > Sarath Kumar Subramanian > >
Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47810/#review149947 --- catalog/pom.xml (line 67) <https://reviews.apache.org/r/47810/#comment217738> for now, catalog can only be used with Titan 0.5.4 - why? graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java (line 33) <https://reviews.apache.org/r/47810/#comment217740> There is AtlasPropertyKey.Cardinality and AtlasCardinality. Whats the difference? graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/GraphDbObjectFactory.java (line 36) <https://reviews.apache.org/r/47810/#comment217743> The javadoc params are out of date graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0DatabaseManager.java (line 40) <https://reviews.apache.org/r/47810/#comment217747> Rename to Titan0GraphManagement? In general, rename Database to Graph for all new classes? pom.xml (line 457) <https://reviews.apache.org/r/47810/#comment217728> where is this used? git grep returns just this result repository/src/main/java/org/apache/atlas/discovery/graph/DefaultGraphPersistenceStrategy.java (line 148) <https://reviews.apache.org/r/47810/#comment217764> repository/src/main/java/org/apache/atlas/repository/graph/AtlasGraphProvider.java (line 45) <https://reviews.apache.org/r/47810/#comment21> You will need synchronized locking here - http://www.cs.umd.edu/~pugh/java/memoryModel/DoubleCheckedLocking.html repository/src/main/java/org/apache/atlas/repository/graph/AtlasGraphProvider.java (line 57) <https://reviews.apache.org/r/47810/#comment217774> add visible for testing annotation. Will be good if you can move to some test class instead repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java (line 82) <https://reviews.apache.org/r/47810/#comment217778> add visiblefortesting annotation repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java (line 414) <https://reviews.apache.org/r/47810/#comment217783> Isn't this also in Titan0Vertex? Enable checkstyle on new modules with this property in pom.xml - true With patch: Running org.apache.atlas.repository.graph.GraphRepoMapperScaleTest Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 113.27 sec - in org.apache.atlas.repository.graph.GraphRepoMapperScaleTest Without patch: Running org.apache.atlas.repository.graph.GraphRepoMapperScaleTest Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 89.008 sec - in org.apache.atlas.repository.graph.GraphRepoMapperScaleTest With the patch, the test takes longer to run. This implies some performance issue with the patch Whats the plan on including titan 1.x implementation - will it be a profile at compile time such that only one implementation(titan 0.x or titan 1.x) will be part of the package. Or are both implementations part of the package and the implementation is chosen from the runtime configuration in application properties? If its compile time binding, then we shouldn't expose the conf atlas.graphdb.backend? - Shwetha GS On Sept. 21, 2016, 1:49 p.m., Jeff Hagelberg wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/47810/ > --- > > (Updated Sept. 21, 2016, 1:49 p.m.) > > > Review request for atlas, David Kantor and Neeru Gupta. > > > Bugs: ATLAS-694 > https://issues.apache.org/jira/browse/ATLAS-694 > > > Repository: atlas > > > Description > --- > > ATLAS-694: Update Atlas to use abstraction layer. All of the Atlas code > (with the exception of the catalog, which was only updated minimally) has > been updated to use the graph database abstraction layer. In addition, there > is now an optional Atlas configuration property that specifies the class of > the abstraction layer database to use. I basically put all of the changes in > here with the exception of the actual Titan 1 implementation of code. This > includes the changes to support Tinkerpop 3 syntax. This is mostly to > expedite getting the changes into Atlas. Originally the TP3 changes were > going to be brought in as part of the Titan 1 implementation task. > > Change Summary: > >- change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc > instead of TitanGraph/Vertex/Edge, etc >- compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, > which was only changed to use AtlasGraphProvider/AtlasGraph
Re: Review Request 51939: Framework to apply updates to types in the type-system
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149811
---
repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java
(line 42)
<https://reviews.apache.org/r/51939/#comment217544>
This class works on jsons directly which is error prone. Use POJOs instead
- HierarchicalTypeDefinition, StructTypeDefinition
repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java
(line 154)
<https://reviews.apache.org/r/51939/#comment217537>
== doesn't work on strings
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
(line 177)
<https://reviews.apache.org/r/51939/#comment217543>
Move this to AtlasTypeAttributePatch
Add tests
1. Currently, the model files(like hive_model.json) are auto generated from
model definitions defined in java(like HiveDataModelGenerator). The patch files
in this case has to be hand coded which is error prone
2. For completeness, readability and debuggability, the type update has to be
done in the corresponding model definitions like HiveDataModelGenerator. So,
same data will be in two places and the model definitions and the patch files
can go out of sync
3. Since the model definitions(like HiveDataModelGenerator) will be updated
anyways, if we modify ReservedTypesRegistrar to do type update instead of type
create, the type updates will automatically be taken care with the same model
json. So, model update patches are not necessary then
3. This jira doesn't implement type versioning - doesn't have support for
storing multiple versions of the type. But it maintains the version of the
latest type definition which I think is useful for debugging, to know the
version of type that the server knows. Can we maintain this info in
HiveDataModelGenerator itself, and hence will be part of hive_model.json
- Shwetha GS
On Sept. 21, 2016, 7:51 a.m., Sarath Kumar Subramanian wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
>
> (Updated Sept. 21, 2016, 7:51 a.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
>
>
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
>
>
> Repository: atlas
>
>
> Description
> ---
>
> 1. Introduce "version" attribute to all types in the type-system, this helps
> to track changes made to the default types (hive, sqoop, falcon and storm
> types) and user created types. If version is not mentioned during creation of
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for
> type system. This framework applies patches to a type using its version
> number and can be used during upgrade - add new attributes to an existing
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> {
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional",
> "isComposite": false, "isUnique": false, "isIndexable": false,
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version
> number and after applying the patch, update the version to the one mentioned
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
>
>
> Diffs
> -
>
> common/src/main/java/org/apache/atlas/AtlasConstants.java 17ffbd7
> common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89
>
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
> a94d157
>
> repository/src/main/java/org/apache/atlas/services/AtlasType
Re: Review Request 51939: Framework to apply updates to types in the type-system
> On Sept. 20, 2016, 1:11 p.m., David Radley wrote:
> > It looks like this is a way of updating the versions of types using a local
> > file on the Atlas server. I think we should be SaaS and expose this sort of
> > functionality primarily as a REST API, otherwise this could become a
> > barrier to integration.
>
> Sarath Kumar Subramanian wrote:
> David, the goal of this framework is to apply patches during upgrade
> scenarios - when user has changed the shape of a type (added new attributes,
> new supertypes) or to update an existing type. The patch files will be
> checked and applied during atlas startup, this will avoid dealing with
> multiple patch scripts for upgrading multiple types. Currrently a type can be
> updated using REST (PUT - /api/atlas/types) by including the type definition
> in request payload. We can add only optional attributes, so instances
> depending on old type will not be broken.
>
> I can add a dedicated REST layer to this once we have ATLAS-1171 ready.
>
> David Radley wrote:
> Hi Sumar, Thank you for explaining. I assume the patch files are used
> once for the upgrade. This is a batch orientated way of changing the shapes
> of types - which I can see could be of use in the short term, especially due
> to the exising REST restrictions . My concern is that this approach, which
> introduces versioning, might make introducing the SaaS runtime REST approach
> more difficult. Also if we put effort into a batch process that we use to
> upgrade type shapes - we may not put effort into the REST API that products
> integrating with Atlas will use at runtime. I assume the REST Implmentaiton
> would be to relax the restriction to only allow updates of optional
> attributes and to add the concept of versioning to the API. Are you planning
> to do a dedicated REST implementation soon? all the best, David.
Type update is already available through REST API. The API takes the complete
new type definition and automatically figures out what has changed and
validates the change
- Shwetha
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149652
---
On Sept. 21, 2016, 7:51 a.m., Sarath Kumar Subramanian wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> -------
>
> (Updated Sept. 21, 2016, 7:51 a.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
>
>
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
>
>
> Repository: atlas
>
>
> Description
> ---
>
> 1. Introduce "version" attribute to all types in the type-system, this helps
> to track changes made to the default types (hive, sqoop, falcon and storm
> types) and user created types. If version is not mentioned during creation of
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for
> type system. This framework applies patches to a type using its version
> number and can be used during upgrade - add new attributes to an existing
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> {
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional",
> "isComposite": false, "isUnique": false, "isIndexable": false,
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version
> number and after applying the patch, update the version to the one mentioned
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
>
>
> Diffs
&
Re: Review Request 52077: Column level lineage in Hive
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52077/#review149804
---
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/ColumnLineageUtils.java
(line 73)
<https://reviews.apache.org/r/52077/#comment217525>
Don't add toString() to arguments
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java (line
634)
<https://reviews.apache.org/r/52077/#comment217532>
Why have you changed Entity update request to entity create request? It
should be update
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java (line
819)
<https://reviews.apache.org/r/52077/#comment217533>
e.key() is the output column name? e.key doesn't contain cluster name, so
might get de-duped across other lineage processes.
Qualified name for lineage process should be
'qualified name of hive command process : column name'.
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java (line
1044)
<https://reviews.apache.org/r/52077/#comment217522>
Use syntax: LOG.debug("Column Lineage Map - {}",
this.lineageInfo.entrySet()); so that toString() is not evaluated if debug log
level is not enabled
addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java
(line 1121)
<https://reviews.apache.org/r/52077/#comment217523>
rename to testColumnLevelLineage
addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java
(line 1143)
<https://reviews.apache.org/r/52077/#comment217524>
Also assert on lineage API response on columns
- Shwetha GS
On Sept. 20, 2016, 9:07 a.m., Vimal Sharma wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52077/
> ---
>
> (Updated Sept. 20, 2016, 9:07 a.m.)
>
>
> Review request for atlas.
>
>
> Bugs: ATLAS-247
> https://issues.apache.org/jira/browse/ATLAS-247
>
>
> Repository: atlas
>
>
> Description
> ---
>
> After a CTAS query, lineage relationship between source columns and
> destination column can be captured. This information can be used to create a
> column lineage process.
>
>
> Diffs
> -
>
>
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/ColumnLineageUtils.java
> PRE-CREATION
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
> a3464a0
>
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
> 45f0bc9
>
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataTypes.java
> e094cb6
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java
> a5838b4
>
> Diff: https://reviews.apache.org/r/52077/diff/
>
>
> Testing
> ---
>
>
> Thanks,
>
> Vimal Sharma
>
>
Re: Review Request 51800: Support getTrait() API
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51800/#review149429
---
Fix it, then Ship it!
webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java (line
657)
<https://reviews.apache.org/r/51800/#comment217047>
Should be:
Referenceable entity =
InstanceSerialization.fromJsonReferenceable(entityDefinition, true);
JSONArray traits = new JSONArray();
for (String traitName : entity.getTraits()) {
IStruct trait = entity.getTrait(traitName);
traits.put(new
JSONObject(InstanceSerialization.toJson(trait, true)));
}
- Shwetha GS
On Sept. 16, 2016, 8:49 a.m., Vimal Sharma wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51800/
> ---
>
> (Updated Sept. 16, 2016, 8:49 a.m.)
>
>
> Review request for atlas.
>
>
> Repository: atlas
>
>
> Description
> ---
>
> Given entity id and trait name, support rest API that returns the trait
> instance for the entity. Currently, the other way of getting it is
> getEntity() which returns full entity with all trait instances
>
>
> Diffs
> -
>
> client/src/main/java/org/apache/atlas/AtlasClient.java 5ed79bc
>
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
> 6a937f4
>
> repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
> 52dcfde
> server-api/src/main/java/org/apache/atlas/services/MetadataService.java
> 60d8790
> webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java
> 82016d0
>
> webapp/src/test/java/org/apache/atlas/web/resources/EntityJerseyResourceIT.java
> a1988ef
>
> Diff: https://reviews.apache.org/r/51800/diff/
>
>
> Testing
> ---
>
>
> File Attachments
>
>
> ATLAS-712-v2.patch
>
> https://reviews.apache.org/media/uploaded/files/2016/09/13/b5079426-19d0-48f2-88f7-08e4e645bb32__ATLAS-712-v2.patch
>
>
> Thanks,
>
> Vimal Sharma
>
>
Re: Review Request 51800: Support getTrait() API
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51800/#review149034 --- client/src/main/java/org/apache/atlas/AtlasClient.java (line 1000) <https://reviews.apache.org/r/51800/#comment216570> Return list client/src/main/java/org/apache/atlas/AtlasClient.java (line 1012) <https://reviews.apache.org/r/51800/#comment216571> Return org.apache.atlas.typesystem.Struct webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java (line 658) <https://reviews.apache.org/r/51800/#comment216572> metadataService.getTraitDefinition() loads the whole entity everytime for every trait of the entity. This is in-efficient. Use getEntityDefinition() once and read all the traits from entity definition You can also change metadataService.getTraitDefinition() to return org.apache.atlas.typesystem.Struct - Shwetha GS On Sept. 15, 2016, 6:53 a.m., Vimal Sharma wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51800/ > --- > > (Updated Sept. 15, 2016, 6:53 a.m.) > > > Review request for atlas. > > > Repository: atlas > > > Description > --- > > Given entity id and trait name, support rest API that returns the trait > instance for the entity. Currently, the other way of getting it is > getEntity() which returns full entity with all trait instances > > > Diffs > - > > client/src/main/java/org/apache/atlas/AtlasClient.java 5ed79bc > webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java > 82016d0 > > webapp/src/test/java/org/apache/atlas/web/resources/EntityJerseyResourceIT.java > a1988ef > > Diff: https://reviews.apache.org/r/51800/diff/ > > > Testing > --- > > > File Attachments > > > ATLAS-712-v2.patch > > https://reviews.apache.org/media/uploaded/files/2016/09/13/b5079426-19d0-48f2-88f7-08e4e645bb32__ATLAS-712-v2.patch > > > Thanks, > > Vimal Sharma > >
Re: Review Request 51660: [ATLAS-1098] Added keyword checks while creating a new tag/trait
> On Sept. 13, 2016, 4:40 a.m., Shwetha GS wrote: > > repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java, > > line 309 > > <https://reviews.apache.org/r/51660/diff/3/?file=1492519#file1492519line309> > > > > This is equals check - typename equals keyword. Does it work if > > typename contains keyword e.g., if typename is 'isa PII'? > > > > Not sure if keywords in DSL is case sensitive. You may want to verify > > Apoorv Naik wrote: > Right now this will go through. > > Apoorv Naik wrote: > I've tried changing the case but seems like the dsl is not case sensitive > for those keywords > > Apoorv Naik wrote: > While testing, I see that if the keyword has any prefix or suffix > separated by a space. The query parser doesn't break. Is that what you're > looking for ? > > Shwetha GS wrote: > You mean DSL is case sensitive for the keywords, so the check need not be > case insensitive? > > Weird that the query "Table isa `isa PII`" works, but "Table isa `isa`" > doesn't work. As David pointed out, its good if we don't add restrictions on > typenames. Can you check queryparser to see if its simple enough fix? The queries have ` around the typenames with keywords. Formatting removed them - Shwetha --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51660/#review148651 --- On Sept. 6, 2016, 11:27 p.m., Apoorv Naik wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51660/ > ------- > > (Updated Sept. 6, 2016, 11:27 p.m.) > > > Review request for atlas, Madhan Neethiraj, Shwetha GS, Suma Shivaprasad, and > Vimal Sharma. > > > Repository: atlas > > > Description > --- > > [ATLAS-1098] Added keyword checks while creating a new tag/trait > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java > 4d05d49 > > Diff: https://reviews.apache.org/r/51660/diff/ > > > Testing > --- > > Tested with all possible keywords specified in the QueryParser, the UI shows > a red exception message stating that the type being created is a keyword > > > Thanks, > > Apoorv Naik > >
Re: Review Request 51660: [ATLAS-1098] Added keyword checks while creating a new tag/trait
> On Sept. 13, 2016, 4:40 a.m., Shwetha GS wrote: > > repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java, > > line 309 > > <https://reviews.apache.org/r/51660/diff/3/?file=1492519#file1492519line309> > > > > This is equals check - typename equals keyword. Does it work if > > typename contains keyword e.g., if typename is 'isa PII'? > > > > Not sure if keywords in DSL is case sensitive. You may want to verify > > Apoorv Naik wrote: > Right now this will go through. > > Apoorv Naik wrote: > I've tried changing the case but seems like the dsl is not case sensitive > for those keywords > > Apoorv Naik wrote: > While testing, I see that if the keyword has any prefix or suffix > separated by a space. The query parser doesn't break. Is that what you're > looking for ? You mean DSL is case sensitive for the keywords, so the check need not be case insensitive? Weird that the query "Table isa `isa PII`" works, but "Table isa `isa`" doesn't work. As David pointed out, its good if we don't add restrictions on typenames. Can you check queryparser to see if its simple enough fix? - Shwetha --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51660/#review148651 --- On Sept. 6, 2016, 11:27 p.m., Apoorv Naik wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51660/ > ------- > > (Updated Sept. 6, 2016, 11:27 p.m.) > > > Review request for atlas, Madhan Neethiraj, Shwetha GS, Suma Shivaprasad, and > Vimal Sharma. > > > Repository: atlas > > > Description > --- > > [ATLAS-1098] Added keyword checks while creating a new tag/trait > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java > 4d05d49 > > Diff: https://reviews.apache.org/r/51660/diff/ > > > Testing > --- > > Tested with all possible keywords specified in the QueryParser, the UI shows > a red exception message stating that the type being created is a keyword > > > Thanks, > > Apoorv Naik > >
Re: Review Request 51515: ATLAS-772 : New attribute has been added to the hive_column type to capture the position of hive columns for sorting
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51515/#review148723 --- addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java (line 588) <https://reviews.apache.org/r/51515/#comment216225> Can you add tests that column position is set correctly in create table and alter table column reorder? You can add extra validations in existing tests as well - Shwetha GS On Aug. 30, 2016, 6:20 a.m., Sarath Kumar Subramanian wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51515/ > --- > > (Updated Aug. 30, 2016, 6:20 a.m.) > > > Review request for atlas, Madhan Neethiraj and Suma Shivaprasad. > > > Bugs: ATLAS-772 > https://issues.apache.org/jira/browse/ATLAS-772 > > > Repository: atlas > > > Description > --- > > When a schema query is issued against a hive table, the hive_column entities > are retrieved from the titan database. Each hive_column entity is a vertex in > titan and currently titan doesn't store any column order position in its > vertex attributes. The hive_columns are retrieved in the sort order of the > Vertex ID which is not same as the Hive column order position. The fix > attached is to introduce a new "position" attribute in hive_column type, > which maintains the column order position. This column order position can be > consumed by any clients - including UI to display the columns in the original > position/order. > > > Diffs > - > > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java > 1f13d74 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java > b308cc9 > > Diff: https://reviews.apache.org/r/51515/diff/ > > > Testing > --- > > Tested the changes with hive column operations – Add new column, Drop column, > Rename column, Move column to first or at a position. > > > Thanks, > > Sarath Kumar Subramanian > >
Re: Review Request 51660: [ATLAS-1098] Added keyword checks while creating a new tag/trait
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51660/#review148651 --- repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java (line 309) <https://reviews.apache.org/r/51660/#comment216152> This is equals check - typename equals keyword. Does it work if typename contains keyword e.g., if typename is 'isa PII'? Not sure if keywords in DSL is case sensitive. You may want to verify - Shwetha GS On Sept. 6, 2016, 11:27 p.m., Apoorv Naik wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51660/ > --- > > (Updated Sept. 6, 2016, 11:27 p.m.) > > > Review request for atlas, Madhan Neethiraj, Shwetha GS, Suma Shivaprasad, and > Vimal Sharma. > > > Repository: atlas > > > Description > --- > > [ATLAS-1098] Added keyword checks while creating a new tag/trait > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java > 4d05d49 > > Diff: https://reviews.apache.org/r/51660/diff/ > > > Testing > --- > > Tested with all possible keywords specified in the QueryParser, the UI shows > a red exception message stating that the type being created is a keyword > > > Thanks, > > Apoorv Naik > >
Re: Review Request 51660: [ATLAS-1098] Added keyword checks while creating a new tag/trait
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51660/#review148465 --- repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java (lines 246 - 247) <https://reviews.apache.org/r/51660/#comment215918> Can you also add tests for this? repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java (line 295) <https://reviews.apache.org/r/51660/#comment215917> TypeSystem.verifyTypes() has all the validations. Please move this logic there. - Shwetha GS On Sept. 6, 2016, 11:27 p.m., Apoorv Naik wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51660/ > --- > > (Updated Sept. 6, 2016, 11:27 p.m.) > > > Review request for atlas, Madhan Neethiraj, Shwetha GS, Suma Shivaprasad, and > Vimal Sharma. > > > Repository: atlas > > > Description > --- > > [ATLAS-1098] Added keyword checks while creating a new tag/trait > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java > 4d05d49 > > Diff: https://reviews.apache.org/r/51660/diff/ > > > Testing > --- > > Tested with all possible keywords specified in the QueryParser, the UI shows > a red exception message stating that the type being created is a keyword > > > Thanks, > > Apoorv Naik > >
Re: Review Request 51035: ATLAS-1111 Data loss is observed when atlas is restarted while hive_table metadata ingestion into kafka topic is in-progress
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51035/ --- (Updated Aug. 12, 2016, noon) Review request for atlas. Changes --- Added prefix for failed messages have renamed KafkaNotificationTest to KafkaNotificationMockTest and added KafkaNotificationTest. But the diff looks big for KafkaNotificationTest Bugs: ATLAS- https://issues.apache.org/jira/browse/ATLAS- Repository: atlas Description --- kafkaconsumer.next() always returns the next message irrespective of auto commit or not. With auto commit disabled, the the offset has to be committed manually, as opposed to auto commit on next(). The last saved offset is used across consumer restarts. During graceful shutdown, because of graph shutdown, all the message processing failed and we did commits even in case of failures. So, after restart, atlas started at last committed offset and ignored message failures during shutdown and hence lost messages. The patch has the following changes: 1. No commit in case of message failure 2. Failed messages are written to different log file Diffs (updated) - distro/src/conf/atlas-log4j.xml eaa4ec5 notification/src/main/java/org/apache/atlas/kafka/KafkaConsumer.java 270215d notification/src/main/java/org/apache/atlas/notification/NotificationConsumer.java 2e861cb notification/src/main/java/org/apache/atlas/notification/hook/HookNotification.java 88a0322 notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java PRE-CREATION notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 2a49634 notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java 0c8990f plugin-classloader/src/main/java/org/apache/atlas/plugin/classloader/AtlasPluginClassLoader.java 92cc406 typesystem/src/test/resources/atlas-application.properties a3b6c90 webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 901b1ed webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java 0a7c5df webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java 6fd1939 webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java d22c5f1 Diff: https://reviews.apache.org/r/51035/diff/ Testing --- UTs and end to end test Thanks, Shwetha GS
Review Request 51035: ATLAS-1111 Data loss is observed when atlas is restarted while hive_table metadata ingestion into kafka topic is in-progress
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51035/ --- Review request for atlas. Bugs: ATLAS- https://issues.apache.org/jira/browse/ATLAS- Repository: atlas Description --- kafkaconsumer.next() always returns the next message irrespective of auto commit or not. With auto commit disabled, the the offset has to be committed manually, as opposed to auto commit on next(). The last saved offset is used across consumer restarts. During graceful shutdown, because of graph shutdown, all the message processing failed and we did commits even in case of failures. So, after restart, atlas started at last committed offset and ignored message failures during shutdown and hence lost messages. The patch has the following changes: 1. No commit in case of message failure 2. Failed messages are written to different log file Diffs - distro/src/conf/atlas-log4j.xml eaa4ec5 notification/src/main/java/org/apache/atlas/kafka/KafkaConsumer.java 270215d notification/src/main/java/org/apache/atlas/notification/NotificationConsumer.java 2e861cb notification/src/main/java/org/apache/atlas/notification/hook/HookNotification.java 88a0322 notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java PRE-CREATION notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 2a49634 notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java 0c8990f plugin-classloader/src/main/java/org/apache/atlas/plugin/classloader/AtlasPluginClassLoader.java 92cc406 typesystem/src/test/resources/atlas-application.properties a3b6c90 webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 901b1ed webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java 0a7c5df webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java 6fd1939 webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java d22c5f1 Diff: https://reviews.apache.org/r/51035/diff/ Testing --- UTs and end to end test Thanks, Shwetha GS
Re: Review Request 50824: ATLAS-1092 Add hive table's CreateTime to process qualified Name for all hive_process
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50824/#review144880 --- Ship it! Ship It! - Shwetha GS On Aug. 4, 2016, 11:13 p.m., Suma Shivaprasad wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/50824/ > --- > > (Updated Aug. 4, 2016, 11:13 p.m.) > > > Review request for atlas. > > > Bugs: ATLAS-1092 > https://issues.apache.org/jira/browse/ATLAS-1092 > > > Repository: atlas > > > Description > --- > > Existing tests cover all cases > > > Diffs > - > > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java > 8d24a67 > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java > 40e8c5f > addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java > e61e916 > > Diff: https://reviews.apache.org/r/50824/diff/ > > > Testing > --- > > > Thanks, > > Suma Shivaprasad > >
Re: Review Request 50823: ATLAS-1096 HiveMetStoreBridge to use getEntity instead of DSL
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50823/#review144873 --- Fix it, then Ship it! will address these comments and commit addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java (line 154) <https://reviews.apache.org/r/50823/#comment211003> shouldn't ignore all exceptions, ignore only if 'not found' status addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java (line 100) <https://reviews.apache.org/r/50823/#comment211008> Add an IT in HiveMetastoreBridgeIT which tests de-duping - Shwetha GS On Aug. 5, 2016, 4:29 a.m., Suma Shivaprasad wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/50823/ > --- > > (Updated Aug. 5, 2016, 4:29 a.m.) > > > Review request for atlas. > > > Bugs: ATLAS-1096 > https://issues.apache.org/jira/browse/ATLAS-1096 > > > Repository: atlas > > > Description > --- > > Changed all searchDSL calls to getEntity by qualifiedName and typeName which > will use indexes. > Also fixed a minor issue with table qualifiedName in case of importTable > > > Diffs > - > > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java > 8d24a67 > > addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java > f8aa93a > > Diff: https://reviews.apache.org/r/50823/diff/ > > > Testing > --- > > > Thanks, > > Suma Shivaprasad > >
Re: Review Request 49869: ATLAS-991: avoid enforcing required attributes on deleted entities
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49869/#review144724 --- repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java (line 442) <https://reviews.apache.org/r/49869/#comment210793> This has to be a set to make sure there are no duplicates. The loop later can exit if already added to set. Instead of adding multiple data structures here, just stack and set will do? repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java (line 446) <https://reviews.apache.org/r/49869/#comment210788> Why two levels of loop - use stack instead repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java (line 503) <https://reviews.apache.org/r/49869/#comment210787> should be mapEdgeLabel repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java (line 886) <https://reviews.apache.org/r/49869/#comment210794> Add tests with lower bounds on map as well - Shwetha GS On July 21, 2016, 12:28 a.m., David Kantor wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/49869/ > --- > > (Updated July 21, 2016, 12:28 a.m.) > > > Review request for atlas. > > > Bugs: ATLAS-991 > https://issues.apache.org/jira/browse/ATLAS-991 > > > Repository: atlas > > > Description > --- > > When deleting entities, avoid enforcing lower bound constraints (i.e. > required attributes) on all entities being deleted. Previously, only the > entities that were specifically requested for deletion were skipped when > enforcing required attributes, but any composite entities that were owned by > 'root' deletion candidates were not skipped, causing > NullRequiredAttributeException to be thrown unnecessarily. > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java > 8d31c1b079a11291a87569fde6f2cb21d4258ca9 > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java > e301a00b16ad316a85714cbda4af43d01f11f4ad > repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java > 81fb76e27019012fada88a70a13b4c5264c236bf > > repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java > 34842c3dc2904b123fecb5087aee357678b00156 > > repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositoryHardDeleteTest.java > cc6026461bcd38b83dde0bcec2e835f5824e894f > > repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositorySoftDeleteTest.java > 90bb635126a02cd38c045bd5332499c4481d3751 > > repository/src/test/java/org/apache/atlas/repository/graph/GraphHelperTest.java > 8a0a6bc60818c3dd58e12386c55a471de2508b5e > > Diff: https://reviews.apache.org/r/49869/diff/ > > > Testing > --- > > Ran all unit and integration tests with no regressions. > > > Thanks, > > David Kantor > >
Review Request 50712: ATLAS-965 Old lineage still exists after dropping tables and re-creating tables with same name
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50712/ --- Review request for atlas. Bugs: ATLAS-965 https://issues.apache.org/jira/browse/ATLAS-965 Repository: atlas Description --- Added table create time to the process qualified name for create table operations Diffs - addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java e0d8024 addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java e27e52c addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java PRE-CREATION addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetastoreBridgeIT.java PRE-CREATION addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 573f7f5 docs/pom.xml 9b8440d pom.xml 4855d3d webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java 1aee467 Diff: https://reviews.apache.org/r/50712/diff/ Testing --- ITs and end to end testing Thanks, Shwetha GS
Re: Review Request 50559: ATLAS-1064 : UI: Pagination for full text search results
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50559/#review143916 --- Ship it! Ship It! - Shwetha GS On July 28, 2016, 10:27 a.m., Kalyani Kashikar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/50559/ > --- > > (Updated July 28, 2016, 10:27 a.m.) > > > Review request for atlas, Gautam Borad, Madhan Neethiraj, Shwetha GS, > Selvamohan Neethiraj, and Suma Shivaprasad. > > > Bugs: ATLAS-1064 > https://issues.apache.org/jira/browse/ATLAS-1064 > > > Repository: atlas > > > Description > --- > > Handle API base pagination for full text search result: > > * Previous and Next buttons is shown on top of the table. > * Count of results shown, it includes page details, for example, 1-25, 26-50. > > > Diffs > - > > dashboardv2/public/js/templates/search/SearchResultLayoutView_tmpl.html > 609ec23 > dashboardv2/public/js/utils/TableLayout.js c83cf76 > dashboardv2/public/js/views/search/SearchResultLayoutView.js f9f82f4 > > Diff: https://reviews.apache.org/r/50559/diff/ > > > Testing > --- > > ** Verified following scenarios: ** > > * Previous and Next buttons is shown on top of the table, it fetches 25 > records in begining. > * As user clicks on next button it fetches next 25 records and count of > results shown page details, ex: 1-25. > * As user clicks on Previous button the previous records are shown. > > > Thanks, > > Kalyani Kashikar > >
Re: Review Request 49625: ATLAS-347 Atlas search APIs should allow pagination of results
> On July 19, 2016, 8:33 p.m., Suma Shivaprasad wrote: > > repository/src/main/scala/org/apache/atlas/query/QueryParser.scala, line 172 > > <https://reviews.apache.org/r/49625/diff/3/?file=1447361#file1447361line172> > > > > This change now means that if there are multiple queries, then select, > > loop, orderby , limit are applied at the end only and not for each query > > like earlier? select in inner query wasn't working, for example queries like 'hive_table select name as tname, hive_db select name as dbname' didn't work. This query should be hive_table as t, hive_db as d select t.name as tname, d.name as dbname. Since select can be specified only once, it makes sense to specify even order and limit only once. The query is simpler as well. Hence fixed the query parser as well - Shwetha --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49625/#review142827 ----------- On July 19, 2016, 4:45 p.m., Shwetha GS wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/49625/ > --- > > (Updated July 19, 2016, 4:45 p.m.) > > > Review request for atlas. > > > Bugs: ATLAS-347 > https://issues.apache.org/jira/browse/ATLAS-347 > > > Repository: atlas > > > Description > --- > > 1. orderby doesn't require fully qualified field name anymore. For example, > 'hive_table orderby name' resolves name to Asset.name > 2. Added limit and offset to search by DSL API. > 3. DSL query adds API's limit and offset to the specified DSL query. For > eaxmple, 'hive_table orderby name' maps to 'hive_table orderby name limit 100 > offset 0'. 'hive_table orderby name limit 200 offset 10' maps to 'hive_table > orderby name limit 100 offset 10'. 100 and 0 are API's limit and offset > respectively here > 4. All existing tests work > > Pending: > 1. Add more tests > 2. Add limit and offset for full text search. I don't think the index search > that we do supports pagination. May have to filter results on atlas server > > > Diffs > - > > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java > 9a5e279 > > addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java > 9f7f6b0 > > addons/sqoop-bridge/src/test/java/org/apache/atlas/sqoop/hook/SqoopHookIT.java > e965c7d > > addons/storm-bridge/src/test/java/org/apache/atlas/storm/hook/StormAtlasHookIT.java > b33bb5f > client/src/main/java/org/apache/atlas/AtlasClient.java d3af6ad > common/src/main/java/org/apache/atlas/AtlasProperties.java PRE-CREATION > common/src/main/java/org/apache/atlas/utils/ParamChecker.java edfe355 > docs/src/site/twiki/Configuration.twiki 3ad0fbe > docs/src/site/twiki/Search.twiki 58c9238 > > repository/src/main/java/org/apache/atlas/discovery/DataSetLineageService.java > c6790de > repository/src/main/java/org/apache/atlas/discovery/DiscoveryService.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/discovery/graph/DefaultGraphPersistenceStrategy.java > e07a54e > > repository/src/main/java/org/apache/atlas/discovery/graph/GraphBackedDiscoveryService.java > 6def78b > repository/src/main/java/org/apache/atlas/repository/Constants.java 893f1b6 > repository/src/main/scala/org/apache/atlas/query/Expressions.scala ab7e81c > repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala d76b173 > repository/src/main/scala/org/apache/atlas/query/QueryParser.scala 60b57d9 > repository/src/main/scala/org/apache/atlas/query/Resolver.scala 5fc9400 > repository/src/main/scala/org/apache/atlas/query/TypeUtils.scala 5a64c53 > > repository/src/test/java/org/apache/atlas/discovery/DataSetLineageServiceTest.java > 89e9b41 > > repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java > c27894e > > repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java > 97dfbcd > > repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java > f2a5b50 > server-api/src/main/java/org/apache/atlas/discovery/DiscoveryService.java > e347c2c > webapp/src/main/java/org/apache/atlas/LocalAtlasClient.java 7173d4d > webapp/src/main/java/org/apache/atlas/examples/QuickStart.java 42bccec > > webapp/src/main/java/org/apache/atlas/web/resources/Metada
Re: Review Request 49625: ATLAS-347 Atlas search APIs should allow pagination of results
> On July 19, 2016, 5:50 p.m., Suma Shivaprasad wrote: > > client/src/main/java/org/apache/atlas/AtlasClient.java, line 1020 > > <https://reviews.apache.org/r/49625/diff/3/?file=1447349#file1447349line1020> > > > > Should we expose 2 APIs - one without offset and limit? > > Suma Shivaprasad wrote: > and which has a default? Don't want to add APIs without limit and offset - want to force users to think about these params while searching. In rest APIs, they are optional though - Shwetha --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49625/#review142775 ------- On July 19, 2016, 4:45 p.m., Shwetha GS wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/49625/ > --- > > (Updated July 19, 2016, 4:45 p.m.) > > > Review request for atlas. > > > Bugs: ATLAS-347 > https://issues.apache.org/jira/browse/ATLAS-347 > > > Repository: atlas > > > Description > --- > > 1. orderby doesn't require fully qualified field name anymore. For example, > 'hive_table orderby name' resolves name to Asset.name > 2. Added limit and offset to search by DSL API. > 3. DSL query adds API's limit and offset to the specified DSL query. For > eaxmple, 'hive_table orderby name' maps to 'hive_table orderby name limit 100 > offset 0'. 'hive_table orderby name limit 200 offset 10' maps to 'hive_table > orderby name limit 100 offset 10'. 100 and 0 are API's limit and offset > respectively here > 4. All existing tests work > > Pending: > 1. Add more tests > 2. Add limit and offset for full text search. I don't think the index search > that we do supports pagination. May have to filter results on atlas server > > > Diffs > - > > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java > 9a5e279 > > addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java > 9f7f6b0 > > addons/sqoop-bridge/src/test/java/org/apache/atlas/sqoop/hook/SqoopHookIT.java > e965c7d > > addons/storm-bridge/src/test/java/org/apache/atlas/storm/hook/StormAtlasHookIT.java > b33bb5f > client/src/main/java/org/apache/atlas/AtlasClient.java d3af6ad > common/src/main/java/org/apache/atlas/AtlasProperties.java PRE-CREATION > common/src/main/java/org/apache/atlas/utils/ParamChecker.java edfe355 > docs/src/site/twiki/Configuration.twiki 3ad0fbe > docs/src/site/twiki/Search.twiki 58c9238 > > repository/src/main/java/org/apache/atlas/discovery/DataSetLineageService.java > c6790de > repository/src/main/java/org/apache/atlas/discovery/DiscoveryService.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/discovery/graph/DefaultGraphPersistenceStrategy.java > e07a54e > > repository/src/main/java/org/apache/atlas/discovery/graph/GraphBackedDiscoveryService.java > 6def78b > repository/src/main/java/org/apache/atlas/repository/Constants.java 893f1b6 > repository/src/main/scala/org/apache/atlas/query/Expressions.scala ab7e81c > repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala d76b173 > repository/src/main/scala/org/apache/atlas/query/QueryParser.scala 60b57d9 > repository/src/main/scala/org/apache/atlas/query/Resolver.scala 5fc9400 > repository/src/main/scala/org/apache/atlas/query/TypeUtils.scala 5a64c53 > > repository/src/test/java/org/apache/atlas/discovery/DataSetLineageServiceTest.java > 89e9b41 > > repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java > c27894e > > repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java > 97dfbcd > > repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java > f2a5b50 > server-api/src/main/java/org/apache/atlas/discovery/DiscoveryService.java > e347c2c > webapp/src/main/java/org/apache/atlas/LocalAtlasClient.java 7173d4d > webapp/src/main/java/org/apache/atlas/examples/QuickStart.java 42bccec > > webapp/src/main/java/org/apache/atlas/web/resources/MetadataDiscoveryResource.java > 256f7d3 > > webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerIT.java > 72f403e > webapp/src/test/java/org/apache/atlas/web/resources/BaseResourceIT.java > 498db92 > > webapp/src/test/java/org/apache/atlas/web/resources/EntityJerseyResourceIT.java > d41041c > > webapp/src/test/java/org/apache/atlas/web/resources/MetadataDiscoveryJerseyResourceIT.java > a8c1213 > > Diff: https://reviews.apache.org/r/49625/diff/ > > > Testing > --- > > > Thanks, > > Shwetha GS > >
Re: Review Request 49625: ATLAS-347 Atlas search APIs should allow pagination of results
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49625/ --- (Updated July 19, 2016, 4:45 p.m.) Review request for atlas. Changes --- Addressed review comments. Pending - re-visit the defaults. Will check tomorrow Bugs: ATLAS-347 https://issues.apache.org/jira/browse/ATLAS-347 Repository: atlas Description --- 1. orderby doesn't require fully qualified field name anymore. For example, 'hive_table orderby name' resolves name to Asset.name 2. Added limit and offset to search by DSL API. 3. DSL query adds API's limit and offset to the specified DSL query. For eaxmple, 'hive_table orderby name' maps to 'hive_table orderby name limit 100 offset 0'. 'hive_table orderby name limit 200 offset 10' maps to 'hive_table orderby name limit 100 offset 10'. 100 and 0 are API's limit and offset respectively here 4. All existing tests work Pending: 1. Add more tests 2. Add limit and offset for full text search. I don't think the index search that we do supports pagination. May have to filter results on atlas server Diffs (updated) - addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 9a5e279 addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java 9f7f6b0 addons/sqoop-bridge/src/test/java/org/apache/atlas/sqoop/hook/SqoopHookIT.java e965c7d addons/storm-bridge/src/test/java/org/apache/atlas/storm/hook/StormAtlasHookIT.java b33bb5f client/src/main/java/org/apache/atlas/AtlasClient.java d3af6ad common/src/main/java/org/apache/atlas/AtlasProperties.java PRE-CREATION common/src/main/java/org/apache/atlas/utils/ParamChecker.java edfe355 docs/src/site/twiki/Configuration.twiki 3ad0fbe docs/src/site/twiki/Search.twiki 58c9238 repository/src/main/java/org/apache/atlas/discovery/DataSetLineageService.java c6790de repository/src/main/java/org/apache/atlas/discovery/DiscoveryService.java PRE-CREATION repository/src/main/java/org/apache/atlas/discovery/graph/DefaultGraphPersistenceStrategy.java e07a54e repository/src/main/java/org/apache/atlas/discovery/graph/GraphBackedDiscoveryService.java 6def78b repository/src/main/java/org/apache/atlas/repository/Constants.java 893f1b6 repository/src/main/scala/org/apache/atlas/query/Expressions.scala ab7e81c repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala d76b173 repository/src/main/scala/org/apache/atlas/query/QueryParser.scala 60b57d9 repository/src/main/scala/org/apache/atlas/query/Resolver.scala 5fc9400 repository/src/main/scala/org/apache/atlas/query/TypeUtils.scala 5a64c53 repository/src/test/java/org/apache/atlas/discovery/DataSetLineageServiceTest.java 89e9b41 repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java c27894e repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java 97dfbcd repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java f2a5b50 server-api/src/main/java/org/apache/atlas/discovery/DiscoveryService.java e347c2c webapp/src/main/java/org/apache/atlas/LocalAtlasClient.java 7173d4d webapp/src/main/java/org/apache/atlas/examples/QuickStart.java 42bccec webapp/src/main/java/org/apache/atlas/web/resources/MetadataDiscoveryResource.java 256f7d3 webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerIT.java 72f403e webapp/src/test/java/org/apache/atlas/web/resources/BaseResourceIT.java 498db92 webapp/src/test/java/org/apache/atlas/web/resources/EntityJerseyResourceIT.java d41041c webapp/src/test/java/org/apache/atlas/web/resources/MetadataDiscoveryJerseyResourceIT.java a8c1213 Diff: https://reviews.apache.org/r/49625/diff/ Testing --- Thanks, Shwetha GS
Re: Review Request 50146: ATLAS-957 Atlas is not capturing topologies that have $ in the data payload
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50146/ --- (Updated July 19, 2016, 6:02 a.m.) Review request for atlas. Changes --- Addressed review comments Bugs: ATLAS-957 https://issues.apache.org/jira/browse/ATLAS-957 Repository: atlas Description --- 1. Allow type names and attribute names with special characters. Titan doesn't allow property keys with special characters. So, encoding property keys in create index, set property, get property and search 2. The property values can contain special characters 3. DSL to gremlin converter, escaped " as its used around string and $ as its used as variable denoter Diffs (updated) - repository/src/main/java/org/apache/atlas/discovery/graph/DefaultGraphPersistenceStrategy.java 8ddbe1b repository/src/main/java/org/apache/atlas/discovery/graph/GraphBackedDiscoveryService.java f97b83d repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 91f9bd0 repository/src/main/java/org/apache/atlas/repository/graph/FullTextMapper.java 37cacb0 repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java e5c0e46 repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 8c8134f repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 4f531e2 repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java e240fb6 repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java 4c1f559 repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java 4503899 repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala 8add6c8 repository/src/test/java/org/apache/atlas/repository/graph/GraphHelperTest.java PRE-CREATION repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java 443dd4a Diff: https://reviews.apache.org/r/50146/diff/ Testing --- Verified create type, create entity and search using DSL Thanks, Shwetha GS
Re: Review Request 50111: ATLAS-1027: Atlas hooks should use properties from atlas-application.properties, instead of component's configuration
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50111/#review142582 --- Ship it! Ship It! - Shwetha GS On July 17, 2016, 11:03 p.m., Madhan Neethiraj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/50111/ > --- > > (Updated July 17, 2016, 11:03 p.m.) > > > Review request for atlas. > > > Bugs: ATLAS-1027 > https://issues.apache.org/jira/browse/ATLAS-1027 > > > Repository: atlas > > > Description > --- > > Updated Atlas hooks to read properties from atlas-application.properties, > instaead of component configuration > > > Diffs > - > > > addons/falcon-bridge/src/test/java/org/apache/atlas/falcon/hook/FalconHookIT.java > 0139bf5 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java > cd0e964 > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java > 99009ba > addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java > 734b606 > > addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java > c412658 > > addons/sqoop-bridge/src/test/java/org/apache/atlas/sqoop/hook/SqoopHookIT.java > 72d9201 > > addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java > 5bd5397 > > Diff: https://reviews.apache.org/r/50111/diff/ > > > Testing > --- > > Verified that the configuration is read from atlas-application.properties > when present; when not present, from the component configuration. > > > Thanks, > > Madhan Neethiraj > >
Re: Review Request 49941: ATLAS-998: determine HA mode from property atlas.server.ids, instead of atlas.server.ha.enabled
> On July 12, 2016, 5:52 a.m., Shwetha GS wrote: > > common/src/main/java/org/apache/atlas/ha/HAConfiguration.java, line 65 > > <https://reviews.apache.org/r/49941/diff/1/?file=1441953#file1441953line65> > > > > To make it backward compatible, as 0.7 is already out, can the check be > > if ha.enabled or ids > 1? > > Madhan Neethiraj wrote: > The fix is to remove 'ha.enabled' property. Instead of keeping it for > backward compatibility, I would suggest to document the removal of this > property in the next release notes. The fix is to use ids property to enable HA. For someone upgrading, all these are manual changes and add to upgrade checklist and its better to minimise if possible. Can you update docs/src/site/twiki/HighAvailability.twiki? Its difficult to compile this list during release. Can you create upgrade.twiki under docs/src/site/twiki/ and add to it? - Shwetha --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49941/#review141855 --- On July 13, 2016, 6:20 a.m., Madhan Neethiraj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/49941/ > --- > > (Updated July 13, 2016, 6:20 a.m.) > > > Review request for atlas. > > > Bugs: ATLAS-998 > https://issues.apache.org/jira/browse/ATLAS-998 > > > Repository: atlas > > > Description > --- > > Updated to use atlas.server.ids property to determine if multiple instances > of Atlas are configured to run (in HA mode), instead of > atlas.server.ha.enabled > > > Diffs > - > > common/src/main/java/org/apache/atlas/ha/HAConfiguration.java f3c9bc5 > common/src/test/java/org/apache/atlas/ha/HAConfigurationTest.java a3129c9 > > repository/src/test/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepositoryHATest.java > 2f7edb4 > > repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexerMockTest.java > 94dd298 > > repository/src/test/java/org/apache/atlas/services/DefaultMetadataServiceMockTest.java > 393b539 > > webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java > 7860eb6 > > webapp/src/test/java/org/apache/atlas/web/service/ActiveInstanceElectorServiceTest.java > 2b7691f > webapp/src/test/java/org/apache/atlas/web/service/ServiceStateTest.java > 77aba88 > > Diff: https://reviews.apache.org/r/49941/diff/ > > > Testing > --- > > - Verified that all unit tests succeed > - Verified that when multiple instances are listed in 'atlas.server.ids' > configuration, Atlas server takes HA code path > > > Thanks, > > Madhan Neethiraj > >
Re: Review Request 49570: ATLAS-986: Ability to differentiate business catalog terms from traits
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49570/#review142004 --- catalog/src/main/java/org/apache/atlas/catalog/DefaultTypeSystem.java (line 209) <https://reviews.apache.org/r/49570/#comment207483> Instead of adding a supertype, lets just add the attribute directly. We need to support migration path for future versions. Adding an optional attribute is supported, but not adding a supertype common/src/main/java/org/apache/atlas/AtlasConstants.java (line 36) <https://reviews.apache.org/r/49570/#comment207482> Move these constants to catalog module Rename to taxonomy.namespace so that it doesn't conflict with namespace that we might add to all entities/types in future typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java (line 19) <https://reviews.apache.org/r/49570/#comment207461> revert, no changes? - Shwetha GS On July 3, 2016, 7:21 a.m., Madhan Neethiraj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/49570/ > --- > > (Updated July 3, 2016, 7:21 a.m.) > > > Review request for atlas and Suma Shivaprasad. > > > Bugs: ATLAS-986 > https://issues.apache.org/jira/browse/ATLAS-986 > > > Repository: atlas > > > Description > --- > > Ability to differentiate business catalog terms from traits (author: > suma.shivaprasad) > > > Diffs > - > > addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java > 8ca47d9 > catalog/pom.xml 7b9b691 > catalog/src/main/java/org/apache/atlas/catalog/AtlasTypeSystem.java 3a58488 > catalog/src/main/java/org/apache/atlas/catalog/DefaultTypeSystem.java > a8e63f3 > > catalog/src/main/java/org/apache/atlas/catalog/definition/TaxonomyResourceDefinition.java > ff22f3b > catalog/src/test/java/org/apache/atlas/catalog/DefaultTypeSystemTest.java > PRE-CREATION > > catalog/src/test/java/org/apache/atlas/catalog/TaxonomyResourceProviderTest.java > 5a03414 > > catalog/src/test/java/org/apache/atlas/catalog/definition/TaxonomyResourceDefinitionTest.java > 33620c3 > common/src/main/java/org/apache/atlas/AtlasConstants.java cb46316 > pom.xml f119525 > repository/pom.xml 9c34f19 > > repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java > 99f0036 > repository/src/test/java/org/apache/atlas/TestUtils.java a159496 > > repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java > e6dd230 > server-api/src/main/java/org/apache/atlas/services/MetadataService.java > 7cc036c > typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java > c355ba9 > webapp/src/main/java/org/apache/atlas/web/resources/EntityService.java > 1f0c944 > webapp/src/main/java/org/apache/atlas/web/resources/TaxonomyService.java > d84f8a2 > > webapp/src/test/java/org/apache/atlas/web/resources/TaxonomyServiceTest.java > b2ff4cf > > Diff: https://reviews.apache.org/r/49570/diff/ > > > Testing > --- > > > Thanks, > > Madhan Neethiraj > >
Re: Review Request 49941: ATLAS-998: determine HA mode from property atlas.server.ids, instead of atlas.server.ha.enabled
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49941/#review141855 --- common/src/main/java/org/apache/atlas/ha/HAConfiguration.java (line 55) <https://reviews.apache.org/r/49941/#comment207255> move it to tests common/src/main/java/org/apache/atlas/ha/HAConfiguration.java (line 64) <https://reviews.apache.org/r/49941/#comment207256> To make it backward compatible, as 0.7 is already out, can the check be if ha.enabled or ids > 1? - Shwetha GS On July 12, 2016, 1:29 a.m., Madhan Neethiraj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/49941/ > --- > > (Updated July 12, 2016, 1:29 a.m.) > > > Review request for atlas. > > > Bugs: ATLAS-998 > https://issues.apache.org/jira/browse/ATLAS-998 > > > Repository: atlas > > > Description > --- > > Updated to use atlas.server.ids property to determine if multiple instances > of Atlas are configured to run (in HA mode), instead of > atlas.server.ha.enabled > > > Diffs > - > > common/src/main/java/org/apache/atlas/ha/HAConfiguration.java f3c9bc5 > common/src/test/java/org/apache/atlas/ha/HAConfigurationTest.java a3129c9 > > repository/src/test/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepositoryHATest.java > 2f7edb4 > > repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexerMockTest.java > 94dd298 > > repository/src/test/java/org/apache/atlas/services/DefaultMetadataServiceMockTest.java > 393b539 > > webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java > 7860eb6 > > webapp/src/test/java/org/apache/atlas/web/service/ActiveInstanceElectorServiceTest.java > 2b7691f > webapp/src/test/java/org/apache/atlas/web/service/ServiceStateTest.java > 77aba88 > > Diff: https://reviews.apache.org/r/49941/diff/ > > > Testing > --- > > - Verified that all unit tests succeed > - Verified that when multiple instances are listed in 'atlas.server.ids' > configuration, Atlas server takes HA code path > > > Thanks, > > Madhan Neethiraj > >
Re: Review Request 49585: ATLAS-987: load dependent libraries of Atlas hooks in a separate class loader to avoid adding them to component CLASSPATH
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49585/#review141243 --- Ship it! Ship It! - Shwetha GS On July 7, 2016, 9:32 p.m., Madhan Neethiraj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/49585/ > --- > > (Updated July 7, 2016, 9:32 p.m.) > > > Review request for atlas. > > > Bugs: ATLAS-987 > https://issues.apache.org/jira/browse/ATLAS-987 > > > Repository: atlas > > > Description > --- > > Dependent libraries of Atlas hooks are loaded in a separate class loader, > there by making these libraries not visibile to components. > > Here is the summary of the changes: > - current contents of atlas/hook/ directiries, including the > hook class implementation, are moved under > atlas/hook//atlas--plugin-impl directory > - a new component specific shim library, that includes the hook class > registered with the component, is placed in directory atlas/hook/ > - the hook class in the shim library loads all files in > atlas--plugin-impl directory in a classloader and forwards all the > calls on the shim class to the real implementation class > > This implementation is same as the one used in Apache Ranger - more details > in RANGER-586. > > > Diffs > - > > addons/falcon-bridge-shim/pom.xml PRE-CREATION > > addons/falcon-bridge-shim/src/main/java/org/apache/atlas/falcon/hook/AtlasService.java > PRE-CREATION > addons/falcon-bridge/pom.xml d79dda9 > addons/hive-bridge-shim/pom.xml PRE-CREATION > > addons/hive-bridge-shim/src/main/java/org/apache/atlas/hive/hook/HiveHook.java > PRE-CREATION > addons/hive-bridge/pom.xml ddefdc2 > addons/sqoop-bridge-shim/pom.xml PRE-CREATION > > addons/sqoop-bridge-shim/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java > PRE-CREATION > addons/sqoop-bridge/pom.xml c792945 > addons/storm-bridge-shim/pom.xml PRE-CREATION > > addons/storm-bridge-shim/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java > PRE-CREATION > addons/storm-bridge/pom.xml 9e8bf2f > plugin-classloader/pom.xml PRE-CREATION > > plugin-classloader/src/main/java/org/apache/atlas/plugin/classloader/AtlasPluginClassLoader.java > PRE-CREATION > > plugin-classloader/src/main/java/org/apache/atlas/plugin/classloader/AtlasPluginClassLoaderUtil.java > PRE-CREATION > > plugin-classloader/src/test/java/org/apache/atlas/plugin/classloader/AtlasPluginClassLoaderTest.java > PRE-CREATION > pom.xml f119525 > > Diff: https://reviews.apache.org/r/49585/diff/ > > > Testing > --- > > Verified that existing tests pass > Verified Hive hook successfully sends notification to Kafka, which in turn > was processed correctly by Atlas server > > > Thanks, > > Madhan Neethiraj > >
Re: Review Request 49585: ATLAS-987: load dependent libraries of Atlas hooks in a separate class loader to avoid adding them to component CLASSPATH
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49585/#review141174
---
Here is the test that works:
public void testClassLoader() throws Exception {
String cls = "org.apache.atlas.service.Services";
try {
loadClass(cls, null);
fail("Expected ClassNotFoundException");
} catch(ClassNotFoundException e) {
//expected
}
AtlasPluginClassLoader classLoader = new
AtlasPluginClassLoader("../common/target");
classLoader.activate();
//org.apache.atlas.service.Services class should be loadable now
//should also load org.apache.atlas.service.Service
Class servicesCls = loadClass(cls, null);
loadClass("org.apache.atlas.service.Service",
servicesCls.getClassLoader());
//Fall back to current class loader should also work
loadClass(AtlasPluginClassLoaderUtil.class.getName(), null);
classLoader.deactivate();
//After disable, class loading should fail again
try {
loadClass(cls, null);
fail("Expected ClassNotFoundException");
} catch(ClassNotFoundException e) {
//expected
}
}
private Class loadClass(String cls, ClassLoader classLoader) throws
ClassNotFoundException {
if (classLoader == null) {
classLoader = Thread.currentThread().getContextClassLoader();
}
return Class.forName(cls, true, classLoader);
}
Also, modified the constructors:
private AtlasPluginClassLoader(String pluginType, Class pluginClass)
throws Exception {
this(AtlasPluginClassLoaderUtil.getPluginImplLibPath(pluginType,
pluginClass));
}
//visible for testing
AtlasPluginClassLoader(String libraryPath) throws Exception {
super(AtlasPluginClassLoaderUtil.getPluginFilesForPluginTypeAndPluginclass(libraryPath),
null);
componentClassLoader = AccessController.doPrivileged(new
PrivilegedAction() {
public MyClassLoader run() {
return new
MyClassLoader(Thread.currentThread().getContextClassLoader());
}
});
}
addons/falcon-bridge-shim/src/main/java/org/apache/atlas/falcon/hook/AtlasService.java
(line 47)
<https://reviews.apache.org/r/49585/#comment206588>
getName() can return this.getClass.getName. Doesn't need to re-direct to
plugin
addons/falcon-bridge-shim/src/main/java/org/apache/atlas/falcon/hook/AtlasService.java
(line 181)
<https://reviews.apache.org/r/49585/#comment206589>
This is a common code across all hooks. Move to utility in
plugin-classloader?
plugin-classloader/pom.xml (line 38)
<https://reviews.apache.org/r/49585/#comment206590>
already in main pom, so not required
plugin-classloader/src/main/java/org/apache/atlas/plugin/classloader/AtlasPluginClassLoader.java
(line 64)
<https://reviews.apache.org/r/49585/#comment206591>
findClass and loadClass are called a lot of times in the component and this
prints way too many logs. For debugging, -verbose:class gives enough
information. So, can you remove these debug statements?
Also, consider removing debug statements in other methods as well. Don't
think we need at both entry and exit of every method
plugin-classloader/src/main/java/org/apache/atlas/plugin/classloader/AtlasPluginClassLoader.java
(line 97)
<https://reviews.apache.org/r/49585/#comment206592>
is synchronised required here?
plugin-classloader/src/main/java/org/apache/atlas/plugin/classloader/AtlasPluginClassLoaderUtil.java
(line 44)
<https://reviews.apache.org/r/49585/#comment206594>
getInstance() not required for utility class. Make the methods static
instead
- Shwetha GS
On July 5, 2016, 1:24 a.m., Madhan Neethiraj wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49585/
> ---
>
> (Updated July 5, 2016, 1:24 a.m.)
>
>
> Review request for atlas.
>
>
> Bugs: ATLAS-987
> https://issues.apache.org/jira/browse/ATLAS-987
>
>
> Repository: atlas
>
>
> Description
> ---
>
> Dependent libraries of Atlas hooks are loaded in a separate class loader,
> there by making these libraries not visibile to components.
>
> Here is the summary of the changes:
> - current contents of atlas/hook/ directiries, including the
> hook class implementation, are moved under
> atlas/hook//atlas--plugin-impl directory
> - a new component specific shim library, that includes the hook class
> registered w
Re: Review Request 49585: ATLAS-987: load dependent libraries of Atlas hooks in a separate class loader to avoid adding them to component CLASSPATH
> On July 4, 2016, 10:13 a.m., Shwetha GS wrote: > > addons/falcon-bridge-shim/src/main/java/org/apache/atlas/falcon/hook/AtlasService.java, > > line 31 > > <https://reviews.apache.org/r/49585/diff/2/?file=1436099#file1436099line31> > > > > Does falcon-shim/AtlasService and falcon/AtasService need to have same > > name and package? If not, can you rename them as its confusing > > > > For example, falcon-shim/AtlasService can be renamed to > > AtlasProxyService > > Madhan Neethiraj wrote: > With the use of classname/package for the shim and the implementation, no > changes are needed in the hook registration in components > (Falcon/Hive/Sqoop/Storm). Hence this choice. > > Madhan Neethiraj wrote: > Typo: "With the use of classname/package" ==> "With the use of same > classname/package" Lets rename falcon/AtasService then. That wouldn't require changes in hook configuration, right? > On July 4, 2016, 10:13 a.m., Shwetha GS wrote: > > addons/falcon-bridge-shim/src/main/java/org/apache/atlas/falcon/hook/AtlasService.java, > > line 46 > > <https://reviews.apache.org/r/49585/diff/2/?file=1436099#file1436099line46> > > > > We should use aspectj and remove loggers and activate and deactive. If > > you can't address in this patch, can you file a bug? > > Madhan Neethiraj wrote: > The comment does not seem related to this source line. Can you review? Every method in falcon-shim/AtlasService has this logic - debug statement, activate, call plugin, deactive, debug statement, which is repeatitive. You can clean this up with aspectj annotations. With this, methods will just have an annotation and plugin call. Check GraphTransactionInterceptor for reference - Shwetha --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49585/#review140632 --- On July 5, 2016, 1:24 a.m., Madhan Neethiraj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/49585/ > --- > > (Updated July 5, 2016, 1:24 a.m.) > > > Review request for atlas. > > > Bugs: ATLAS-987 > https://issues.apache.org/jira/browse/ATLAS-987 > > > Repository: atlas > > > Description > --- > > Dependent libraries of Atlas hooks are loaded in a separate class loader, > there by making these libraries not visibile to components. > > Here is the summary of the changes: > - current contents of atlas/hook/ directiries, including the > hook class implementation, are moved under > atlas/hook//atlas--plugin-impl directory > - a new component specific shim library, that includes the hook class > registered with the component, is placed in directory atlas/hook/ > - the hook class in the shim library loads all files in > atlas--plugin-impl directory in a classloader and forwards all the > calls on the shim class to the real implementation class > > This implementation is same as the one used in Apache Ranger - more details > in RANGER-586. > > > Diffs > - > > addons/falcon-bridge-shim/pom.xml PRE-CREATION > > addons/falcon-bridge-shim/src/main/java/org/apache/atlas/falcon/hook/AtlasService.java > PRE-CREATION > addons/falcon-bridge/pom.xml d79dda9 > addons/hive-bridge-shim/pom.xml PRE-CREATION > > addons/hive-bridge-shim/src/main/java/org/apache/atlas/hive/hook/HiveHook.java > PRE-CREATION > addons/hive-bridge/pom.xml ddefdc2 > addons/sqoop-bridge-shim/pom.xml PRE-CREATION > > addons/sqoop-bridge-shim/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java > PRE-CREATION > addons/sqoop-bridge/pom.xml c792945 > addons/storm-bridge-shim/pom.xml PRE-CREATION > > addons/storm-bridge-shim/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java > PRE-CREATION > addons/storm-bridge/pom.xml 9e8bf2f > plugin-classloader/pom.xml PRE-CREATION > > plugin-classloader/src/main/java/org/apache/atlas/plugin/classloader/AtlasPluginClassLoader.java > PRE-CREATION > > plugin-classloader/src/main/java/org/apache/atlas/plugin/classloader/AtlasPluginClassLoaderUtil.java > PRE-CREATION > pom.xml f119525 > > Diff: https://reviews.apache.org/r/49585/diff/ > > > Testing > --- > > Verified that existing tests pass > Verified Hive hook successfully sends notification to Kafka, which in turn > was processed correctly by Atlas server > > > Thanks, > > Madhan Neethiraj > >
Re: Review Request 49717: Atlas-435: Fix the NPE when proeprty in order clause is not defined.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49717/#review141129 --- Ship it! Ship It! - Shwetha GS On July 6, 2016, 7:19 p.m., Neeru Gupta wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/49717/ > --- > > (Updated July 6, 2016, 7:19 p.m.) > > > Review request for atlas and Shwetha GS. > > > Bugs: Atlas-993 > https://issues.apache.org/jira/browse/Atlas-993 > > > Repository: atlas > > > Description > --- > > Atlas-435 fixed the NPE when order by condtion is not defined > > > Atlas-435 fixed the NPE when order by condtion is not defined > > > Diffs > - > > repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala > 73981c04d410a7292de357d9f63b6f3b3d241ffa > > repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java > 64c8bc76810484e3287e8aad0db8c63243bb354e > > Diff: https://reviews.apache.org/r/49717/diff/ > > > Testing > --- > > Execute all the existing tests successfully. added new test with undefined > property in order by clause. > > > Thanks, > > Neeru Gupta > >
Re: Review Request 49350: ATLAS-917 Add HDFS paths for non-partition queries
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49350/#review140119 --- Ship it! Ship It! - Shwetha GS On June 29, 2016, 10:05 p.m., Suma Shivaprasad wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/49350/ > --- > > (Updated June 29, 2016, 10:05 p.m.) > > > Review request for atlas. > > > Repository: atlas > > > Description > --- > > this is added for LOAD, IMPORT, EXPORT . Could not add to INSERT OVERWRITE > hdfs paths due to some hive issues which does not provide partition related > information in this case. That will continue to work as earlier. > > > Diffs > - > > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java > 0045780 > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java > a1a00b3 > addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java > f9e1926 > notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 8bbe2d7 > > Diff: https://reviews.apache.org/r/49350/diff/ > > > Testing > --- > > Added tests for partitioned and unpartitioned export, import, load > > > Thanks, > > Suma Shivaprasad > >
Re: Review Request 49357: ATLAS-958 : Add licenses for dashboard dependencies
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49357/#review140004
---
Ship it!
Ship It!
- Shwetha GS
On June 29, 2016, 2:58 p.m., keval bhatt wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49357/
> ---
>
> (Updated June 29, 2016, 2:58 p.m.)
>
>
> Review request for atlas, Gautam Borad, Madhan Neethiraj, Shwetha GS, and
> Hemanth Yamijala.
>
>
> Bugs: ATLAS-958
> https://issues.apache.org/jira/browse/ATLAS-958
>
>
> Repository: atlas
>
>
> Description
> ---
>
> Patch include following changes.
>
> * Removed unused font (glyphicon)
> * Added licence file to all js libraries (added grunt task to copy license
> file to dist folder)
> * Added all libraries license details in {atlas_home}/LICENSE.txt file
> * Removed unused plugin (tree.js)
>
>
> Diffs
> -
>
> 3party-licenses/animate-LICENSE PRE-CREATION
> 3party-licenses/bootstrap-sidebar-LICENSE PRE-CREATION
> 3party-licenses/handlebars-LICENSE PRE-CREATION
> 3party-licenses/hbs-LICENSE PRE-CREATION
> 3party-licenses/require-handlebars-plugin-LICENSE PRE-CREATION
> LICENSE 8f075f5
> dashboardv2/gruntfile.js 2f30f3b
> dashboardv2/package.json 9ec4feb
> dashboardv2/public/css/bootstrap-sidebar.css 5f9d5c3
> dashboardv2/public/css/font-awesome.min.css acce2ef
> dashboardv2/public/css/fonts/FontAwesome.otf
> 3ed7f8b48ad9bfab52eb03822fefcd6b77d2e680
> dashboardv2/public/css/fonts/fontawesome-webfont.eot
> 9b6afaedc0fd7aaf927a07f82da9c11022251b8b
> dashboardv2/public/css/fonts/fontawesome-webfont.svg d05688e
> dashboardv2/public/css/fonts/fontawesome-webfont.ttf
> 26dea7951a73079223b50653c455c5adf46a4648
> dashboardv2/public/css/fonts/fontawesome-webfont.woff
> dc35ce3c2cf688c89b0bd0d4a82bc4be82b14c40
> dashboardv2/public/css/fonts/fontawesome-webfont.woff2
> 500e5172534171f678e01f7569d66f9257036a09
> dashboardv2/public/css/fonts/glyphicons-halflings-regular.eot
> b93a4953fff68df523aa7656497ee339d6026d64
> dashboardv2/public/css/fonts/glyphicons-halflings-regular.svg 94fb549
> dashboardv2/public/css/fonts/glyphicons-halflings-regular.ttf
> 1413fc609ab6f21774de0cb7e01360095584f65b
> dashboardv2/public/css/fonts/glyphicons-halflings-regular.woff
> 9e612858f802245ddcbf59788a0db942224bab35
> dashboardv2/public/css/fonts/glyphicons-halflings-regular.woff2
> 64539b54c3751a6d9adb44c8e3a45ba5a73b77f0
> dashboardv2/public/css/scss/breadcrumb.scss eaa89e7
> dashboardv2/public/css/scss/business-catlog.scss 824bf10
> dashboardv2/public/css/scss/form.scss 82bd960
> dashboardv2/public/css/scss/graph.scss 7c6571c
> dashboardv2/public/css/scss/loader.scss 6c255c6
> dashboardv2/public/css/scss/login.scss c7237cc
> dashboardv2/public/css/scss/main.scss e9fb452
> dashboardv2/public/css/scss/nav.scss 7277ae1
> dashboardv2/public/css/scss/old-style.scss f62c494
> dashboardv2/public/css/scss/override.scss 6fc1e1f
> dashboardv2/public/css/scss/panel.scss ee85715
> dashboardv2/public/css/scss/search.scss 858b974
> dashboardv2/public/css/scss/tab.scss 5117efe
> dashboardv2/public/css/scss/tag.scss 1a07ce2
> dashboardv2/public/css/scss/theme.scss b05f7a2
> dashboardv2/public/css/scss/tree.scss e6c8ebf
> dashboardv2/public/index.html 59e2c7e
> dashboardv2/public/js/main.js 95f64a1
> dashboardv2/public/js/views/business_catalog/MoveTermLayoutView.js 3add56a
> dashboardv2/public/js/views/search/SearchLayoutView.js 083e515
> pom.xml 4e7d4fd
>
> Diff: https://reviews.apache.org/r/49357/diff/
>
>
> Testing
> ---
>
> After removing glyphicon --> did manual testing of UI using users of all 3
> groups.
>
> Did CRUD operations for Terms, Tags and verified Lineage graph generation
> using hive queries.
>
>
> Thanks,
>
> keval bhatt
>
>
Re: Review Request 49350: ATLAS-917 Add HDFS paths for non-partition queries
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49350/#review139964
---
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java (line
746)
<https://reviews.apache.org/r/49350/#comment205304>
There is only one {}, but two arguments. So, the 2nd argument will not be
logged
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java (line
782)
<https://reviews.apache.org/r/49350/#comment205327>
The loop is over inputs, so the input qualified table names that are added
to qualified name are not sorted anymore.
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java (line
790)
<https://reviews.apache.org/r/49350/#comment205325>
We are not adding hdfs paths in qualified name for partitioned tables, but
we add it to inputs for lineage. With process update, lineage will get updated,
right?
- Shwetha GS
On June 29, 2016, 4:41 a.m., Suma Shivaprasad wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49350/
> ---
>
> (Updated June 29, 2016, 4:41 a.m.)
>
>
> Review request for atlas.
>
>
> Repository: atlas
>
>
> Description
> ---
>
> this is added for LOAD, IMPORT, EXPORT . Could not add to INSERT OVERWRITE
> hdfs paths due to some hive issues which does not provide partition related
> information in this case. That will continue to work as earlier.
>
>
> Diffs
> -
>
>
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
> 0045780
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
> a1a00b3
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java
> f9e1926
> notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 8bbe2d7
>
> Diff: https://reviews.apache.org/r/49350/diff/
>
>
> Testing
> ---
>
> Added tests for partitioned and unpartitioned export, import, load
>
>
> Thanks,
>
> Suma Shivaprasad
>
>
Re: Review Request 49357: ATLAS-958 : Add licenses for dashboard dependencies
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49357/#review139962
---
1. Remove glyphicons files
2. Add licenses in LICENSE in this format:
This product bundles Font Awesome which is available under a
SIL Open Font License (OFL). For details, see 3party-licenses/SILOFL-LICENSE
(replace version)
3. Add apache headers in all .scss files in dashboardv2/public/css/scss. Its in
style.scss, but not others
- Shwetha GS
On June 29, 2016, 7:43 a.m., keval bhatt wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49357/
> ---
>
> (Updated June 29, 2016, 7:43 a.m.)
>
>
> Review request for atlas, Gautam Borad, Madhan Neethiraj, Shwetha GS, and
> Hemanth Yamijala.
>
>
> Bugs: ATLAS-958
> https://issues.apache.org/jira/browse/ATLAS-958
>
>
> Repository: atlas
>
>
> Description
> ---
>
> Patch include following changes.
>
> * Removed unused font (glyphicon)
> * Added licence file to all js libraries (added grunt task to copy license
> file to dist folder)
> * Added all libraries license details in {atlas_home}/LICENSE.txt file
> * Removed unused plugin (tree.js)
>
>
> Diffs
> -
>
> LICENSE 8f075f5
> dashboardv2/gruntfile.js 2f30f3b
> dashboardv2/public/js/main.js 95f64a1
> dashboardv2/public/js/views/business_catalog/MoveTermLayoutView.js 3add56a
> dashboardv2/public/js/views/search/SearchLayoutView.js 083e515
>
> Diff: https://reviews.apache.org/r/49357/diff/
>
>
> Testing
> ---
>
> After removing glyphicon --> did manual testing of UI using users of all 3
> groups.
>
> Did CRUD operations for Terms, Tags and verified Lineage graph generation
> using hive queries.
>
>
> Thanks,
>
> keval bhatt
>
>
Re: Review Request 49191: ATLAS-947 Return state information in inputs and outputs lineage API
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49191/ --- (Updated June 24, 2016, 10:12 a.m.) Review request for atlas. Bugs: ATLAS-947 https://issues.apache.org/jira/browse/ATLAS-947 Repository: atlas Description --- State is returned in lineage APIs Diffs (updated) - addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 2464646 repository/src/main/java/org/apache/atlas/discovery/DataSetLineageService.java a2f47f0 repository/src/main/java/org/apache/atlas/discovery/graph/DefaultGraphPersistenceStrategy.java 90718ed repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 43e9f85 repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java a4bdef7 repository/src/main/scala/org/apache/atlas/query/GraphPersistenceStrategies.scala 34d101a repository/src/test/java/org/apache/atlas/discovery/DataSetLineageServiceTest.java 3028dde typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 90b5815 typesystem/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 6d43359 typesystem/src/test/resources/atlas-application.properties 28d8af3 webapp/src/test/java/org/apache/atlas/web/resources/DataSetLineageJerseyResourceIT.java bc4f239 Diff: https://reviews.apache.org/r/49191/diff/ Testing --- UTs Thanks, Shwetha GS
Review Request 49191: ATLAS-947 Return state information in inputs and outputs lineage API
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49191/ --- Review request for atlas. Bugs: ATLAS-947 https://issues.apache.org/jira/browse/ATLAS-947 Repository: atlas Description --- State is returned in lineage APIs Diffs - addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 2464646 repository/src/main/java/org/apache/atlas/discovery/DataSetLineageService.java a2f47f0 repository/src/main/java/org/apache/atlas/discovery/graph/DefaultGraphPersistenceStrategy.java 90718ed repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 43e9f85 repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java a4bdef7 repository/src/main/scala/org/apache/atlas/query/GraphPersistenceStrategies.scala 34d101a repository/src/test/java/org/apache/atlas/discovery/DataSetLineageServiceTest.java 3028dde typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 90b5815 typesystem/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 6d43359 typesystem/src/test/resources/atlas-application.properties 28d8af3 webapp/src/test/java/org/apache/atlas/web/resources/DataSetLineageJerseyResourceIT.java bc4f239 Diff: https://reviews.apache.org/r/49191/diff/ Testing --- UTs Thanks, Shwetha GS
Re: Review Request 48565: ATLAS-856: lazy-load type cache
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48565/#review139016 --- Ship it! Ship It! - Shwetha GS On June 22, 2016, 2:28 a.m., David Kantor wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48565/ > --- > > (Updated June 22, 2016, 2:28 a.m.) > > > Review request for atlas. > > > Bugs: ATLAS-856 > https://issues.apache.org/jira/browse/ATLAS-856 > > > Repository: atlas > > > Description > --- > > ATLAS-856: Implemented StoreBackedTypeCache, an extension of DefaultTypeCache > which loads the requested type from the type store if it is not found in the > cache, and adds it to the cache if it's found in the store. Any attribute > and super types that are required by the requested type are also loaded from > the store if they are not already in the cache. > > > Diffs > - > > common/src/main/java/org/apache/atlas/ApplicationProperties.java > 741e1a6ad5e10315a9a565d69f9838c438c2e18e > repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java > c4c767827014a265471250306f35316a0b4ed67f > > repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java > 5ed9e02eb85ec25b63540cab40d148914d410786 > > repository/src/main/java/org/apache/atlas/repository/typestore/ITypeStore.java > 790c4b31c37e51289585f6f891366a8d8c86b168 > > repository/src/main/java/org/apache/atlas/repository/typestore/StoreBackedTypeCache.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java > 37e7b66e073df294f5e38c79ba33c7ffcb394a8e > > repository/src/test/java/org/apache/atlas/repository/typestore/GraphBackedTypeStoreTest.java > dd530a755231c3d487d9f5104cb0aa7119863266 > > repository/src/test/java/org/apache/atlas/repository/typestore/StoreBackedTypeCacheConfigurationTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/typestore/StoreBackedTypeCacheTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/typestore/StoreBackedTypeCacheTestModule.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/service/StoreBackedTypeCacheMetadataServiceTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/services/DefaultMetadataServiceMockTest.java > effee2a43d8c1e1369b52f1c3899cb882ea52aa1 > typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java > 14f1968c2dd6765793c1c43e3ea1ed5f3a9c8a42 > > typesystem/src/main/java/org/apache/atlas/typesystem/types/cache/DefaultTypeCacheProvider.java > 700bda6463e039b8f6ad2ac75840f9e637a883b0 > > typesystem/src/main/java/org/apache/atlas/typesystem/types/cache/ITypeCacheProvider.java > 55e9ddc463c748d3cd631d11860235dbe1a92714 > typesystem/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java > 06796420361573d41ca084f4c45d7a5f17cf4ff0 > > typesystem/src/test/java/org/apache/atlas/typesystem/types/cache/DefaultTypeCacheProviderTest.java > c426d50f52dfd99835d9fcf1c6f95841ba4b5494 > webapp/src/test/java/org/apache/atlas/web/listeners/TestModule.java > 9cb76d3d1660a0c806296c41d8108acc8925d3d0 > > Diff: https://reviews.apache.org/r/48565/diff/ > > > Testing > --- > > Ran all unit and integration tests with no regressions. > > > Thanks, > > David Kantor > >
Re: Review Request 49022: ATLAS-822 Type updates - don't allow updating supertypes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49022/ --- (Updated June 21, 2016, 12:04 p.m.) Review request for atlas. Changes --- fixed webapp IT Bugs: ATLAS-822 https://issues.apache.org/jira/browse/ATLAS-822 Repository: atlas Description --- Type update will not allow modifying super types Diffs (updated) - addons/falcon-bridge/pom.xml f945a6c addons/hdfs-model/pom.xml 80e74a2 addons/hive-bridge/pom.xml 24ab22f addons/sqoop-bridge/pom.xml 5418885 addons/storm-bridge/pom.xml b51ab01 catalog/pom.xml 3817012 client/pom.xml a6b5a2f notification/pom.xml f4cfdd3 pom.xml ecf76d0 repository/pom.xml 096a4b5 repository/src/test/java/org/apache/atlas/repository/typestore/GraphBackedTypeStoreTest.java 961442b typesystem/pom.xml 219075b typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java 859ec72 typesystem/src/test/java/org/apache/atlas/typesystem/types/HierarchicalTypeTest.java 81da0be webapp/pom.xml ce812c6 webapp/src/test/java/org/apache/atlas/web/resources/TypesJerseyResourceIT.java 477e4bf Diff: https://reviews.apache.org/r/49022/diff/ Testing --- UTs Thanks, Shwetha GS
Review Request 49022: ATLAS-822 Type updates - don't allow updating supertypes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49022/ --- Review request for atlas. Bugs: ATLAS-822 https://issues.apache.org/jira/browse/ATLAS-822 Repository: atlas Description --- Type update will not allow modifying super types Diffs - addons/falcon-bridge/pom.xml f945a6c addons/hdfs-model/pom.xml 80e74a2 addons/hive-bridge/pom.xml 24ab22f addons/sqoop-bridge/pom.xml 5418885 addons/storm-bridge/pom.xml b51ab01 catalog/pom.xml 3817012 client/pom.xml a6b5a2f notification/pom.xml f4cfdd3 pom.xml ecf76d0 repository/pom.xml 096a4b5 repository/src/test/java/org/apache/atlas/repository/typestore/GraphBackedTypeStoreTest.java 961442b typesystem/pom.xml 219075b typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java 859ec72 typesystem/src/test/java/org/apache/atlas/typesystem/types/HierarchicalTypeTest.java 81da0be webapp/pom.xml ce812c6 Diff: https://reviews.apache.org/r/49022/diff/ Testing --- UTs Thanks, Shwetha GS
Re: Review Request 48945: ATLAs-819 All user defined types should have a set of common attributes
> On June 20, 2016, 7:35 p.m., David Kantor wrote: > > webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java, > > line 124 > > <https://reviews.apache.org/r/48945/diff/2/?file=1425255#file1425255line124> > > > > Why has the log level has been changed to INFO in several places > > throught EntityResource? I would think that in a production Atlas > > environment this would generate a lot of log noise. Shouldn't these remain > > at DEBUG? I also changed info logs in GraphBackedMetadataRepository to debug. At the least, we should have API inputs and outputs, else we can't figure out whats happening. If these logs are not relevant for you, you can change the log level to warn in atlas-log4j.xml, anyway its configurable We also need a way of refreshing log levels at runtime so that we can enable debug logs whenever required - Shwetha --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48945/#review138663 ----------- On June 20, 2016, 12:01 p.m., Shwetha GS wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48945/ > --- > > (Updated June 20, 2016, 12:01 p.m.) > > > Review request for atlas. > > > Bugs: ATLAS-819 > https://issues.apache.org/jira/browse/ATLAS-819 > > > Repository: atlas > > > Description > --- > > Added base type with the attributes name, owner and description that any > type can extend and the users can set these attributes. > > > Diffs > - > > > addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/Util/EventUtil.java > c1ccd05 > > addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/bridge/FalconBridge.java > 1621d95 > > addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/event/FalconEvent.java > 37df6da > > addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java > 95f255e > > addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/model/FalconDataModelGenerator.java > 81cd5e0 > > addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/service/AtlasService.java > c92bd43 > > addons/falcon-bridge/src/test/java/org/apache/atlas/falcon/hook/FalconHookIT.java > 8a5736a > > addons/hdfs-model/src/main/scala/org/apache/atlas/fs/model/FSDataModel.scala > cf81406 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java > c956a32 > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java > 5d9950f > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java > 3686fa8 > addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java > 5a175e7 > > addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java > 9db8180 > > addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/model/SqoopDataModelGenerator.java > c3bdbfa > > addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java > b2171c2 > > addons/storm-bridge/src/main/scala/org/apache/atlas/storm/model/StormDataModel.scala > bae4c1a > client/src/main/java/org/apache/atlas/AtlasClient.java 4c88da9 > distro/src/conf/atlas-log4j.xml e14afa3 > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java > dd7dd74 > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java > c385df3 > > repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java > d9e6cb9 > > typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java > 9dc93ae > webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java > 0713d30 > webapp/src/main/java/org/apache/atlas/web/resources/TypesResource.java > 74d89ce > > Diff: https://reviews.apache.org/r/48945/diff/ > > > Testing > --- > > UTs > > > Thanks, > > Shwetha GS > >
Re: Review Request 48945: ATLAs-819 All user defined types should have a set of common attributes
> On June 20, 2016, 10:19 a.m., Hemanth Yamijala wrote: > > addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/model/FalconDataModelGenerator.java, > > line 107 > > <https://reviews.apache.org/r/48945/diff/1/?file=1423907#file1423907line107> > > > > I didn't get why this was removed - possibly something specific to > > Falcon? Likewise, is the USER attribute going to be replaced by OWNER? timestamp was set to current time and didn't have any other significance. Deleted it in favor of atlas system attributes created time and last modified time. user is deleted as owner is set now > On June 20, 2016, 10:19 a.m., Hemanth Yamijala wrote: > > addons/hdfs-model/src/main/scala/org/apache/atlas/fs/model/FSDataModel.scala, > > line 51 > > <https://reviews.apache.org/r/48945/diff/1/?file=1423910#file1423910line51> > > > > This is removed, but should extend Asset? Have made fs_path extend dataset and hdfs_path extend fs_path. Will check with Suma once - Shwetha --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48945/#review138569 ----------- On June 20, 2016, 9:01 a.m., Shwetha GS wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48945/ > --- > > (Updated June 20, 2016, 9:01 a.m.) > > > Review request for atlas. > > > Bugs: ATLAS-819 > https://issues.apache.org/jira/browse/ATLAS-819 > > > Repository: atlas > > > Description > --- > > Added base type with the attributes name, owner and description that any > type can extend and the users can set these attributes. > > > Diffs > - > > > addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/Util/EventUtil.java > c1ccd05 > > addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/bridge/FalconBridge.java > 1621d95 > > addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/event/FalconEvent.java > 37df6da > > addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java > 95f255e > > addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/model/FalconDataModelGenerator.java > 81cd5e0 > > addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/service/AtlasService.java > c92bd43 > > addons/falcon-bridge/src/test/java/org/apache/atlas/falcon/hook/FalconHookIT.java > 8a5736a > > addons/hdfs-model/src/main/scala/org/apache/atlas/fs/model/FSDataModel.scala > cf81406 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java > c956a32 > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java > 5d9950f > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java > 3686fa8 > addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java > 5a175e7 > > addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java > 9db8180 > > addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/model/SqoopDataModelGenerator.java > c3bdbfa > > addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java > b2171c2 > > addons/storm-bridge/src/main/scala/org/apache/atlas/storm/model/StormDataModel.scala > bae4c1a > client/src/main/java/org/apache/atlas/AtlasClient.java 4c88da9 > distro/src/conf/atlas-log4j.xml e14afa3 > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java > dd7dd74 > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java > c385df3 > > repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java > d9e6cb9 > > typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java > 9dc93ae > webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java > 0713d30 > webapp/src/main/java/org/apache/atlas/web/resources/TypesResource.java > 74d89ce > > Diff: https://reviews.apache.org/r/48945/diff/ > > > Testing > --- > > UTs > > > Thanks, > > Shwetha GS > >
Re: Review Request 48945: ATLAs-819 All user defined types should have a set of common attributes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48945/ --- (Updated June 20, 2016, 12:01 p.m.) Review request for atlas. Changes --- Addressed review comments Bugs: ATLAS-819 https://issues.apache.org/jira/browse/ATLAS-819 Repository: atlas Description --- Added base type with the attributes name, owner and description that any type can extend and the users can set these attributes. Diffs (updated) - addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/Util/EventUtil.java c1ccd05 addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/bridge/FalconBridge.java 1621d95 addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/event/FalconEvent.java 37df6da addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java 95f255e addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/model/FalconDataModelGenerator.java 81cd5e0 addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/service/AtlasService.java c92bd43 addons/falcon-bridge/src/test/java/org/apache/atlas/falcon/hook/FalconHookIT.java 8a5736a addons/hdfs-model/src/main/scala/org/apache/atlas/fs/model/FSDataModel.scala cf81406 addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java c956a32 addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 5d9950f addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java 3686fa8 addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 5a175e7 addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 9db8180 addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/model/SqoopDataModelGenerator.java c3bdbfa addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java b2171c2 addons/storm-bridge/src/main/scala/org/apache/atlas/storm/model/StormDataModel.scala bae4c1a client/src/main/java/org/apache/atlas/AtlasClient.java 4c88da9 distro/src/conf/atlas-log4j.xml e14afa3 repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java dd7dd74 repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java c385df3 repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java d9e6cb9 typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java 9dc93ae webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java 0713d30 webapp/src/main/java/org/apache/atlas/web/resources/TypesResource.java 74d89ce Diff: https://reviews.apache.org/r/48945/diff/ Testing --- UTs Thanks, Shwetha GS
Review Request 48945: ATLAs-819 All user defined types should have a set of common attributes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48945/ --- Review request for atlas. Bugs: ATLAS-819 https://issues.apache.org/jira/browse/ATLAS-819 Repository: atlas Description --- Added base type with the attributes name, owner and description that any type can extend and the users can set these attributes. Diffs - addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/Util/EventUtil.java c1ccd05 addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/bridge/FalconBridge.java 1621d95 addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/event/FalconEvent.java 37df6da addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java 95f255e addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/model/FalconDataModelGenerator.java 81cd5e0 addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/service/AtlasService.java c92bd43 addons/falcon-bridge/src/test/java/org/apache/atlas/falcon/hook/FalconHookIT.java 8a5736a addons/hdfs-model/src/main/scala/org/apache/atlas/fs/model/FSDataModel.scala cf81406 addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java 3686fa8 addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/model/SqoopDataModelGenerator.java c3bdbfa addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java b2171c2 addons/storm-bridge/src/main/scala/org/apache/atlas/storm/model/StormDataModel.scala bae4c1a client/src/main/java/org/apache/atlas/AtlasClient.java 4c88da9 distro/src/conf/atlas-log4j.xml e14afa3 repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java dd7dd74 repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java c385df3 repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java d9e6cb9 typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java 9dc93ae webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java 0713d30 webapp/src/main/java/org/apache/atlas/web/resources/TypesResource.java 74d89ce Diff: https://reviews.apache.org/r/48945/diff/ Testing --- UTs Thanks, Shwetha GS
Re: Review Request 48565: ATLAS-856: lazy-load type cache provider
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48565/#review138278 --- repository/src/main/java/org/apache/atlas/repository/typestore/StoreBackedTypeCacheProvider.java (line 205) <https://reviews.apache.org/r/48565/#comment203457> this should have handled NOT_FOUND. So, throwing exception from getTypeFromCacheOrStore() will work in all cases - Shwetha GS On June 15, 2016, 3:05 p.m., David Kantor wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48565/ > --- > > (Updated June 15, 2016, 3:05 p.m.) > > > Review request for atlas. > > > Bugs: ATLAS-856 > https://issues.apache.org/jira/browse/ATLAS-856 > > > Repository: atlas > > > Description > --- > > ATLAS-856: An extension of DefaultTypeCacheProvider which loads the requested > type from the type store if it is not found in the cache, and adds it to the > cache if it's found in the store. Any attribute and super types that are > required by the requested type are also loaded from the store if they are not > already in the cache. > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java > 5ed9e02eb85ec25b63540cab40d148914d410786 > > repository/src/main/java/org/apache/atlas/repository/typestore/ITypeStore.java > 790c4b31c37e51289585f6f891366a8d8c86b168 > > repository/src/main/java/org/apache/atlas/repository/typestore/StoreBackedTypeCacheProvider.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/typestore/GraphBackedTypeStoreTest.java > 961442bbcaff86b472a876e6c348aa9e65c1efca > > repository/src/test/java/org/apache/atlas/repository/typestore/StoreBackedTypeCacheProviderTest.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/48565/diff/ > > > Testing > --- > > Ran all unit and integration tests with no regressions. > > > Thanks, > > David Kantor > >
Re: Review Request 48565: ATLAS-856: lazy-load type cache provider
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48565/#review138277 --- One comment, looks good otherwise repository/src/main/java/org/apache/atlas/repository/typestore/StoreBackedTypeCacheProvider.java (line 166) <https://reviews.apache.org/r/48565/#comment203456> The only return type valid here is NOT_FOUND. Lets throw exception in this case and make the function return void instead - Shwetha GS On June 15, 2016, 3:05 p.m., David Kantor wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48565/ > --- > > (Updated June 15, 2016, 3:05 p.m.) > > > Review request for atlas. > > > Bugs: ATLAS-856 > https://issues.apache.org/jira/browse/ATLAS-856 > > > Repository: atlas > > > Description > --- > > ATLAS-856: An extension of DefaultTypeCacheProvider which loads the requested > type from the type store if it is not found in the cache, and adds it to the > cache if it's found in the store. Any attribute and super types that are > required by the requested type are also loaded from the store if they are not > already in the cache. > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java > 5ed9e02eb85ec25b63540cab40d148914d410786 > > repository/src/main/java/org/apache/atlas/repository/typestore/ITypeStore.java > 790c4b31c37e51289585f6f891366a8d8c86b168 > > repository/src/main/java/org/apache/atlas/repository/typestore/StoreBackedTypeCacheProvider.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/typestore/GraphBackedTypeStoreTest.java > 961442bbcaff86b472a876e6c348aa9e65c1efca > > repository/src/test/java/org/apache/atlas/repository/typestore/StoreBackedTypeCacheProviderTest.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/48565/diff/ > > > Testing > --- > > Ran all unit and integration tests with no regressions. > > > Thanks, > > David Kantor > >
Re: Review Request 48845: ATLAS-835 Falcon Integration with Atlas
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48845/ --- (Updated June 17, 2016, 6:13 p.m.) Review request for atlas. Changes --- fixed storm error Bugs: ATLAS-835 https://issues.apache.org/jira/browse/ATLAS-835 Repository: atlas Description --- Falcon Integration with Atlas Diffs (updated) - addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/Util/EventUtil.java PRE-CREATION addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/bridge/FalconBridge.java PRE-CREATION addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/event/FalconEvent.java PRE-CREATION addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java f27a8b0 addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/model/FalconDataModelGenerator.java 397dea4 addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/model/FalconDataTypes.java f1f350b addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/publisher/FalconEventPublisher.java PRE-CREATION addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/service/AtlasService.java PRE-CREATION addons/falcon-bridge/src/main/java/org/apache/falcon/atlas/Util/EventUtil.java 7f67407 addons/falcon-bridge/src/main/java/org/apache/falcon/atlas/event/FalconEvent.java e587e73 addons/falcon-bridge/src/main/java/org/apache/falcon/atlas/publisher/FalconEventPublisher.java 8029be9 addons/falcon-bridge/src/main/java/org/apache/falcon/atlas/service/AtlasService.java 373846d addons/falcon-bridge/src/test/java/org/apache/atlas/falcon/hook/FalconHookIT.java 7adccef addons/falcon-bridge/src/test/resources/feed-replication.xml PRE-CREATION addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 0ccb18b addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 00c17e8 distro/src/conf/atlas-log4j.xml 600b4f1 docs/src/site/twiki/Bridge-Falcon.twiki 4f5e676 notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 2ca8d85 notification/src/test/java/org/apache/atlas/hook/AtlasHookTest.java 9854bcc typesystem/src/main/resources/atlas-application.properties 11253e6 typesystem/src/main/resources/atlas-log4j.xml 2bb49d3 webapp/src/main/java/org/apache/atlas/web/filters/AuditFilter.java eeaddd6 Diff: https://reviews.apache.org/r/48845/diff/ Testing --- UTs and end to end test Thanks, Shwetha GS
Re: Review Request 48845: ATLAS-835 Falcon Integration with Atlas
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48845/ --- (Updated June 17, 2016, 5:48 p.m.) Review request for atlas. Changes --- Addressed review comments Bugs: ATLAS-835 https://issues.apache.org/jira/browse/ATLAS-835 Repository: atlas Description --- Falcon Integration with Atlas Diffs (updated) - addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/Util/EventUtil.java PRE-CREATION addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/bridge/FalconBridge.java PRE-CREATION addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/event/FalconEvent.java PRE-CREATION addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java f27a8b0 addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/model/FalconDataModelGenerator.java 397dea4 addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/model/FalconDataTypes.java f1f350b addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/publisher/FalconEventPublisher.java PRE-CREATION addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/service/AtlasService.java PRE-CREATION addons/falcon-bridge/src/main/java/org/apache/falcon/atlas/Util/EventUtil.java 7f67407 addons/falcon-bridge/src/main/java/org/apache/falcon/atlas/event/FalconEvent.java e587e73 addons/falcon-bridge/src/main/java/org/apache/falcon/atlas/publisher/FalconEventPublisher.java 8029be9 addons/falcon-bridge/src/main/java/org/apache/falcon/atlas/service/AtlasService.java 373846d addons/falcon-bridge/src/test/java/org/apache/atlas/falcon/hook/FalconHookIT.java 7adccef addons/falcon-bridge/src/test/resources/feed-replication.xml PRE-CREATION addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 0ccb18b addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 00c17e8 distro/src/conf/atlas-log4j.xml 600b4f1 docs/src/site/twiki/Bridge-Falcon.twiki 4f5e676 notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 2ca8d85 notification/src/test/java/org/apache/atlas/hook/AtlasHookTest.java 9854bcc pom.xml 6f75740 typesystem/src/main/resources/atlas-application.properties 11253e6 typesystem/src/main/resources/atlas-log4j.xml 2bb49d3 webapp/src/main/java/org/apache/atlas/web/filters/AuditFilter.java eeaddd6 Diff: https://reviews.apache.org/r/48845/diff/ Testing --- UTs and end to end test Thanks, Shwetha GS
Re: Review Request 48845: ATLAS-835 Falcon Integration with Atlas
> On June 17, 2016, 12:03 p.m., Hemanth Yamijala wrote: > > addons/storm-bridge/pom.xml, line 102 > > <https://reviews.apache.org/r/48845/diff/2/?file=1422190#file1422190line102> > > > > Why this change? These dependency changes might need to be tested in > > the host component environments. Also, I see we have a dependency exclusion > > when specifying storm dependency pointing to slf4j-log4j12, should that > > change? slf4j-log4j12 and log4j-over-slf4j can't both co-exist. The build was failing because of that. Since we already depend on slf4j-log4j12, we have to exclude log4j-over-slf4j. Not sure why this didn't come up earlier. Will revert for now and check - Shwetha --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48845/#review138203 ------- On June 17, 2016, 10:47 a.m., Shwetha GS wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48845/ > --- > > (Updated June 17, 2016, 10:47 a.m.) > > > Review request for atlas. > > > Bugs: ATLAS-835 > https://issues.apache.org/jira/browse/ATLAS-835 > > > Repository: atlas > > > Description > --- > > Falcon Integration with Atlas > > > Diffs > - > > > addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/Util/EventUtil.java > PRE-CREATION > > addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/bridge/FalconBridge.java > PRE-CREATION > > addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/event/FalconEvent.java > PRE-CREATION > > addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java > f27a8b0 > > addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/model/FalconDataModelGenerator.java > 397dea4 > > addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/model/FalconDataTypes.java > f1f350b > > addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/publisher/FalconEventPublisher.java > PRE-CREATION > > addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/service/AtlasService.java > PRE-CREATION > > addons/falcon-bridge/src/main/java/org/apache/falcon/atlas/Util/EventUtil.java > 7f67407 > > addons/falcon-bridge/src/main/java/org/apache/falcon/atlas/event/FalconEvent.java > e587e73 > > addons/falcon-bridge/src/main/java/org/apache/falcon/atlas/publisher/FalconEventPublisher.java > 8029be9 > > addons/falcon-bridge/src/main/java/org/apache/falcon/atlas/service/AtlasService.java > 373846d > > addons/falcon-bridge/src/test/java/org/apache/atlas/falcon/hook/FalconHookIT.java > 7adccef > addons/falcon-bridge/src/test/resources/feed-replication.xml PRE-CREATION > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java > 0ccb18b > addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java > 00c17e8 > addons/storm-bridge/pom.xml b1a7a9b > distro/src/conf/atlas-log4j.xml 600b4f1 > docs/src/site/twiki/Bridge-Falcon.twiki 4f5e676 > notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 2ca8d85 > pom.xml 6f75740 > typesystem/src/main/resources/atlas-application.properties 11253e6 > typesystem/src/main/resources/atlas-log4j.xml 2bb49d3 > webapp/src/main/java/org/apache/atlas/web/filters/AuditFilter.java eeaddd6 > > Diff: https://reviews.apache.org/r/48845/diff/ > > > Testing > --- > > UTs and end to end test > > > Thanks, > > Shwetha GS > >
Re: Review Request 48845: ATLAS-835 Falcon Integration with Atlas
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48845/ --- (Updated June 17, 2016, 10:47 a.m.) Review request for atlas. Changes --- re-based patch Bugs: ATLAS-835 https://issues.apache.org/jira/browse/ATLAS-835 Repository: atlas Description --- Falcon Integration with Atlas Diffs (updated) - addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/Util/EventUtil.java PRE-CREATION addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/bridge/FalconBridge.java PRE-CREATION addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/event/FalconEvent.java PRE-CREATION addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java f27a8b0 addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/model/FalconDataModelGenerator.java 397dea4 addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/model/FalconDataTypes.java f1f350b addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/publisher/FalconEventPublisher.java PRE-CREATION addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/service/AtlasService.java PRE-CREATION addons/falcon-bridge/src/main/java/org/apache/falcon/atlas/Util/EventUtil.java 7f67407 addons/falcon-bridge/src/main/java/org/apache/falcon/atlas/event/FalconEvent.java e587e73 addons/falcon-bridge/src/main/java/org/apache/falcon/atlas/publisher/FalconEventPublisher.java 8029be9 addons/falcon-bridge/src/main/java/org/apache/falcon/atlas/service/AtlasService.java 373846d addons/falcon-bridge/src/test/java/org/apache/atlas/falcon/hook/FalconHookIT.java 7adccef addons/falcon-bridge/src/test/resources/feed-replication.xml PRE-CREATION addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 0ccb18b addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 00c17e8 addons/storm-bridge/pom.xml b1a7a9b distro/src/conf/atlas-log4j.xml 600b4f1 docs/src/site/twiki/Bridge-Falcon.twiki 4f5e676 notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 2ca8d85 pom.xml 6f75740 typesystem/src/main/resources/atlas-application.properties 11253e6 typesystem/src/main/resources/atlas-log4j.xml 2bb49d3 webapp/src/main/java/org/apache/atlas/web/filters/AuditFilter.java eeaddd6 Diff: https://reviews.apache.org/r/48845/diff/ Testing --- UTs and end to end test Thanks, Shwetha GS
Review Request 48845: ATLAS-835 Falcon Integration with Atlas
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48845/ --- Review request for atlas. Bugs: ATLAS-835 https://issues.apache.org/jira/browse/ATLAS-835 Repository: atlas Description --- Falcon Integration with Atlas Diffs - addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/Util/EventUtil.java PRE-CREATION addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/bridge/FalconBridge.java PRE-CREATION addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/event/FalconEvent.java PRE-CREATION addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java f27a8b0 addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/model/FalconDataModelGenerator.java 397dea4 addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/model/FalconDataTypes.java f1f350b addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/publisher/FalconEventPublisher.java PRE-CREATION addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/service/AtlasService.java PRE-CREATION addons/falcon-bridge/src/main/java/org/apache/falcon/atlas/Util/EventUtil.java 7f67407 addons/falcon-bridge/src/main/java/org/apache/falcon/atlas/event/FalconEvent.java e587e73 addons/falcon-bridge/src/main/java/org/apache/falcon/atlas/publisher/FalconEventPublisher.java 8029be9 addons/falcon-bridge/src/main/java/org/apache/falcon/atlas/service/AtlasService.java 373846d addons/falcon-bridge/src/test/java/org/apache/atlas/falcon/hook/FalconHookIT.java 7adccef addons/falcon-bridge/src/test/resources/feed-replication.xml PRE-CREATION addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 0ccb18b addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 00c17e8 addons/storm-bridge/pom.xml b1a7a9b distro/src/conf/atlas-log4j.xml 600b4f1 docs/src/site/twiki/Bridge-Falcon.twiki 4f5e676 notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 71029b0 pom.xml 6f75740 typesystem/src/main/resources/atlas-application.properties 11253e6 typesystem/src/main/resources/atlas-log4j.xml 2bb49d3 webapp/src/main/java/org/apache/atlas/web/filters/AuditFilter.java eeaddd6 Diff: https://reviews.apache.org/r/48845/diff/ Testing --- UTs and end to end test Thanks, Shwetha GS
Review Request 48837: ATLAS-911 Get entity by unique attribute doesn't enforce type
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48837/ --- Review request for atlas. Bugs: ATLAS-911 https://issues.apache.org/jira/browse/ATLAS-911 Repository: atlas Description --- Currently, get entity by unique attribute just filters on the attribute property. Since the attribute name contains the typename as well, type filter was not added. But if multiple types have the same super type, and the super type has the unique attribute, the attribute name will be same for multiple types. With this, the type returned will not be same as the type requested. We should add filter on the type as well Diffs - repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java 0d82d90 repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 4f6d011 repository/src/test/java/org/apache/atlas/TestUtils.java 345e874 repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java 5e7de88 repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java 449e066 repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java 2d1c33a Diff: https://reviews.apache.org/r/48837/diff/ Testing --- UTs Thanks, Shwetha GS
Re: Review Request 48349: ATLAS-871: Add mechanism to disable HBaseBasedEntityAuditRepository
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48349/#review136971 --- Ship it! Ship It! - Shwetha GS On June 9, 2016, 3:45 p.m., Jeff Hagelberg wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48349/ > --- > > (Updated June 9, 2016, 3:45 p.m.) > > > Review request for atlas and David Kantor. > > > Bugs: ATLAS-871 > https://issues.apache.org/jira/browse/ATLAS-871 > > > Repository: atlas > > > Description > --- > > This task adds the ability to disable the HBaseBasedEntityAuditRepository and > use Atlas in environments where hbase is not available. This is needed > because some graph database implementations do not use or support hbase. It > also makes development easier in environments where hbase is not available. > > This task does the following: > - add new optional configuration property to specify the audit repository > implementation > - adds new NoopEntityAuditRepository that can be used in environments where > a suitable EntityAuditRepository has not been implemented yet (or where there > is not an auditing requirement) > - Updates atlas_start.py so that it does not fail if hbase is not selected. > - updated default entity repository used in tests to > InMemoryEntityAuditRepository. If the "distributed" profile is enabled, the > HBase EntityAuditRepository is used in the tests. > - Updated distro logic so that if the berkeley-elasticsearch profile is > enabled, the no-op EntityAuditRepository is selected by default in > atlas-application.properties > > > Diffs > - > > distro/pom.xml 131a4882f4728f5c9df8a50e283463f8aa35ddbd > distro/src/bin/atlas_start.py 1ba38f8683c4274475fd097ff182c5cfa8db1dd4 > distro/src/conf/atlas-application.properties > 1cdd424d8e126a6bdbf99db8bf4e5d4329e806bb > pom.xml eac95c8526011a3a86de8dd154545abec0f7bcf5 > repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java > 15201dd45913539b9b3bd430b46ce6cf30c327ac > > repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java > 93f224fa01c494e5765ccbdb691aa950111a8c20 > > repository/src/main/java/org/apache/atlas/repository/audit/NoopEntityAuditRepository.java > PRE-CREATION > typesystem/src/main/resources/atlas-application.properties > 3318ff306a8b8ab5e320129a8450d144215f009f > > Diff: https://reviews.apache.org/r/48349/diff/ > > > Testing > --- > > Ran Atlas build and integration tests, no regressions found. > > > Thanks, > > Jeff Hagelberg > >
Re: Review Request 47658: ATLAS-499: Update reverse references
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47658/#review136791 --- The following tests failed for me with this patch. Did they work for you? Failed tests: HiveHookIT.testAlterTablePartitionColumnType:911->assertColumnIsRegistered:276->assertEntityIsRegistered:1498->waitFor:1591 Assertions failed. Failing after waiting for timeout 8 msecs HiveHookIT.testAlterTableRename:659->assertColumnIsRegistered:271->assertColumnIsRegistered:276->assertEntityIsRegistered:1498->waitFor:1591 Assertions failed. Failing after waiting for timeout 8 msecs Tests run: 43, Failures: 2, Errors: 0, Skipped: 0 - Shwetha GS On June 6, 2016, 10:53 p.m., David Kantor wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/47658/ > --- > > (Updated June 6, 2016, 10:53 p.m.) > > > Review request for atlas and Shwetha GS. > > > Bugs: ATLAS-499 > https://issues.apache.org/jira/browse/ATLAS-499 > > > Repository: atlas > > > Description > --- > > ATLAS-499: Automatically update inverse references to prevent repository > corruption from unbalanced references. > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java > 91f9bd008a6939dfe78656f5c1845637a30ee9eb > repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java > 4f6d0118072380342f5ea302d0aaf7902784849a > > repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java > 4c1f5591f4adead41c3336e64b0bc956836f7edb > > repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java > 449e066a167ba4546b118a77c8e3de5fd99f077b > > repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateHardDeleteTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateSoftDeleteTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateTestBase.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/47658/diff/ > > > Testing > --- > > Ran all unit and integration tests with no regressions. > > > Thanks, > > David Kantor > >
Re: Review Request 47658: ATLAS-499: Update reverse references
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47658/#review136790 --- Looks good to me. Suma, do you have any other comments - Shwetha GS On June 6, 2016, 10:53 p.m., David Kantor wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/47658/ > --- > > (Updated June 6, 2016, 10:53 p.m.) > > > Review request for atlas and Shwetha GS. > > > Bugs: ATLAS-499 > https://issues.apache.org/jira/browse/ATLAS-499 > > > Repository: atlas > > > Description > --- > > ATLAS-499: Automatically update inverse references to prevent repository > corruption from unbalanced references. > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java > 91f9bd008a6939dfe78656f5c1845637a30ee9eb > repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java > 4f6d0118072380342f5ea302d0aaf7902784849a > > repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java > 4c1f5591f4adead41c3336e64b0bc956836f7edb > > repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java > 449e066a167ba4546b118a77c8e3de5fd99f077b > > repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateHardDeleteTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateSoftDeleteTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateTestBase.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/47658/diff/ > > > Testing > --- > > Ran all unit and integration tests with no regressions. > > > Thanks, > > David Kantor > >
Re: Review Request 48408: ATLAS-749 Make EntityAuditRepository implementation configurable
> On June 8, 2016, 2:08 p.m., Jeff Hagelberg wrote: > > One minor comment - there is really no point in accumulating the audit > > repository events in memory in production. I could see it potentially using > > a lot of memory, since it is never cleaned up. It may make sense to add a > > Noop implementation that can be used in production environments where hbase > > is unavailable. > > Jeff Hagelberg wrote: > One more comment - atlas_start.py currently fails if hbase is not > selected as the repository. My changes for ATLAS-749 included logic to > tolerate something other than hbase being used as the repository. I also > added the delete handler property to atlas-application.properties, since it > was missing. You can see the changes I made here: > https://reviews.apache.org/r/48349/ Yes, in-meory is bad, I added it as it was already there. With no-op implementation, audit functionality will not work. The default implementation should use some persistent store, without hbase, it can probably be derby. But any other implementation, lets go with no-op for now. I will close this, lets get your patch in. Can you re-open your review and add atlas group in the review request - Shwetha --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48408/#review136641 ----------- On June 8, 2016, 11:41 a.m., Shwetha GS wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48408/ > --- > > (Updated June 8, 2016, 11:41 a.m.) > > > Review request for atlas. > > > Bugs: ATLAS-749 > https://issues.apache.org/jira/browse/ATLAS-749 > > > Repository: atlas > > > Description > --- > > 1. Makes audit repository implementation configurable > 2. Made in memory audit repo as default for berkley, es profile > 3. Updated docs > > > Diffs > - > > distro/pom.xml 131a488 > distro/src/conf/atlas-application.properties 1cdd424 > docs/src/site/twiki/Configuration.twiki 7150483 > repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java > 15201dd > typesystem/src/main/resources/atlas-application.properties 3318ff3 > webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java > 010fa2a > > webapp/src/test/java/org/apache/atlas/web/listeners/TestGuiceServletConfig.java > 08bb125 > webapp/src/test/java/org/apache/atlas/web/listeners/TestModule.java 9cb76d3 > > Diff: https://reviews.apache.org/r/48408/diff/ > > > Testing > --- > > Existing tests passed, tested end to end > > > Thanks, > > Shwetha GS > >
Review Request 48408: ATLAS-749 Make EntityAuditRepository implementation configurable
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48408/ --- Review request for atlas. Bugs: ATLAS-749 https://issues.apache.org/jira/browse/ATLAS-749 Repository: atlas Description --- 1. Makes audit repository implementation configurable 2. Made in memory audit repo as default for berkley, es profile 3. Updated docs Diffs - distro/pom.xml 131a488 distro/src/conf/atlas-application.properties 1cdd424 docs/src/site/twiki/Configuration.twiki 7150483 repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 15201dd typesystem/src/main/resources/atlas-application.properties 3318ff3 webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java 010fa2a webapp/src/test/java/org/apache/atlas/web/listeners/TestGuiceServletConfig.java 08bb125 webapp/src/test/java/org/apache/atlas/web/listeners/TestModule.java 9cb76d3 Diff: https://reviews.apache.org/r/48408/diff/ Testing --- Existing tests passed, tested end to end Thanks, Shwetha GS
Re: Review Request 47782: Atlas should use external HBase and SOLR
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47782/#review135220 --- Ship it! Since the default build works, committing this. Filed https://issues.apache.org/jira/browse/ATLAS-832 to track starting hbase and solr with atlas_start with 'embedded-services' profile - Shwetha GS On May 26, 2016, 7:14 p.m., Tom Beerbower wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/47782/ > --- > > (Updated May 26, 2016, 7:14 p.m.) > > > Review request for atlas and Hemanth Yamijala. > > > Bugs: ATLAS-823 > https://issues.apache.org/jira/browse/ATLAS-823 > > > Repository: atlas > > > Description > --- > > Use of embedded HBase and SOLR is not preferred due to multiple concerns, > including: > > 1. lack of support by Ambari makes it difficult to handle kerberization, > upgrade/migration, start/stop > 2. not suitable for production use - no HA, local-storage > > Atlas should remove support for embedded HBase and SOLR. Instead Atlas should > be updated to use external HBase and SOLR. > > > Diffs > - > > distro/pom.xml 1ab092d > distro/src/bin/atlas_config.py 8f3479c > distro/src/bin/atlas_start.py cb869bb > distro/src/conf/atlas-application.properties 26f84e9 > distro/src/conf/atlas-env.sh 4f1f46c > distro/src/main/assemblies/standalone-package.xml 09ff47d > docs/src/site/twiki/Configuration.twiki 879c06c > docs/src/site/twiki/InstallationSteps.twiki 006e1e8 > > Diff: https://reviews.apache.org/r/47782/diff/ > > > Testing > --- > > updated unit test > > mvn clean test > > > Thanks, > > Tom Beerbower > >
Re: Review Request 47782: Atlas should use external HBase and SOLR
On May 26, 2016, 6:04 p.m., Tom Beerbower wrote:
> > In the package built using '-Pdist,embedded_services', atlas_start doesn't
> > start embedded hbase and solr. is that expected?
>
> Tom Beerbower wrote:
> You have to run atlas-env.sh first.
>
> Shwetha GS wrote:
> Can't atlas_start.py and atlas_stop.py in trun run atlas-env.sh, instead
> of adding this as an extra manual step?
>
> Tom Beerbower wrote:
> I guess that it could but atlas-env.sh isn't new. It has always been a
> requirement to run it first.
>
> Shwetha GS wrote:
> From InstallationSteps.twiki:
> atlas-env.sh has been added to the Atlas conf. This file can be used to
> set various environment
> variables that you need for you services. In addition you can set any
> other environment
> variables you might need. This file will be sourced by atlas scripts
> before any commands are
> executed.
>
> I think the intention was to run atlas-env.sh as part of atlas_start
> always, but was missed. Thats how it is even in hive:
> bin$ grep -r hive-env.sh *
> hive:if [ -f "${HIVE_CONF_DIR}/hive-env.sh" ]; then
> hive: . "${HIVE_CONF_DIR}/hive-env.sh"
>
> Your call if you want to address that as part of another jira
Running atlas-env.sh before atlas_start didn't start hbase and solr:
apache-atlas-0.7-incubating-SNAPSHOT$ ./conf/atlas-env.sh
apache-atlas-0.7-incubating-SNAPSHOT$ ./bin/atlas_start.py
Apache Atlas Server started!!!
- Shwetha
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47782/#review135028
---
On May 26, 2016, 7:14 p.m., Tom Beerbower wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47782/
> ---
>
> (Updated May 26, 2016, 7:14 p.m.)
>
>
> Review request for atlas and Hemanth Yamijala.
>
>
> Bugs: ATLAS-823
> https://issues.apache.org/jira/browse/ATLAS-823
>
>
> Repository: atlas
>
>
> Description
> ---
>
> Use of embedded HBase and SOLR is not preferred due to multiple concerns,
> including:
>
> 1. lack of support by Ambari makes it difficult to handle kerberization,
> upgrade/migration, start/stop
> 2. not suitable for production use - no HA, local-storage
>
> Atlas should remove support for embedded HBase and SOLR. Instead Atlas should
> be updated to use external HBase and SOLR.
>
>
> Diffs
> -
>
> distro/pom.xml 1ab092d
> distro/src/bin/atlas_config.py 8f3479c
> distro/src/bin/atlas_start.py cb869bb
> distro/src/conf/atlas-application.properties 26f84e9
> distro/src/conf/atlas-env.sh 4f1f46c
> distro/src/main/assemblies/standalone-package.xml 09ff47d
> docs/src/site/twiki/Configuration.twiki 879c06c
> docs/src/site/twiki/InstallationSteps.twiki 006e1e8
>
> Diff: https://reviews.apache.org/r/47782/diff/
>
>
> Testing
> ---
>
> updated unit test
>
> mvn clean test
>
>
> Thanks,
>
> Tom Beerbower
>
>
Re: Review Request 47782: Atlas should use external HBase and SOLR
On May 26, 2016, 6:04 p.m., Tom Beerbower wrote:
> > In the package built using '-Pdist,embedded_services', atlas_start doesn't
> > start embedded hbase and solr. is that expected?
>
> Tom Beerbower wrote:
> You have to run atlas-env.sh first.
>
> Shwetha GS wrote:
> Can't atlas_start.py and atlas_stop.py in trun run atlas-env.sh, instead
> of adding this as an extra manual step?
>
> Tom Beerbower wrote:
> I guess that it could but atlas-env.sh isn't new. It has always been a
> requirement to run it first.
>From InstallationSteps.twiki:
atlas-env.sh has been added to the Atlas conf. This file can be used to set
various environment
variables that you need for you services. In addition you can set any other
environment
variables you might need. This file will be sourced by atlas scripts before any
commands are
executed.
I think the intention was to run atlas-env.sh as part of atlas_start always,
but was missed. Thats how it is even in hive:
bin$ grep -r hive-env.sh *
hive:if [ -f "${HIVE_CONF_DIR}/hive-env.sh" ]; then
hive: . "${HIVE_CONF_DIR}/hive-env.sh"
Your call if you want to address that as part of another jira
- Shwetha
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47782/#review135028
---
On May 26, 2016, 7:14 p.m., Tom Beerbower wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47782/
> ---
>
> (Updated May 26, 2016, 7:14 p.m.)
>
>
> Review request for atlas and Hemanth Yamijala.
>
>
> Bugs: ATLAS-823
> https://issues.apache.org/jira/browse/ATLAS-823
>
>
> Repository: atlas
>
>
> Description
> ---
>
> Use of embedded HBase and SOLR is not preferred due to multiple concerns,
> including:
>
> 1. lack of support by Ambari makes it difficult to handle kerberization,
> upgrade/migration, start/stop
> 2. not suitable for production use - no HA, local-storage
>
> Atlas should remove support for embedded HBase and SOLR. Instead Atlas should
> be updated to use external HBase and SOLR.
>
>
> Diffs
> -
>
> distro/pom.xml 1ab092d
> distro/src/bin/atlas_config.py 8f3479c
> distro/src/bin/atlas_start.py cb869bb
> distro/src/conf/atlas-application.properties 26f84e9
> distro/src/conf/atlas-env.sh 4f1f46c
> distro/src/main/assemblies/standalone-package.xml 09ff47d
> docs/src/site/twiki/Configuration.twiki 879c06c
> docs/src/site/twiki/InstallationSteps.twiki 006e1e8
>
> Diff: https://reviews.apache.org/r/47782/diff/
>
>
> Testing
> ---
>
> updated unit test
>
> mvn clean test
>
>
> Thanks,
>
> Tom Beerbower
>
>
Re: Review Request 47782: Atlas should use external HBase and SOLR
On May 26, 2016, 6:04 p.m., Tom Beerbower wrote: > > In the package built using '-Pdist,embedded_services', atlas_start doesn't > > start embedded hbase and solr. is that expected? > > Tom Beerbower wrote: > You have to run atlas-env.sh first. Can't atlas_start.py and atlas_stop.py in trun run atlas-env.sh, instead of adding this as an extra manual step? - Shwetha --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47782/#review135028 --- On May 26, 2016, 7:14 p.m., Tom Beerbower wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/47782/ > --- > > (Updated May 26, 2016, 7:14 p.m.) > > > Review request for atlas and Hemanth Yamijala. > > > Bugs: ATLAS-823 > https://issues.apache.org/jira/browse/ATLAS-823 > > > Repository: atlas > > > Description > --- > > Use of embedded HBase and SOLR is not preferred due to multiple concerns, > including: > > 1. lack of support by Ambari makes it difficult to handle kerberization, > upgrade/migration, start/stop > 2. not suitable for production use - no HA, local-storage > > Atlas should remove support for embedded HBase and SOLR. Instead Atlas should > be updated to use external HBase and SOLR. > > > Diffs > - > > distro/pom.xml 1ab092d > distro/src/bin/atlas_config.py 8f3479c > distro/src/bin/atlas_start.py cb869bb > distro/src/conf/atlas-application.properties 26f84e9 > distro/src/conf/atlas-env.sh 4f1f46c > distro/src/main/assemblies/standalone-package.xml 09ff47d > docs/src/site/twiki/Configuration.twiki 879c06c > docs/src/site/twiki/InstallationSteps.twiki 006e1e8 > > Diff: https://reviews.apache.org/r/47782/diff/ > > > Testing > --- > > updated unit test > > mvn clean test > > > Thanks, > > Tom Beerbower > >
Re: Review Request 47782: Atlas should use external HBase and SOLR
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47782/#review135028 --- distro/src/conf/hbase_solr-atlas-application.properties (line 2) <https://reviews.apache.org/r/47782/#comment200015> Instead of adding new files, where configs can go out of sync. Can you use single file and use resource filtering to populate the values correctly. Have added ATLAS-823-modified.patch on the jira, check if it makes sense In the package built using '-Pdist,embedded_services', atlas_start doesn't start embedded hbase and solr. is that expected? - Shwetha GS On May 26, 2016, 2:17 p.m., Tom Beerbower wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/47782/ > --- > > (Updated May 26, 2016, 2:17 p.m.) > > > Review request for atlas and Hemanth Yamijala. > > > Bugs: ATLAS-823 > https://issues.apache.org/jira/browse/ATLAS-823 > > > Repository: atlas > > > Description > --- > > Use of embedded HBase and SOLR is not preferred due to multiple concerns, > including: > > 1. lack of support by Ambari makes it difficult to handle kerberization, > upgrade/migration, start/stop > 2. not suitable for production use - no HA, local-storage > > Atlas should remove support for embedded HBase and SOLR. Instead Atlas should > be updated to use external HBase and SOLR. > > > Diffs > - > > distro/pom.xml 1ab092d > distro/src/bin/atlas_config.py 8f3479c > distro/src/bin/atlas_start.py cb869bb > distro/src/conf/atlas-application.properties 26f84e9 > distro/src/conf/hbase_solr-atlas-application.properties PRE-CREATION > distro/src/conf/hbase_solr-atlas-env.sh PRE-CREATION > distro/src/main/assemblies/standalone-package.xml 09ff47d > docs/src/site/twiki/Configuration.twiki 879c06c > docs/src/site/twiki/InstallationSteps.twiki 006e1e8 > > Diff: https://reviews.apache.org/r/47782/diff/ > > > Testing > --- > > updated unit test > > mvn clean test > > > Thanks, > > Tom Beerbower > >
Re: Review Request 47782: Atlas should use external HBase and SOLR
> On May 25, 2016, 10:28 a.m., Shwetha GS wrote: > > 1. Change distro/atlas-application.properties to use berkleydb and > > elasticsearch > > 2. Remove packaging of hbase and solr > > Tom Beerbower wrote: > So we want to completely revert the embedded HBase and SOLR patches? > > Madhan Neethiraj wrote: > Yes, I think we should remove HBase and Solr from packaging. Alternatively, we can move the packaging to another profile. But we should revert the config changes, so that default packaging with default configs works - Shwetha --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47782/#review134747 --- On May 24, 2016, 5:42 p.m., Tom Beerbower wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/47782/ > --- > > (Updated May 24, 2016, 5:42 p.m.) > > > Review request for atlas and Hemanth Yamijala. > > > Bugs: ATLAS-823 > https://issues.apache.org/jira/browse/ATLAS-823 > > > Repository: atlas > > > Description > --- > > Use of embedded HBase and SOLR is not preferred due to multiple concerns, > including: > > 1. lack of support by Ambari makes it difficult to handle kerberization, > upgrade/migration, start/stop > 2. not suitable for production use - no HA, local-storage > > Atlas should remove support for embedded HBase and SOLR. Instead Atlas should > be updated to use external HBase and SOLR. > > > Diffs > - > > distro/src/bin/atlas_start.py cb869bb > distro/src/bin/atlas_stop.py a72c724 > distro/src/test/python/scripts/TestMetadata.py 00ca3e2 > docs/src/site/twiki/Configuration.twiki 879c06c > docs/src/site/twiki/InstallationSteps.twiki 006e1e8 > > Diff: https://reviews.apache.org/r/47782/diff/ > > > Testing > --- > > updated unit test > > mvn clean test > > > Thanks, > > Tom Beerbower > >
Re: Review Request 47782: Atlas should use external HBase and SOLR
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47782/#review134747 --- 1. Change distro/atlas-application.properties to use berkleydb and elasticsearch 2. Remove packaging of hbase and solr - Shwetha GS On May 24, 2016, 5:42 p.m., Tom Beerbower wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/47782/ > --- > > (Updated May 24, 2016, 5:42 p.m.) > > > Review request for atlas and Hemanth Yamijala. > > > Bugs: ATLAS-823 > https://issues.apache.org/jira/browse/ATLAS-823 > > > Repository: atlas > > > Description > --- > > Use of embedded HBase and SOLR is not preferred due to multiple concerns, > including: > > 1. lack of support by Ambari makes it difficult to handle kerberization, > upgrade/migration, start/stop > 2. not suitable for production use - no HA, local-storage > > Atlas should remove support for embedded HBase and SOLR. Instead Atlas should > be updated to use external HBase and SOLR. > > > Diffs > - > > distro/src/bin/atlas_start.py cb869bb > distro/src/bin/atlas_stop.py a72c724 > distro/src/test/python/scripts/TestMetadata.py 00ca3e2 > docs/src/site/twiki/Configuration.twiki 879c06c > docs/src/site/twiki/InstallationSteps.twiki 006e1e8 > > Diff: https://reviews.apache.org/r/47782/diff/ > > > Testing > --- > > updated unit test > > mvn clean test > > > Thanks, > > Tom Beerbower > >
Re: Review Request 46731: ATLAS-716 Entity update/delete notifications
> On May 23, 2016, 5:02 a.m., Suma Shivaprasad wrote: > > repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java, > > line 318 > > <https://reviews.apache.org/r/46731/diff/3/?file=1387706#file1387706line318> > > > > why is the check for isComposite removed? Can you pls explain? > > Shwetha GS wrote: > For example, in case of soft delete, if the column is deleted, we don't > update the table's columns attribute. So, table definition returns the list > of columns that includes deleted columns as well. > > For hive_process, when table is deleted, the reference is not removed. To > be consistent, for hive_table, when column is deleted, reference is not > removed as well > > Suma Shivaprasad wrote: > Wouldnt this be an issue in case of hard deletes? the iscomposite condition is used to update the reverse reference like removing column reference from table columns currently, it uses just shouldUpdateReverseAttribute check shouldUpdateReverseAttribute is set to true for hard delete shouldUpdateReverseAttribute is false for soft delete after updating the reference, it calls deleteEdge(edge, false); which takes care of actually deleting the column and its edge - Shwetha --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46731/#review134305 ----------- On May 24, 2016, 7:34 a.m., Shwetha GS wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/46731/ > --- > > (Updated May 24, 2016, 7:34 a.m.) > > > Review request for atlas. > > > Bugs: ATLAS-716 > https://issues.apache.org/jira/browse/ATLAS-716 > > > Repository: atlas > > > Description > --- > > 1. Update/delete entity APIs return list of entities created, updated and > deleted > 2. Listeners are called for all the affected entities > > > Diffs > - > > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java > 254e150 > addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java > 70100f1 > client/src/main/java/org/apache/atlas/AtlasClient.java 7e32cc2 > client/src/main/java/org/apache/atlas/EntityAuditEvent.java 460f708 > client/src/main/java/org/apache/atlas/SerDe.java PRE-CREATION > client/src/test/java/org/apache/atlas/AtlasClientTest.java 0e80573 > > notification/src/main/java/org/apache/atlas/notification/MessageVersion.java > 3f16a9a > > notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java > 384f383 > > notification/src/main/java/org/apache/atlas/notification/VersionedMessageDeserializer.java > 290be59 > > notification/src/main/java/org/apache/atlas/notification/entity/NotificationEntityChangeListener.java > 300cbb5 > > notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java > e8b55ef > > repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java > 540c308 > > repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListener.java > 5b4bdbf > > repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java > 8f11322 > > repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java > a9e4f39 > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java > 3604277 > repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java > cccafc2 > > repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java > df28ab3 > > repository/src/main/java/org/apache/atlas/repository/graph/HardDeleteHandler.java > f8bbf73 > > repository/src/main/java/org/apache/atlas/repository/graph/SoftDeleteHandler.java > aa78582 > > repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java > a017536 > > repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java > 3fb128c > > repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java > 7cd83f8 > > repository/src/test/java/org/apache/atlas/repository/audit/AuditRepositoryTestBase.java > be407a5 > > repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java > 1aeedb5 > > repos
Re: Review Request 46731: ATLAS-716 Entity update/delete notifications
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46731/ --- (Updated May 24, 2016, 7:34 a.m.) Review request for atlas. Changes --- Re-based, addressed review comments Bugs: ATLAS-716 https://issues.apache.org/jira/browse/ATLAS-716 Repository: atlas Description --- 1. Update/delete entity APIs return list of entities created, updated and deleted 2. Listeners are called for all the affected entities Diffs (updated) - addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 254e150 addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 70100f1 client/src/main/java/org/apache/atlas/AtlasClient.java 7e32cc2 client/src/main/java/org/apache/atlas/EntityAuditEvent.java 460f708 client/src/main/java/org/apache/atlas/SerDe.java PRE-CREATION client/src/test/java/org/apache/atlas/AtlasClientTest.java 0e80573 notification/src/main/java/org/apache/atlas/notification/MessageVersion.java 3f16a9a notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java 384f383 notification/src/main/java/org/apache/atlas/notification/VersionedMessageDeserializer.java 290be59 notification/src/main/java/org/apache/atlas/notification/entity/NotificationEntityChangeListener.java 300cbb5 notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java e8b55ef repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 540c308 repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListener.java 5b4bdbf repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java 8f11322 repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java a9e4f39 repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java 3604277 repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java cccafc2 repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java df28ab3 repository/src/main/java/org/apache/atlas/repository/graph/HardDeleteHandler.java f8bbf73 repository/src/main/java/org/apache/atlas/repository/graph/SoftDeleteHandler.java aa78582 repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java a017536 repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java 3fb128c repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 7cd83f8 repository/src/test/java/org/apache/atlas/repository/audit/AuditRepositoryTestBase.java be407a5 repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java 1aeedb5 repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositoryHardDeleteTest.java d2109d3 repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositorySoftDeleteTest.java d9e3ec9 repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java 1f906ed server-api/src/main/java/org/apache/atlas/RequestContext.java b1d87ea server-api/src/main/java/org/apache/atlas/services/MetadataService.java c8c1067 typesystem/src/main/java/org/apache/atlas/typesystem/Referenceable.java 31f157e webapp/src/main/java/org/apache/atlas/LocalAtlasClient.java c6ed85d webapp/src/main/java/org/apache/atlas/notification/NotificationEntityChangeListener.java PRE-CREATION webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java 709fec5 webapp/src/main/java/org/apache/atlas/web/service/ServiceModule.java 0f8bcb1 webapp/src/test/java/org/apache/atlas/LocalAtlasClientTest.java 7f20652 webapp/src/test/java/org/apache/atlas/notification/EntityNotificationIT.java 6985152 webapp/src/test/java/org/apache/atlas/notification/NotificationEntityChangeListenerTest.java PRE-CREATION webapp/src/test/java/org/apache/atlas/web/resources/EntityJerseyResourceIT.java aa92bc0 webapp/src/test/java/org/apache/atlas/web/service/CuratorFactoryTest.java 2d510a0 Diff: https://reviews.apache.org/r/46731/diff/ Testing --- UTs Thanks, Shwetha GS
Re: Review Request 46731: ATLAS-716 Entity update/delete notifications
> On May 23, 2016, 4:57 a.m., Suma Shivaprasad wrote: > > repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java, > > line 429 > > <https://reviews.apache.org/r/46731/diff/3/?file=1387714#file1387714line429> > > > > if this needs to be fixed, pls raise a jira AtlasClient.EntityResult is used in the REST API response. For a deleted entity, just the guid makes sense. But NotificationEntityChangeListener on the server requires type name as well as the message contains entity always. Ideally, there should be different message formats for different notifications - entity create/delete etc. This is just small code cleanup. Will remove TODO, change to NOTE - Shwetha --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46731/#review134304 ------- On May 19, 2016, 5:41 a.m., Shwetha GS wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/46731/ > --- > > (Updated May 19, 2016, 5:41 a.m.) > > > Review request for atlas. > > > Bugs: ATLAS-716 > https://issues.apache.org/jira/browse/ATLAS-716 > > > Repository: atlas > > > Description > --- > > 1. Update/delete entity APIs return list of entities created, updated and > deleted > 2. Listeners are called for all the affected entities > > > Diffs > - > > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java > 254e150 > addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java > 70100f1 > client/src/main/java/org/apache/atlas/AtlasClient.java b3ec95c > client/src/main/java/org/apache/atlas/EntityAuditEvent.java 460f708 > client/src/main/java/org/apache/atlas/SerDe.java PRE-CREATION > client/src/test/java/org/apache/atlas/AtlasClientTest.java 0e80573 > > notification/src/main/java/org/apache/atlas/notification/MessageVersion.java > 3f16a9a > > notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java > 384f383 > > notification/src/main/java/org/apache/atlas/notification/VersionedMessageDeserializer.java > 290be59 > > notification/src/main/java/org/apache/atlas/notification/entity/NotificationEntityChangeListener.java > 300cbb5 > > notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java > e8b55ef > > repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java > 540c308 > > repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListener.java > 5b4bdbf > > repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java > 8f11322 > > repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java > a9e4f39 > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java > 3604277 > repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java > cccafc2 > > repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java > df28ab3 > > repository/src/main/java/org/apache/atlas/repository/graph/HardDeleteHandler.java > f8bbf73 > > repository/src/main/java/org/apache/atlas/repository/graph/SoftDeleteHandler.java > aa78582 > > repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java > a017536 > > repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java > 3fb128c > > repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java > 5195cbe > > repository/src/test/java/org/apache/atlas/repository/audit/AuditRepositoryTestBase.java > be407a5 > > repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java > 1aeedb5 > > repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositoryHardDeleteTest.java > d2109d3 > > repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositorySoftDeleteTest.java > d9e3ec9 > > repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java > 1f906ed > server-api/src/main/java/org/apache/atlas/RequestContext.java b1d87ea > server-api/src/main/java/org/apache/atlas/services/MetadataService.java > 13d20d8 > typesystem/src/main/java/org/apache/atla
Re: Review Request 47571: JAAS configuration needed for Kafka interaction via Atlas config file
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47571/#review134312 --- Ship it! Ship It! - Shwetha GS On May 21, 2016, 12:18 a.m., Abhay Kulkarni wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/47571/ > --- > > (Updated May 21, 2016, 12:18 a.m.) > > > Review request for atlas, Madhan Neethiraj, Ramesh Mani, Shwetha GS, Suma > Shivaprasad, and Hemanth Yamijala. > > > Bugs: ATLAS-809 > https://issues.apache.org/jira/browse/ATLAS-809 > > > Repository: atlas > > > Description > --- > > In-Memory JAAS configurator obviates need for passing > -Djava.security.auth.login.config command line argument to JVMs when starting > various components that may interact with Kafka. > > > Diffs > - > > common/pom.xml ba1210a > common/src/main/java/org/apache/atlas/ApplicationProperties.java 877d7db > > common/src/main/java/org/apache/atlas/security/InMemoryJAASConfiguration.java > PRE-CREATION > > common/src/test/java/org/apache/atlas/security/InMemoryJAASConfigurationTest.java > PRE-CREATION > common/src/test/resources/atlas-jaas.properties PRE-CREATION > distro/src/conf/atlas-application.properties d1deae4 > docs/src/site/twiki/security.twiki 88e35d2 > pom.xml 98184ae > typesystem/src/main/resources/atlas-application.properties a8e77bb > > Diff: https://reviews.apache.org/r/47571/diff/ > > > Testing > --- > > Passed unit tests. > > > Thanks, > > Abhay Kulkarni > >
Re: Review Request 47591: Quickstart fails to run after queries to the business taxonomy API
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47591/#review134150
---
Ship it!
Ship It!
- Shwetha GS
On May 19, 2016, 2:05 p.m., John Speidel wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47591/
> ---
>
> (Updated May 19, 2016, 2:05 p.m.)
>
>
> Review request for atlas and Hemanth Yamijala.
>
>
> Bugs: ATLAS-405
> https://issues.apache.org/jira/browse/ATLAS-405
>
>
> Repository: atlas
>
>
> Description
> ---
>
> On a fresh Atlas server, using a REST client like curl or Postman, make a
> call to the business taxonomy API.
> E.g. http://localhost:21000/api/atlas/v1/taxonomies
> This will return an empty result set
> Run quick_start.py.
>
> This fails with an exception as follows:
> Exception in thread "main" org.apache.atlas.AtlasServiceException: Metadata
> service API CREATE_ENTITY failed with status 500(Internal Server Error)
> Response Body ({"error":"Could not commit transaction due to exception during
> persistence","stackTrace":"com.thinkaurelius.titan.core.TitanException: Could
> not commit transaction due to exception during persistence\n\tat
> com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1348)\n\tat
>
> com.thinkaurelius.titan.graphdb.blueprints.TitanBlueprintsGraph.commit(TitanBlueprintsGraph.java:60)\n\tat
>
> org.apache.atlas.GraphTransactionInterceptor.invoke(GraphTransactionInterceptor.java:43)\n\tat
>
> org.apache.atlas.services.DefaultMetadataService.createEntities(DefaultMetadataService.java:321)\n\tat
>
> org.apache.atlas.services.DefaultMetadataService.createEntities(DefaultMetadataService.java:316)\n\tat
>
> org.apache.atlas.web.resources.EntityResource.submit(EntityResource.java:122)\n\tat
> sun.reflect.NativeMethodAccessorImp
l.invoke0(Native Method)\n\tat
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\n\tat
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat
java.lang.reflect.Method.invoke(Method.java:606)\n\tat
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)\n\tat
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)\n\tat
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)\n\tat
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)\n\tat
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)\n\tat
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)\n\tat
com.sun.jersey.server.impl.uri.rules.RootResour
ceClassesRule.accept(RootResourceClassesRule.java:84)\n\tat
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)\n\tat
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)\n\tat
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)\n\tat
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)\n\tat
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)\n\tat
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)\n\tat
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)\n\tat
javax.servlet.http.HttpServlet.service(HttpServlet.java:790)\n\tat
com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)\n\tat
com.google.inject.servlet.ServletDefinition.doService(ServletDefini
tion.java:277)\n\tat
com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182)\n\tat
com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)\n\tat
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)\n\tat
org.apache.atlas.web.filters.AuditFilter.doFilter(AuditFilter.java:71)\n\tat
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)\n\tat
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)\n\tat
com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)\n\tat
com.google.inject.servlet.GuiceFilter$
Re: Review Request 47638: ATLAS-752 Column rename in Hive should retain traits/tags associated with it
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47638/#review134143 --- addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java (line 326) <https://reviews.apache.org/r/47638/#comment198811> Instead of double loop twice, create maps for each and use maps - Shwetha GS On May 20, 2016, 6:42 a.m., Vimal Sharma wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/47638/ > --- > > (Updated May 20, 2016, 6:42 a.m.) > > > Review request for atlas. > > > Bugs: ATLAS-752 > https://issues.apache.org/jira/browse/ATLAS-752 > > > Repository: atlas > > > Description > --- > > ATLAS-752 When a column is renamed, the traits associted with it are lost. > The traits should be retained with the new column > > > Diffs > - > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java > 418e755 > addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java > 70100f1 > > Diff: https://reviews.apache.org/r/47638/diff/ > > > Testing > --- > > > Thanks, > > Vimal Sharma > >
Re: Review Request 47612: ATLAS-495 : Atlas Ranger Authorization Plugin
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47612/#review134144 --- authorization/src/main/java/org/apache/atlas/authorize/simple/AtlasAuthorizationUtils.java (line 114) <https://reviews.apache.org/r/47612/#comment198806> Only /types should be mapped to TYPE resource. Others should be just ENTITY resource authorization/src/main/java/org/apache/atlas/authorize/simple/FileReaderUtil.java (line 30) <https://reviews.apache.org/r/47612/#comment198807> You should use java.nio.file.Files.readAllLines() and remove this class authorization/src/main/java/org/apache/atlas/authorize/simple/FileReaderUtil.java (line 52) <https://reviews.apache.org/r/47612/#comment198805> should be in finally clause distro/src/conf/policy-store.txt (line 6) <https://reviews.apache.org/r/47612/#comment198809> can we remove albert user and policy? distro/src/conf/users-credentials.properties (line 2) <https://reviews.apache.org/r/47612/#comment198810> Can you keep only admin user? This conf is the default conf used in package and shouldn't have random users - Shwetha GS On May 20, 2016, 2:02 p.m., Nixon Rodrigues wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/47612/ > --- > > (Updated May 20, 2016, 2:02 p.m.) > > > Review request for atlas, Madhan Neethiraj, Shwetha GS, Selvamohan Neethiraj, > and Hemanth Yamijala. > > > Bugs: ATLAS-495 > https://issues.apache.org/jira/browse/ATLAS-495 > > > Repository: atlas > > > Description > --- > > Patch contains following changes : > > 1) Refactoring of authorization code to authorizer module to be reused by > webapp, ranger plugin modules. > 1) Default policy file path to atlas-home/conf/policy-store.txt if property > is not set in atlas-application.properties file. > 2) Renamed create action instead of write. > 4) Added authorizer impl class in atlas properties and fall back to simple > authorizer if authorizer is not set in property. > 5) AtlasAuthorizerFactory for authorizer. > 6) Handled changes in Test cases and in filter as per new implementation of > AtlasAuthorizerFactory. > > > Diffs > - > > authorization/pom.xml PRE-CREATION > > authorization/src/main/java/org/apache/atlas/authorize/AtlasAccessRequest.java > PRE-CREATION > > authorization/src/main/java/org/apache/atlas/authorize/AtlasActionTypes.java > PRE-CREATION > > authorization/src/main/java/org/apache/atlas/authorize/AtlasAuthorizationException.java > PRE-CREATION > authorization/src/main/java/org/apache/atlas/authorize/AtlasAuthorizer.java > PRE-CREATION > > authorization/src/main/java/org/apache/atlas/authorize/AtlasAuthorizerFactory.java > PRE-CREATION > > authorization/src/main/java/org/apache/atlas/authorize/AtlasResourceTypes.java > PRE-CREATION > > authorization/src/main/java/org/apache/atlas/authorize/simple/AtlasAuthorizationUtils.java > PRE-CREATION > > authorization/src/main/java/org/apache/atlas/authorize/simple/FileReaderUtil.java > PRE-CREATION > > authorization/src/main/java/org/apache/atlas/authorize/simple/PolicyDef.java > PRE-CREATION > > authorization/src/main/java/org/apache/atlas/authorize/simple/PolicyParser.java > PRE-CREATION > > authorization/src/main/java/org/apache/atlas/authorize/simple/PolicyUtil.java > PRE-CREATION > > authorization/src/main/java/org/apache/atlas/authorize/simple/SimpleAtlasAuthorizer.java > PRE-CREATION > > authorization/src/test/java/org/apache/atlas/authorize/simple/AtlasAuthorizationUtilsTest.java > PRE-CREATION > > authorization/src/test/java/org/apache/atlas/authorize/simple/PolicyParserTest.java > PRE-CREATION > > authorization/src/test/java/org/apache/atlas/authorize/simple/PolicyUtilTest.java > PRE-CREATION > > authorization/src/test/java/org/apache/atlas/authorize/simple/SimpleAtlasAuthorizerTest.java > PRE-CREATION > common/pom.xml 614b3f6 > common/src/main/java/org/apache/atlas/utils/PropertiesUtil.java > PRE-CREATION > common/src/main/java/org/apache/atlas/utils/XMLPropertiesUtil.java > PRE-CREATION > distro/src/conf/atlas-application.properties e2082ef > distro/src/conf/policy-store.txt 5bcc1cf > distro/src/conf/users-credentials.properties 212d018 > pom.xml 685eb7d > webapp/pom.xml 4b67ffa > webapp/src/main/java/org/apac
Re: Review Request 47612: ATLAS-495 : Atlas Ranger Authorization Plugin
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47612/#review134131
---
authorizer/src/main/java/org/apache/atlas/authorize/simple/AtlasAuthorizationUtils.java
(line 75)
<https://reviews.apache.org/r/47612/#comment198784>
Use this syntax for all log statements. It already takes care of string
concatenation only if the log level is enabled. if debug enabled check is not
required:
LOG.debug("<== AtlasAuthorizationFilter getAtlasAction HTTP Method {}
mapped to AtlasAction : {}", method, action);
pom.xml (line 466)
<https://reviews.apache.org/r/47612/#comment198783>
rename module to authorization
Please avoid un-ncessary log statements. Log only when needed
- Shwetha GS
On May 19, 2016, 6:49 p.m., Nixon Rodrigues wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47612/
> ---
>
> (Updated May 19, 2016, 6:49 p.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Shwetha GS, Selvamohan Neethiraj,
> and Hemanth Yamijala.
>
>
> Bugs: ATLAS-495
> https://issues.apache.org/jira/browse/ATLAS-495
>
>
> Repository: atlas
>
>
> Description
> ---
>
> Patch contains following changes :
>
> 1) Refactoring of authorization code to authorizer module to be reused by
> webapp, ranger plugin modules.
> 1) Default policy file path to atlas-home/conf/policy-store.txt if property
> is not set in atlas-application.properties file.
> 2) Renamed create action instead of write.
> 4) Added authorizer impl class in atlas properties and fall back to simple
> authorizer if authorizer is not set in property.
> 5) AtlasAuthorizerFactory for authorizer.
> 6) Handled changes in Test cases and in filter as per new implementation of
> AtlasAuthorizerFactory.
>
>
> Diffs
> -
>
> authorizer/pom.xml PRE-CREATION
> authorizer/src/main/java/org/apache/atlas/authorize/AtlasAccessRequest.java
> PRE-CREATION
> authorizer/src/main/java/org/apache/atlas/authorize/AtlasActionTypes.java
> PRE-CREATION
>
> authorizer/src/main/java/org/apache/atlas/authorize/AtlasAuthorizationException.java
> PRE-CREATION
> authorizer/src/main/java/org/apache/atlas/authorize/AtlasAuthorizer.java
> PRE-CREATION
>
> authorizer/src/main/java/org/apache/atlas/authorize/AtlasAuthorizerFactory.java
> PRE-CREATION
> authorizer/src/main/java/org/apache/atlas/authorize/AtlasResourceTypes.java
> PRE-CREATION
>
> authorizer/src/main/java/org/apache/atlas/authorize/simple/AtlasAuthorizationUtils.java
> PRE-CREATION
>
> authorizer/src/main/java/org/apache/atlas/authorize/simple/FileReaderUtil.java
> PRE-CREATION
> authorizer/src/main/java/org/apache/atlas/authorize/simple/PolicyDef.java
> PRE-CREATION
>
> authorizer/src/main/java/org/apache/atlas/authorize/simple/PolicyParser.java
> PRE-CREATION
> authorizer/src/main/java/org/apache/atlas/authorize/simple/PolicyUtil.java
> PRE-CREATION
>
> authorizer/src/main/java/org/apache/atlas/authorize/simple/SimpleAtlasAuthorizer.java
> PRE-CREATION
>
> authorizer/src/test/java/org/apache/atlas/authorize/simple/AtlasAuthorizationUtilsTest.java
> PRE-CREATION
>
> authorizer/src/test/java/org/apache/atlas/authorize/simple/PolicyParserTest.java
> PRE-CREATION
>
> authorizer/src/test/java/org/apache/atlas/authorize/simple/PolicyUtilTest.java
> PRE-CREATION
>
> authorizer/src/test/java/org/apache/atlas/authorize/simple/SimpleAtlasAuthorizerTest.java
> PRE-CREATION
> common/pom.xml 614b3f6
> common/src/main/java/org/apache/atlas/utils/PropertiesUtil.java
> PRE-CREATION
> common/src/main/java/org/apache/atlas/utils/XMLPropertiesUtil.java
> PRE-CREATION
> distro/src/conf/atlas-application.properties d4722fb
> pom.xml 30fb95a
> webapp/pom.xml 4b67ffa
> webapp/src/main/java/org/apache/atlas/authorize/AtlasAccessRequest.java
> 5db9646
> webapp/src/main/java/org/apache/atlas/authorize/AtlasAccessorTypes.java
> 5f3827a
> webapp/src/main/java/org/apache/atlas/authorize/AtlasActionTypes.java
> 13c8b53
>
> webapp/src/main/java/org/apache/atlas/authorize/AtlasAuthorizationException.java
> 676c9f9
>
> webapp/src/main/java/org/apache/atlas/authorize/AtlasAuthorizationUtils.java
> 14a2aac
> webapp/src/main/java/org/apache/atlas/authorize/AtlasAuthorizer.java
> 7c93c7a
> webapp/src/main/java/org/apache/atlas/authorize/Atl
