[jira] [Resolved] (ATLAS-2249) Upgrade Atlas Client to use Jersey 2

2018-11-15 Thread Pierre Padovani (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-2249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Padovani resolved ATLAS-2249.

Resolution: Won't Fix

> Upgrade Atlas Client to use Jersey 2
> 
>
> Key: ATLAS-2249
> URL: https://issues.apache.org/jira/browse/ATLAS-2249
> Project: Atlas
>  Issue Type: Improvement
>  Components: atlas-intg
>Affects Versions: 1.0.0
>    Reporter: Pierre Padovani
>Priority: Major
>  Labels: patch-available
> Attachments: client-jersey2-V2.patch, client-jersey2-V3.patch, 
> client-jersey2-V4.patch, client-jersey2.patch
>
>
> Using the existing Atlas client within a Jersey 2 service causes conflicts. I 
> recommend upgrading just the client code to Jersey 2. The enclosed patch 
> fully upgrades the V1 and V2 client to Jersey 2, and we have tested this 
> against our own installation of 1.0.0-SNAPSHOT.
> NOTE: We have not tested SSL/Kerberos authentication/security



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (ATLAS-2249) Upgrade Atlas Client to use Jersey 2

2018-11-15 Thread Pierre Padovani (JIRA)


[ 
https://issues.apache.org/jira/browse/ATLAS-2249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688558#comment-16688558
 ] 

Pierre Padovani commented on ATLAS-2249:


My apologies for taking so long to answer this it got buried in email. I would 
actually not recommend using this patch due to an issue in terms of json 
serialization around nulls we discovered. Our solution/workaround was to 
uber-jar the client changing all of the packaging of the underlying Jersey 1 
code to avoid conflicts.

I believe that only by updating the entire Atlas code base to Jersey 2 can this 
be resolved in a correct manner.

> Upgrade Atlas Client to use Jersey 2
> 
>
> Key: ATLAS-2249
> URL: https://issues.apache.org/jira/browse/ATLAS-2249
> Project: Atlas
>  Issue Type: Improvement
>  Components: atlas-intg
>Affects Versions: 1.0.0
>    Reporter: Pierre Padovani
>Priority: Major
>  Labels: patch-available
> Attachments: client-jersey2-V2.patch, client-jersey2-V3.patch, 
> client-jersey2-V4.patch, client-jersey2.patch
>
>
> Using the existing Atlas client within a Jersey 2 service causes conflicts. I 
> recommend upgrading just the client code to Jersey 2. The enclosed patch 
> fully upgrades the V1 and V2 client to Jersey 2, and we have tested this 
> against our own installation of 1.0.0-SNAPSHOT.
> NOTE: We have not tested SSL/Kerberos authentication/security



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Closed] (ATLAS-2249) Upgrade Atlas Client to use Jersey 2

2018-11-15 Thread Pierre Padovani (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-2249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Padovani closed ATLAS-2249.
--

> Upgrade Atlas Client to use Jersey 2
> 
>
> Key: ATLAS-2249
> URL: https://issues.apache.org/jira/browse/ATLAS-2249
> Project: Atlas
>  Issue Type: Improvement
>  Components: atlas-intg
>Affects Versions: 1.0.0
>    Reporter: Pierre Padovani
>Priority: Major
>  Labels: patch-available
> Attachments: client-jersey2-V2.patch, client-jersey2-V3.patch, 
> client-jersey2-V4.patch, client-jersey2.patch
>
>
> Using the existing Atlas client within a Jersey 2 service causes conflicts. I 
> recommend upgrading just the client code to Jersey 2. The enclosed patch 
> fully upgrades the V1 and V2 client to Jersey 2, and we have tested this 
> against our own installation of 1.0.0-SNAPSHOT.
> NOTE: We have not tested SSL/Kerberos authentication/security



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (ATLAS-2752) Atlas start up fail with error "Error creating bean with name 'graphTransactionAdvisor'

2018-06-12 Thread Pierre Padovani (JIRA)


[ 
https://issues.apache.org/jira/browse/ATLAS-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509682#comment-16509682
 ] 

Pierre Padovani commented on ATLAS-2752:


This error sounds like a jar is missing. How did you get the installation? Did 
you build it, or pull the binary distribution?

 

Check

$ATLAS_HOME/server/webapp/atlas/WEB-INF/lib

You should have these solr jars:
 * server/webapp/atlas/WEB-INF/lib/janusgraph-solr-0.2.0.jar
 * server/webapp/atlas/WEB-INF/lib/solr-solrj-7.0.0.jar
 * server/webapp/atlas/WEB-INF/lib/solr-core-7.0.0.jar

If you do have those jars, then your issue is outside my knowledge area. I 
would close this issue, and open another specific to solr, and an expert in 
that area will pick up the issue and help you out.

Pierre

> Atlas start up fail with error "Error creating bean with name 
> 'graphTransactionAdvisor'
> ---
>
> Key: ATLAS-2752
> URL: https://issues.apache.org/jira/browse/ATLAS-2752
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core, atlas-webui
>Affects Versions: 1.0.0
> Environment: test
>Reporter: Mahesh Kakol
>Priority: Major
>
> I am getting below while starting atlas  and graph stoage as cassandra
>  
>  
> 2018-06-11 11:54:21,226 WARN - [main:] ~ Using @Deprecated Class 
> org.springframework.web.util.Log4jConfigListener (DeprecationWarning:43)
> 2018-06-11 11:54:23,400 WARN - [main:] ~ Exception encountered during context 
> initialization - cancelling refresh attempt: 
> org.springframework.beans.factory.BeanCreationExceptio
> n: Error creating bean with name 
> 'org.springframework.context.event.internalEventListenerProcessor': 
> BeanPostProcessor before instantiation of bean failed; nested exception is or
> g.springframework.beans.factory.UnsatisfiedDependencyException: Error 
> creating bean with name 'graphTransactionAdvisor' defined in URL 
> [jar:[file:/data/apache-atlas/apache-atlas-s|file:///data/apache-atlas/apache-atlas-s]
> ources-1.0.0/distro/target/apache-atlas-1.0.0-bin/apache-atlas-1.0.0/server/webapp/atlas/WEB-INF/lib/atlas-repository-1.0.0.jar!/org/apache/atlas/GraphTransactionAdvisor.class]:
> Unsatisfied dependency expressed through constructor parameter 0; nested 
> exception is 
> org.springframework.beans.factory.UnsatisfiedDependencyException: Error 
> creating bean with n
> ame 'graphTransactionInterceptor' defined in URL 
> [jar:[file:/data/apache-atlas/apache-atlas-sources-1.0.0/distro/target/apache-atlas-1.0.0-bin/apache-atlas-1.0.0/server/webapp/atl|file:///data/apache-atlas/apache-atlas-sources-1.0.0/distro/target/apache-atlas-1.0.0-bin/apache-atlas-1.0.0/server/webapp/atl]
> as/WEB-INF/lib/atlas-repository-1.0.0.jar!/org/apache/atlas/GraphTransactionInterceptor.class]:
>  Unsatisfied dependency expressed through constructor parameter 0; nested 
> exception
> is org.springframework.beans.factory.BeanCreationException: Error creating 
> bean with name 'get' defined in 
> org.apache.atlas.repository.graph.AtlasGraphProvider: Bean instantiati
> on via factory method failed; nested exception is 
> org.springframework.beans.BeanInstantiationException: Failed to instantiate 
> [org.apache.atlas.repository.graphdb.AtlasGraph]: Fa
> ctory method 'get' threw exception; nested exception is 
> java.lang.IllegalArgumentException: Could not instantiate implementation: 
> org.janusgraph.diskstorage.cassandra.astyanax.As
> tyanaxStoreManager (AbstractApplicationContext:551)
> 2018-06-11 11:54:23,402 ERROR - [main:] ~ Context initialization failed 
> (ContextLoader:350)
> org.springframework.beans.factory.BeanCreationException: Error creating bean 
> with name 'org.springframework.context.event.internalEventListenerProcessor': 
> BeanPostProcessor befor
> e instantiation of bean failed; nested exception is 
> org.springframework.beans.factory.UnsatisfiedDependencyException: Error 
> creating bean with name 'graphTransactionAdvisor' defi
> ned in URL 
> [jar:[file:/data/apache-atlas/apache-atlas-sources-1.0.0/distro/target/apache-atlas-1.0.0-bin/apache-atlas-1.0.0/server/webapp/atlas/WEB-INF/lib/atlas-repository-1.0.0|file:///data/apache-atlas/apache-atlas-sources-1.0.0/distro/target/apache-atlas-1.0.0-bin/apache-atlas-1.0.0/server/webapp/atlas/WEB-INF/lib/atlas-repository-1.0.0].
> jar!/org/apache/atlas/GraphTransactionAdvisor.class]: Unsatisfied dependency 
> expressed through constructor parameter 0; nested exception is 
> org.springframework.beans.factory.Unsa
> tisfiedDependencyException: Error creating bean with name 
> 'graphTransactionInterceptor' defined in URL 
> [jar:[

[jira] [Commented] (ATLAS-2752) Atlas start up fail with error "Error creating bean with name 'graphTransactionAdvisor'

2018-06-11 Thread Pierre Padovani (JIRA)


[ 
https://issues.apache.org/jira/browse/ATLAS-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508430#comment-16508430
 ] 

Pierre Padovani commented on ATLAS-2752:


Your issue is the " atlas.graph.storage.port" setting for the backend. Port 
9042 is for the native transport not thrift, so the data frames used are 
different. For my company's installation (Cassandra + Elasticsearch) we just 
leave the port setting blank, and the default thrift port is automatically 
used. If you have configured your cluster with a custom thrift port, that 
should be the value used in the " atlas.graph.storage.port" settings.

 

Pierre

> Atlas start up fail with error "Error creating bean with name 
> 'graphTransactionAdvisor'
> ---
>
> Key: ATLAS-2752
> URL: https://issues.apache.org/jira/browse/ATLAS-2752
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core, atlas-webui
>Affects Versions: 1.0.0
> Environment: test
>Reporter: Mahesh Kakol
>Priority: Major
>
> I am getting below while starting atlas  and graph stoage as cassandra
>  
>  
> 2018-06-11 11:54:21,226 WARN - [main:] ~ Using @Deprecated Class 
> org.springframework.web.util.Log4jConfigListener (DeprecationWarning:43)
> 2018-06-11 11:54:23,400 WARN - [main:] ~ Exception encountered during context 
> initialization - cancelling refresh attempt: 
> org.springframework.beans.factory.BeanCreationExceptio
> n: Error creating bean with name 
> 'org.springframework.context.event.internalEventListenerProcessor': 
> BeanPostProcessor before instantiation of bean failed; nested exception is or
> g.springframework.beans.factory.UnsatisfiedDependencyException: Error 
> creating bean with name 'graphTransactionAdvisor' defined in URL 
> [jar:[file:/data/apache-atlas/apache-atlas-s|file:///data/apache-atlas/apache-atlas-s]
> ources-1.0.0/distro/target/apache-atlas-1.0.0-bin/apache-atlas-1.0.0/server/webapp/atlas/WEB-INF/lib/atlas-repository-1.0.0.jar!/org/apache/atlas/GraphTransactionAdvisor.class]:
> Unsatisfied dependency expressed through constructor parameter 0; nested 
> exception is 
> org.springframework.beans.factory.UnsatisfiedDependencyException: Error 
> creating bean with n
> ame 'graphTransactionInterceptor' defined in URL 
> [jar:[file:/data/apache-atlas/apache-atlas-sources-1.0.0/distro/target/apache-atlas-1.0.0-bin/apache-atlas-1.0.0/server/webapp/atl|file:///data/apache-atlas/apache-atlas-sources-1.0.0/distro/target/apache-atlas-1.0.0-bin/apache-atlas-1.0.0/server/webapp/atl]
> as/WEB-INF/lib/atlas-repository-1.0.0.jar!/org/apache/atlas/GraphTransactionInterceptor.class]:
>  Unsatisfied dependency expressed through constructor parameter 0; nested 
> exception
> is org.springframework.beans.factory.BeanCreationException: Error creating 
> bean with name 'get' defined in 
> org.apache.atlas.repository.graph.AtlasGraphProvider: Bean instantiati
> on via factory method failed; nested exception is 
> org.springframework.beans.BeanInstantiationException: Failed to instantiate 
> [org.apache.atlas.repository.graphdb.AtlasGraph]: Fa
> ctory method 'get' threw exception; nested exception is 
> java.lang.IllegalArgumentException: Could not instantiate implementation: 
> org.janusgraph.diskstorage.cassandra.astyanax.As
> tyanaxStoreManager (AbstractApplicationContext:551)
> 2018-06-11 11:54:23,402 ERROR - [main:] ~ Context initialization failed 
> (ContextLoader:350)
> org.springframework.beans.factory.BeanCreationException: Error creating bean 
> with name 'org.springframework.context.event.internalEventListenerProcessor': 
> BeanPostProcessor befor
> e instantiation of bean failed; nested exception is 
> org.springframework.beans.factory.UnsatisfiedDependencyException: Error 
> creating bean with name 'graphTransactionAdvisor' defi
> ned in URL 
> [jar:[file:/data/apache-atlas/apache-atlas-sources-1.0.0/distro/target/apache-atlas-1.0.0-bin/apache-atlas-1.0.0/server/webapp/atlas/WEB-INF/lib/atlas-repository-1.0.0|file:///data/apache-atlas/apache-atlas-sources-1.0.0/distro/target/apache-atlas-1.0.0-bin/apache-atlas-1.0.0/server/webapp/atlas/WEB-INF/lib/atlas-repository-1.0.0].
> jar!/org/apache/atlas/GraphTransactionAdvisor.class]: Unsatisfied dependency 
> expressed through constructor parameter 0; nested exception is 
> org.springframework.beans.factory.Unsa
> tisfiedDependencyException: Error creating bean with name 
> 'graphTransactionInterceptor' defined in URL 
> [jar:[file:/data/apache-atlas/apache-atlas-sources-1.0.0/distro/target/apach|file:///data/apache-atlas/apache-atlas-sources-

[jira] [Comment Edited] (ATLAS-2752) Atlas start up fail with error "Error creating bean with name 'graphTransactionAdvisor'

2018-06-11 Thread Pierre Padovani (JIRA)


[ 
https://issues.apache.org/jira/browse/ATLAS-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508176#comment-16508176
 ] 

Pierre Padovani edited comment on ATLAS-2752 at 6/11/18 2:56 PM:
-

A few questions for you:
 * What version of Cassandra are you using?
 * Did you enable thrift on the Cassandra cluster?

 

Pierre


was (Author: ppadovani):
A few questions for you:
 * What version of Cassandra are you using?
 * Did you enable thrift?

 

Pierre

> Atlas start up fail with error "Error creating bean with name 
> 'graphTransactionAdvisor'
> ---
>
> Key: ATLAS-2752
> URL: https://issues.apache.org/jira/browse/ATLAS-2752
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core, atlas-webui
>Affects Versions: 1.0.0
> Environment: test
>Reporter: Mahesh Kakol
>Priority: Major
>
> I am getting below while starting atlas  and graph stoage as cassandra
>  
>  
> 2018-06-11 11:54:21,226 WARN - [main:] ~ Using @Deprecated Class 
> org.springframework.web.util.Log4jConfigListener (DeprecationWarning:43)
> 2018-06-11 11:54:23,400 WARN - [main:] ~ Exception encountered during context 
> initialization - cancelling refresh attempt: 
> org.springframework.beans.factory.BeanCreationExceptio
> n: Error creating bean with name 
> 'org.springframework.context.event.internalEventListenerProcessor': 
> BeanPostProcessor before instantiation of bean failed; nested exception is or
> g.springframework.beans.factory.UnsatisfiedDependencyException: Error 
> creating bean with name 'graphTransactionAdvisor' defined in URL 
> [jar:[file:/data/apache-atlas/apache-atlas-s|file:///data/apache-atlas/apache-atlas-s]
> ources-1.0.0/distro/target/apache-atlas-1.0.0-bin/apache-atlas-1.0.0/server/webapp/atlas/WEB-INF/lib/atlas-repository-1.0.0.jar!/org/apache/atlas/GraphTransactionAdvisor.class]:
> Unsatisfied dependency expressed through constructor parameter 0; nested 
> exception is 
> org.springframework.beans.factory.UnsatisfiedDependencyException: Error 
> creating bean with n
> ame 'graphTransactionInterceptor' defined in URL 
> [jar:[file:/data/apache-atlas/apache-atlas-sources-1.0.0/distro/target/apache-atlas-1.0.0-bin/apache-atlas-1.0.0/server/webapp/atl|file:///data/apache-atlas/apache-atlas-sources-1.0.0/distro/target/apache-atlas-1.0.0-bin/apache-atlas-1.0.0/server/webapp/atl]
> as/WEB-INF/lib/atlas-repository-1.0.0.jar!/org/apache/atlas/GraphTransactionInterceptor.class]:
>  Unsatisfied dependency expressed through constructor parameter 0; nested 
> exception
> is org.springframework.beans.factory.BeanCreationException: Error creating 
> bean with name 'get' defined in 
> org.apache.atlas.repository.graph.AtlasGraphProvider: Bean instantiati
> on via factory method failed; nested exception is 
> org.springframework.beans.BeanInstantiationException: Failed to instantiate 
> [org.apache.atlas.repository.graphdb.AtlasGraph]: Fa
> ctory method 'get' threw exception; nested exception is 
> java.lang.IllegalArgumentException: Could not instantiate implementation: 
> org.janusgraph.diskstorage.cassandra.astyanax.As
> tyanaxStoreManager (AbstractApplicationContext:551)
> 2018-06-11 11:54:23,402 ERROR - [main:] ~ Context initialization failed 
> (ContextLoader:350)
> org.springframework.beans.factory.BeanCreationException: Error creating bean 
> with name 'org.springframework.context.event.internalEventListenerProcessor': 
> BeanPostProcessor befor
> e instantiation of bean failed; nested exception is 
> org.springframework.beans.factory.UnsatisfiedDependencyException: Error 
> creating bean with name 'graphTransactionAdvisor' defi
> ned in URL 
> [jar:[file:/data/apache-atlas/apache-atlas-sources-1.0.0/distro/target/apache-atlas-1.0.0-bin/apache-atlas-1.0.0/server/webapp/atlas/WEB-INF/lib/atlas-repository-1.0.0|file:///data/apache-atlas/apache-atlas-sources-1.0.0/distro/target/apache-atlas-1.0.0-bin/apache-atlas-1.0.0/server/webapp/atlas/WEB-INF/lib/atlas-repository-1.0.0].
> jar!/org/apache/atlas/GraphTransactionAdvisor.class]: Unsatisfied dependency 
> expressed through constructor parameter 0; nested exception is 
> org.springframework.beans.factory.Unsa
> tisfiedDependencyException: Error creating bean with name 
> 'graphTransactionInterceptor' defined in URL 
> [jar:[file:/data/apache-atlas/apache-atlas-sources-1.0.0/distro/target/apach|file:///data/apache-atlas/apache-atlas-sources-1.0.0/distro/target/apach]
> e-atlas-1.0.0-bin/apache-atlas-1.0.0/server/webapp/atlas/WEB-INF/lib/atlas-repository-1.0.0.jar!/org/apache/atlas/GraphTransactionIntercept

[jira] [Commented] (ATLAS-2752) Atlas start up fail with error "Error creating bean with name 'graphTransactionAdvisor'

2018-06-11 Thread Pierre Padovani (JIRA)


[ 
https://issues.apache.org/jira/browse/ATLAS-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508176#comment-16508176
 ] 

Pierre Padovani commented on ATLAS-2752:


A few questions for you:
 * What version of Cassandra are you using?
 * Did you enable thrift?

 

Pierre

> Atlas start up fail with error "Error creating bean with name 
> 'graphTransactionAdvisor'
> ---
>
> Key: ATLAS-2752
> URL: https://issues.apache.org/jira/browse/ATLAS-2752
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core, atlas-webui
>Affects Versions: 1.0.0
> Environment: test
>Reporter: Mahesh Kakol
>Priority: Major
>
> I am getting below while starting atlas  and graph stoage as cassandra
>  
>  
> 2018-06-11 11:54:21,226 WARN - [main:] ~ Using @Deprecated Class 
> org.springframework.web.util.Log4jConfigListener (DeprecationWarning:43)
> 2018-06-11 11:54:23,400 WARN - [main:] ~ Exception encountered during context 
> initialization - cancelling refresh attempt: 
> org.springframework.beans.factory.BeanCreationExceptio
> n: Error creating bean with name 
> 'org.springframework.context.event.internalEventListenerProcessor': 
> BeanPostProcessor before instantiation of bean failed; nested exception is or
> g.springframework.beans.factory.UnsatisfiedDependencyException: Error 
> creating bean with name 'graphTransactionAdvisor' defined in URL 
> [jar:[file:/data/apache-atlas/apache-atlas-s|file:///data/apache-atlas/apache-atlas-s]
> ources-1.0.0/distro/target/apache-atlas-1.0.0-bin/apache-atlas-1.0.0/server/webapp/atlas/WEB-INF/lib/atlas-repository-1.0.0.jar!/org/apache/atlas/GraphTransactionAdvisor.class]:
> Unsatisfied dependency expressed through constructor parameter 0; nested 
> exception is 
> org.springframework.beans.factory.UnsatisfiedDependencyException: Error 
> creating bean with n
> ame 'graphTransactionInterceptor' defined in URL 
> [jar:[file:/data/apache-atlas/apache-atlas-sources-1.0.0/distro/target/apache-atlas-1.0.0-bin/apache-atlas-1.0.0/server/webapp/atl|file:///data/apache-atlas/apache-atlas-sources-1.0.0/distro/target/apache-atlas-1.0.0-bin/apache-atlas-1.0.0/server/webapp/atl]
> as/WEB-INF/lib/atlas-repository-1.0.0.jar!/org/apache/atlas/GraphTransactionInterceptor.class]:
>  Unsatisfied dependency expressed through constructor parameter 0; nested 
> exception
> is org.springframework.beans.factory.BeanCreationException: Error creating 
> bean with name 'get' defined in 
> org.apache.atlas.repository.graph.AtlasGraphProvider: Bean instantiati
> on via factory method failed; nested exception is 
> org.springframework.beans.BeanInstantiationException: Failed to instantiate 
> [org.apache.atlas.repository.graphdb.AtlasGraph]: Fa
> ctory method 'get' threw exception; nested exception is 
> java.lang.IllegalArgumentException: Could not instantiate implementation: 
> org.janusgraph.diskstorage.cassandra.astyanax.As
> tyanaxStoreManager (AbstractApplicationContext:551)
> 2018-06-11 11:54:23,402 ERROR - [main:] ~ Context initialization failed 
> (ContextLoader:350)
> org.springframework.beans.factory.BeanCreationException: Error creating bean 
> with name 'org.springframework.context.event.internalEventListenerProcessor': 
> BeanPostProcessor befor
> e instantiation of bean failed; nested exception is 
> org.springframework.beans.factory.UnsatisfiedDependencyException: Error 
> creating bean with name 'graphTransactionAdvisor' defi
> ned in URL 
> [jar:[file:/data/apache-atlas/apache-atlas-sources-1.0.0/distro/target/apache-atlas-1.0.0-bin/apache-atlas-1.0.0/server/webapp/atlas/WEB-INF/lib/atlas-repository-1.0.0|file:///data/apache-atlas/apache-atlas-sources-1.0.0/distro/target/apache-atlas-1.0.0-bin/apache-atlas-1.0.0/server/webapp/atlas/WEB-INF/lib/atlas-repository-1.0.0].
> jar!/org/apache/atlas/GraphTransactionAdvisor.class]: Unsatisfied dependency 
> expressed through constructor parameter 0; nested exception is 
> org.springframework.beans.factory.Unsa
> tisfiedDependencyException: Error creating bean with name 
> 'graphTransactionInterceptor' defined in URL 
> [jar:[file:/data/apache-atlas/apache-atlas-sources-1.0.0/distro/target/apach|file:///data/apache-atlas/apache-atlas-sources-1.0.0/distro/target/apach]
> e-atlas-1.0.0-bin/apache-atlas-1.0.0/server/webapp/atlas/WEB-INF/lib/atlas-repository-1.0.0.jar!/org/apache/atlas/GraphTransactionInterceptor.class]:
>  Unsatisfied dependency expre
> ssed through constructor parameter 0; nested exception is 
> org.springframework.beans.factory.BeanCreationException: Error creating bean 
> with

Re: Performance of relationships

2018-06-04 Thread Pierre Padovani
Here is the raw data:

child count v2 post v2 get container v1 post v1 get container v2 hybrid post v2
hybrid get
0 210 85 96 478 487 185
1 188 83 129 140 213 75
2 177 83 209 357 188 118
3 162 78 117 72 387 323
4 188 92 324 71 360 169
5 166 80 453 100 314 299
6 188 87 364 79 332 154
7 152 69 174 341 282 244
8 168 67 252 318 184 172
9 201 88 404 102 192 235
10 144 88 169 253 313 380
11 234 81 261 419 270 185
12 177 67 366 99 197 259
13 146 76 167 259 176 270
14 179 87 187 80 267 352
15 273 105 255 82 161 276
16 159 64 199 246 176 291
17 173 84 190 110 125 389
18 187 99 422 194 171 324
19 158 84 320 389 192 442
20 179 94 339 92 225 410
21 193 211 221 89 185 301
22 233 77 375 97 166 284
23 247 103 244 124 190 463
24 172 159 459 119 150 256
25 181 102 385 85 279 443
26 188 93 290 99 137 326
27 169 88 577 277 171 423
28 192 82 345 143 121 355
29 253 88 497 190 205 406
30 232 81 287 288 119 396
31 178 89 393 263 219 441
32 208 75 400 166 148 528
33 147 83 366 152 160 451
34 179 98 322 193 121 452
35 166 77 331 184 126 464
36 348 69 526 161 140 468
37 164 85 322 164 121 443
38 166 93 349 162 128 534
39 184 83 504 152 152 459
40 179 234 666 156 179 469
41 188 86 661 154 138 558
42 180 72 666 157 114 461
43 171 87 663 184 127 557
44 189 91 532 211 147 474
45 202 93 712 156 140 464
46 341 79 662 162 103 541
47 206 101 658 184 102 535
48 153 85 634 173 149 552
49 374 164 648 187 141 674
50 193 72 638 169 142 568
51 149 92 643 177 137 681
52 213 105 642 167 176 580
53 195 127 653 240 121 498
54 152 74 579 235 109 592
55 165 81 584 223 150 616
56 190 107 595 241 113 586
57 375 74 667 268 137 560
58 256 128 701 248 151 621
59 140 81 624 235 141 580
60 164 92 701 236 133 599
61 181 110 991 252 186 646
62 396 88 980 190 129 617
63 220 147 630 202 158 690
64 164 78 920 232 215 847
65 230 196 652 241 147 580
66 192 89 27 253 141 759
67 230 152 975 212 119 641
68 149 90 14 196 132 627
69 167 231 866 243 142 647
70 160 98 153 247 116 666
71 204 137 980 278 150 679
72 142 81 791 268 168 648
73 209 165 122 241 147 703
74 172 77 939 194 132 595
75 145 80 81 209 154 690
76 134 91 25 261 119 913
77 151 77 962 241 129 799
78 139 75 54 261 101 746
79 151 95 904 453 121 672
80 131 78 184 252 150 741
81 142 82 26 268 109 745
82 170 74 963 261 165 658
83 157 78 970 263 191 778
84 329 78 320 309 112 756
85 136 78 326 314 114 734
86 140 89 324 306 136 722
87 144 186 339 321 117 671
88 164 76 316 307 110 792
89 154 81 326 327 127 820
90 143 77 315 267 114 743
91 156 94 19 272 137 772
92 137 184 152 312 116 868
93 214 136 127 329 83 815
94 136 78 301 324 114 819
95 131 77 158 342 116 741
96 144 77 210 331 155 824
97 144 84 556 324 95 862
98 191 134 257 284 120 862
99 143 80 492 340 117 879

On Fri, Jun 1, 2018 at 3:48 PM, Apoorv Naik  wrote:

> Can you share the text of the benchmarks you performed ? My email is not
> rendering the attached image.
>
> From: Pierre Padovani 
> Reply-To: "dev@atlas.apache.org" 
> Date: Friday, June 1, 2018 at 12:49 PM
> To: "dev@atlas.apache.org" 
> Subject: Re: Performance of relationships
>
> I was in a hurry writing the previous email and realized that I did not
> specify that the relationship type was COMPOSITION.
>
> On Fri, Jun 1, 2018 at 2:37 PM, Pierre Padovani  civitaslearning.com<mailto:pierre.padov...@civitaslearning.com>> wrote:
> All,
>
> I decided to spend a few cycles performing from basic performance tests to
> help us refine our type system. The goal was to determine how we would add
> relationships to our entities and if we would use the new relationship APIs
> or not. The test was simple. I created a type called container and a type
> called child both extending DataSet. I think created an instance of
> container, and added one child at a time to the container recording the
> cost of the add, and the cost of fetching the container after each add.
>
> There were three tests run:
>
> - Legacy - Assign the entities to the attributes in the attribute map
> - Hybrid - Leave the legacy attributes in the type system, and use the
> relationship APIs
> - V2 - Remove the legacy attributes and use the relationship APIs
>
> Here are the results for 100 children:
>
> [cid:ii_jhwdc79w0_163bcd934880f531]
>
> Is there a reason we are continuing to use legacy attributes in the type
> system(s) we are shipping with Atlas? From a performance perspective this
> does not seem like a reasonable thing to do.
>
> Thanks!
>
> Pierre
>
>
>


Re: Performance of relationships

2018-06-01 Thread Pierre Padovani
I was in a hurry writing the previous email and realized that I did not
specify that the relationship type was COMPOSITION.

On Fri, Jun 1, 2018 at 2:37 PM, Pierre Padovani <
pierre.padov...@civitaslearning.com> wrote:

> All,
>
> I decided to spend a few cycles performing from basic performance tests to
> help us refine our type system. The goal was to determine how we would add
> relationships to our entities and if we would use the new relationship APIs
> or not. The test was simple. I created a type called container and a type
> called child both extending DataSet. I think created an instance of
> container, and added one child at a time to the container recording the
> cost of the add, and the cost of fetching the container after each add.
>
> There were three tests run:
>
> - Legacy - Assign the entities to the attributes in the attribute map
> - Hybrid - Leave the legacy attributes in the type system, and use the
> relationship APIs
> - V2 - Remove the legacy attributes and use the relationship APIs
>
> Here are the results for 100 children:
>
>
>
> Is there a reason we are continuing to use legacy attributes in the type
> system(s) we are shipping with Atlas? From a performance perspective this
> does not seem like a reasonable thing to do.
>
> Thanks!
>
> Pierre
>
>


Performance of relationships

2018-06-01 Thread Pierre Padovani
All,

I decided to spend a few cycles performing from basic performance tests to
help us refine our type system. The goal was to determine how we would add
relationships to our entities and if we would use the new relationship APIs
or not. The test was simple. I created a type called container and a type
called child both extending DataSet. I think created an instance of
container, and added one child at a time to the container recording the
cost of the add, and the cost of fetching the container after each add.

There were three tests run:

- Legacy - Assign the entities to the attributes in the attribute map
- Hybrid - Leave the legacy attributes in the type system, and use the
relationship APIs
- V2 - Remove the legacy attributes and use the relationship APIs

Here are the results for 100 children:



Is there a reason we are continuing to use legacy attributes in the type
system(s) we are shipping with Atlas? From a performance perspective this
does not seem like a reasonable thing to do.

Thanks!

Pierre


[jira] [Resolved] (ATLAS-2727) Cannot delete a child of a COMPOSITION/AGGREGATION relationship

2018-05-31 Thread Pierre Padovani (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-2727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Padovani resolved ATLAS-2727.

   Resolution: Fixed
Fix Version/s: 1.0.0

> Cannot delete a child of a COMPOSITION/AGGREGATION relationship
> ---
>
> Key: ATLAS-2727
> URL: https://issues.apache.org/jira/browse/ATLAS-2727
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 1.0.0
>    Reporter: Pierre Padovani
>Priority: Critical
> Fix For: 1.0.0
>
>
> I am unable to delete a child of a composition/aggregation relationship. I 
> get this exception:
> {code:java}
> java.lang.IllegalArgumentException: Invalid edge label r:DiscoveryPackTables: 
> expected 2 or 3 label components but found 1 at 
> org.apache.atlas.repository.graph.AtlasEdgeLabel.(AtlasEdgeLabel.java:37)
>  at 
> org.apache.atlas.repository.store.graph.v1.DeleteHandlerV1.getAttributeForEdge(DeleteHandlerV1.java:722)
>  at 
> org.apache.atlas.repository.store.graph.v1.DeleteHandlerV1.deleteVertex(DeleteHandlerV1.java:865)
>  at 
> org.apache.atlas.repository.store.graph.v1.DeleteHandlerV1.deleteTypeVertex(DeleteHandlerV1.java:718)
>  at 
> org.apache.atlas.repository.store.graph.v1.DeleteHandlerV1.deleteEntities(DeleteHandlerV1.java:140)
>  at 
> org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.deleteVertices(AtlasEntityStoreV1.java:704)
>  at 
> org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.deleteById(AtlasEntityStoreV1.java:297)
>  at 
> org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1$$FastClassBySpringCGLIB$$80c00649.invoke()
>  at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) 
> at 
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
>  at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
>  at 
> org.apache.atlas.GraphTransactionInterceptor.invoke(GraphTransactionInterceptor.java:75)
>  at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
>  at 
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
>  at 
> org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1$$EnhancerBySpringCGLIB$$2072786c.deleteById()
>  at org.apache.atlas.web.rest.EntityREST.deleteByGuid(EntityREST.java:327) at 
> sun.reflect.GeneratedMethodAccessor231.invoke(Unknown Source) at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498) at 
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> {code}
> Type system:
> {code:java}
> Entites:
> {
>   "name": "DiscoveryPack",
>   "superTypes": [
> "DataSet"
>   ],
>   "typeVersion": "0.1",
>   "attributeDefs": [
>   ]
> },
> {
>   "name": "DiscoveryPackTable",
>   "superTypes": [
> "Table"
>   ],
>   "typeVersion": "0.1",
>   "attributeDefs": [
>   ]
> },
> {
>   "name": "Table",
>   "superTypes": [
> "DataSet"
>   ],
>   "typeVersion": "0.1",
>   "attributeDefs": [
>   ]
> }
> relationship:
> {
>   "endDef1": {
> "type": "DiscoveryPack",
> "name": "tables",
> "isContainer": true,
> "cardinality": "SET",
> "isLegacyAttribute": false
>   },
>   "endDef2": {
> "type": "DiscoveryPackTable",
> "name": "discoveryPack",
> "isContainer": false,
> "cardinality": "SINGLE",
> "isLegacyAttribute": false
>   },
>   "name": "DiscoveryPackTables",
>   "propagateTags": "ONE_TO_TWO",
>   "relationshipCategory": "COMPOSITION",
>   "typeVersion": "1.0"
> }{code}
>  
>  * Create a DiscoveryPack
>  * Create a DiscoveryPackTable.
>  * Create a relationship between them
>  * Delete the DiscoveryPackTable created previously
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Closed] (ATLAS-2727) Cannot delete a child of a COMPOSITION/AGGREGATION relationship

2018-05-31 Thread Pierre Padovani (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-2727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Padovani closed ATLAS-2727.
--

> Cannot delete a child of a COMPOSITION/AGGREGATION relationship
> ---
>
> Key: ATLAS-2727
> URL: https://issues.apache.org/jira/browse/ATLAS-2727
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 1.0.0
>    Reporter: Pierre Padovani
>Priority: Critical
> Fix For: 1.0.0
>
>
> I am unable to delete a child of a composition/aggregation relationship. I 
> get this exception:
> {code:java}
> java.lang.IllegalArgumentException: Invalid edge label r:DiscoveryPackTables: 
> expected 2 or 3 label components but found 1 at 
> org.apache.atlas.repository.graph.AtlasEdgeLabel.(AtlasEdgeLabel.java:37)
>  at 
> org.apache.atlas.repository.store.graph.v1.DeleteHandlerV1.getAttributeForEdge(DeleteHandlerV1.java:722)
>  at 
> org.apache.atlas.repository.store.graph.v1.DeleteHandlerV1.deleteVertex(DeleteHandlerV1.java:865)
>  at 
> org.apache.atlas.repository.store.graph.v1.DeleteHandlerV1.deleteTypeVertex(DeleteHandlerV1.java:718)
>  at 
> org.apache.atlas.repository.store.graph.v1.DeleteHandlerV1.deleteEntities(DeleteHandlerV1.java:140)
>  at 
> org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.deleteVertices(AtlasEntityStoreV1.java:704)
>  at 
> org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.deleteById(AtlasEntityStoreV1.java:297)
>  at 
> org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1$$FastClassBySpringCGLIB$$80c00649.invoke()
>  at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) 
> at 
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
>  at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
>  at 
> org.apache.atlas.GraphTransactionInterceptor.invoke(GraphTransactionInterceptor.java:75)
>  at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
>  at 
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
>  at 
> org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1$$EnhancerBySpringCGLIB$$2072786c.deleteById()
>  at org.apache.atlas.web.rest.EntityREST.deleteByGuid(EntityREST.java:327) at 
> sun.reflect.GeneratedMethodAccessor231.invoke(Unknown Source) at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498) at 
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> {code}
> Type system:
> {code:java}
> Entites:
> {
>   "name": "DiscoveryPack",
>   "superTypes": [
> "DataSet"
>   ],
>   "typeVersion": "0.1",
>   "attributeDefs": [
>   ]
> },
> {
>   "name": "DiscoveryPackTable",
>   "superTypes": [
> "Table"
>   ],
>   "typeVersion": "0.1",
>   "attributeDefs": [
>   ]
> },
> {
>   "name": "Table",
>   "superTypes": [
> "DataSet"
>   ],
>   "typeVersion": "0.1",
>   "attributeDefs": [
>   ]
> }
> relationship:
> {
>   "endDef1": {
> "type": "DiscoveryPack",
> "name": "tables",
> "isContainer": true,
> "cardinality": "SET",
> "isLegacyAttribute": false
>   },
>   "endDef2": {
> "type": "DiscoveryPackTable",
> "name": "discoveryPack",
> "isContainer": false,
> "cardinality": "SINGLE",
> "isLegacyAttribute": false
>   },
>   "name": "DiscoveryPackTables",
>   "propagateTags": "ONE_TO_TWO",
>   "relationshipCategory": "COMPOSITION",
>   "typeVersion": "1.0"
> }{code}
>  
>  * Create a DiscoveryPack
>  * Create a DiscoveryPackTable.
>  * Create a relationship between them
>  * Delete the DiscoveryPackTable created previously
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: Problem deleting edges with 1.0.0 - pre rc

2018-05-31 Thread Pierre Padovani
Yep... issue is fixed.

Sorry for the false alarm.

On Thu, May 31, 2018 at 12:19 PM, Pierre Padovani <
pierre.padov...@civitaslearning.com> wrote:

> Ok... thanks.. couldn't find anything on my search. I'll rebuild with
> latest today... if latest fixes the issue then I'll close my issue as fixed.
>
> On Thu, May 31, 2018 at 12:12 PM, Apoorv Naik 
> wrote:
>
>> Hi Pierre,
>>
>> Are you testing this with the rc 2 or a build prior to that ? The delete
>> issue was filed in one of the recent commits (maybe a week ago), I don't
>> remember the issue number off the top of my head. Let me share the same
>> once I get hold of the details.
>>
>> Sent from Nine<http://www.9folders.com/>
>> 
>> From: Pierre Padovani 
>> Sent: Thursday, May 31, 2018 10:07 AM
>> To: dev@atlas.apache.org; Madhan Neethiraj
>> Subject: Re: Problem deleting edges with 1.0.0 - pre rc
>>
>> Created issue: https://issues.apache.org/jira/browse/ATLAS-2727
>>
>>
>>
>> On Thu, May 31, 2018 at 11:47 AM, Pierre Padovani <
>> pierre.padov...@civitaslearning.com> wrote:
>>
>> > Sorry for the spam... One last data point. It looks like the deletion
>> of B
>> > is the issue. I can delete A, but this doesn't delete B.
>> >
>> > On Thu, May 31, 2018 at 11:21 AM, Pierre Padovani > > civitaslearning.com> wrote:
>> >
>> >> The level we are using is a few weeks old, but I didn't see an issue
>> that
>> >> fixed this in the interim.
>> >>
>> >> Problem:
>> >>
>> >> Entity A -> Composition Relationship -> Entity B
>> >>
>> >> Delete Entity A and Entity B
>> >>
>> >> I get the following exception trace:
>> >>
>> >> java.lang.IllegalArgumentException: Invalid edge label
>> >> r:DiscoveryPackTables: expected 2 or 3 label components but found 1 at
>> >> org.apache.atlas.repository.graph.AtlasEdgeLabel.(Atla
>> sEdgeLabel.java:37)
>> >> at org.apache.atlas.repository.store.graph.v1.DeleteHandlerV1.g
>> >> etAttributeForEdge(DeleteHandlerV1.java:722) at
>> >> org.apache.atlas.repository.store.graph.v1.DeleteHandlerV1.
>> >> deleteVertex(DeleteHandlerV1.java:865) at
>> org.apache.atlas.repository.st
>> >> ore.graph.v1.DeleteHandlerV1.deleteTypeVertex(DeleteHandlerV
>> 1.java:718)
>> >> at org.apache.atlas.repository.store.graph.v1.DeleteHandlerV1.d
>> >> eleteEntities(DeleteHandlerV1.java:140) at
>> org.apache.atlas.repository.st
>> >> ore.graph.v1.AtlasEntityStoreV1.deleteVertices(AtlasEntitySt
>> oreV1.java:704)
>> >> at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV
>> >> 1.deleteById(AtlasEntityStoreV1.java:297) at
>> >> org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV
>> >> 1$$FastClassBySpringCGLIB$$80c00649.invoke() at
>> >> org.springframework.cglib.proxy.MethodProxy.invoke(MethodPro
>> xy.java:204)
>> >> at org.springframework.aop.framework.CglibAopProxy$CglibMethodI
>> >> nvocation.invokeJoinpoint(CglibAopProxy.java:738) at
>> >> org.springframework.aop.framework.ReflectiveMethodInvocation
>> >> .proceed(ReflectiveMethodInvocation.java:157) at
>> >> org.apache.atlas.GraphTransactionInterceptor.invoke(GraphTra
>> nsactionInterceptor.java:75)
>> >> at org.springframework.aop.framework.ReflectiveMethodInvocation
>> >> .proceed(ReflectiveMethodInvocation.java:179) at
>> >> org.springframework.aop.framework.CglibAopProxy$DynamicAdvis
>> >> edInterceptor.intercept(CglibAopProxy.java:673) at
>> >> org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV
>> >> 1$$EnhancerBySpringCGLIB$$2072786c.deleteById() at
>> >> org.apache.atlas.web.rest.EntityREST.deleteByGuid(EntityREST.java:327)
>> >> at sun.reflect.GeneratedMethodAccessor231.invoke(Unknown Source) at
>> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>> thodAccessorImpl.java:43)
>> >> at java.lang.reflect.Method.invoke(Method.java:498) at
>> >> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invo
>> >> ke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.mod
>> >> el.method.dispatch.AbstractResourceMethodDispatchProvider$
>> >> TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvi
>> der.java:185)
>> >> at com.sun.jersey.server.impl.model.method

Re: Problem deleting edges with 1.0.0 - pre rc

2018-05-31 Thread Pierre Padovani
Ok... thanks.. couldn't find anything on my search. I'll rebuild with
latest today... if latest fixes the issue then I'll close my issue as fixed.

On Thu, May 31, 2018 at 12:12 PM, Apoorv Naik  wrote:

> Hi Pierre,
>
> Are you testing this with the rc 2 or a build prior to that ? The delete
> issue was filed in one of the recent commits (maybe a week ago), I don't
> remember the issue number off the top of my head. Let me share the same
> once I get hold of the details.
>
> Sent from Nine<http://www.9folders.com/>
> ____
> From: Pierre Padovani 
> Sent: Thursday, May 31, 2018 10:07 AM
> To: dev@atlas.apache.org; Madhan Neethiraj
> Subject: Re: Problem deleting edges with 1.0.0 - pre rc
>
> Created issue: https://issues.apache.org/jira/browse/ATLAS-2727
>
>
>
> On Thu, May 31, 2018 at 11:47 AM, Pierre Padovani <
> pierre.padov...@civitaslearning.com> wrote:
>
> > Sorry for the spam... One last data point. It looks like the deletion of
> B
> > is the issue. I can delete A, but this doesn't delete B.
> >
> > On Thu, May 31, 2018 at 11:21 AM, Pierre Padovani  > civitaslearning.com> wrote:
> >
> >> The level we are using is a few weeks old, but I didn't see an issue
> that
> >> fixed this in the interim.
> >>
> >> Problem:
> >>
> >> Entity A -> Composition Relationship -> Entity B
> >>
> >> Delete Entity A and Entity B
> >>
> >> I get the following exception trace:
> >>
> >> java.lang.IllegalArgumentException: Invalid edge label
> >> r:DiscoveryPackTables: expected 2 or 3 label components but found 1 at
> >> org.apache.atlas.repository.graph.AtlasEdgeLabel.(
> AtlasEdgeLabel.java:37)
> >> at org.apache.atlas.repository.store.graph.v1.DeleteHandlerV1.g
> >> etAttributeForEdge(DeleteHandlerV1.java:722) at
> >> org.apache.atlas.repository.store.graph.v1.DeleteHandlerV1.
> >> deleteVertex(DeleteHandlerV1.java:865) at
> org.apache.atlas.repository.st
> >> ore.graph.v1.DeleteHandlerV1.deleteTypeVertex(DeleteHandlerV1.java:718)
> >> at org.apache.atlas.repository.store.graph.v1.DeleteHandlerV1.d
> >> eleteEntities(DeleteHandlerV1.java:140) at
> org.apache.atlas.repository.st
> >> ore.graph.v1.AtlasEntityStoreV1.deleteVertices(
> AtlasEntityStoreV1.java:704)
> >> at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV
> >> 1.deleteById(AtlasEntityStoreV1.java:297) at
> >> org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV
> >> 1$$FastClassBySpringCGLIB$$80c00649.invoke() at
> >> org.springframework.cglib.proxy.MethodProxy.invoke(
> MethodProxy.java:204)
> >> at org.springframework.aop.framework.CglibAopProxy$CglibMethodI
> >> nvocation.invokeJoinpoint(CglibAopProxy.java:738) at
> >> org.springframework.aop.framework.ReflectiveMethodInvocation
> >> .proceed(ReflectiveMethodInvocation.java:157) at
> >> org.apache.atlas.GraphTransactionInterceptor.invoke(
> GraphTransactionInterceptor.java:75)
> >> at org.springframework.aop.framework.ReflectiveMethodInvocation
> >> .proceed(ReflectiveMethodInvocation.java:179) at
> >> org.springframework.aop.framework.CglibAopProxy$DynamicAdvis
> >> edInterceptor.intercept(CglibAopProxy.java:673) at
> >> org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV
> >> 1$$EnhancerBySpringCGLIB$$2072786c.deleteById() at
> >> org.apache.atlas.web.rest.EntityREST.deleteByGuid(EntityREST.java:327)
> >> at sun.reflect.GeneratedMethodAccessor231.invoke(Unknown Source) at
> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> >> at java.lang.reflect.Method.invoke(Method.java:498) at
> >> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invo
> >> ke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.mod
> >> el.method.dispatch.AbstractResourceMethodDispatchProvider$
> >> TypeOutInvoker._dispatch(AbstractResourceMethodDispatch
> Provider.java:185)
> >> at com.sun.jersey.server.impl.model.method.dispatch.ResourceJav
> >> aMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at
> >> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.
> accept(HttpMethodRule.java:302)
> >> at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.
> accept(RightHandPathRule.java:147)
> >> at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.
> accept(ResourceClassRule.java:108)
> >> at com.sun.jersey.server.impl.uri.rules.RightHandPa

[jira] [Created] (ATLAS-2727) Cannot delete a child of a COMPOSITION/AGGREGATION relationship

2018-05-31 Thread Pierre Padovani (JIRA)
Pierre Padovani created ATLAS-2727:
--

 Summary: Cannot delete a child of a COMPOSITION/AGGREGATION 
relationship
 Key: ATLAS-2727
 URL: https://issues.apache.org/jira/browse/ATLAS-2727
 Project: Atlas
  Issue Type: Bug
Affects Versions: 1.0.0
Reporter: Pierre Padovani


I am unable to delete a child of a composition/aggregation relationship. I get 
this exception:


{code:java}
java.lang.IllegalArgumentException: Invalid edge label r:DiscoveryPackTables: 
expected 2 or 3 label components but found 1 at 
org.apache.atlas.repository.graph.AtlasEdgeLabel.(AtlasEdgeLabel.java:37) 
at 
org.apache.atlas.repository.store.graph.v1.DeleteHandlerV1.getAttributeForEdge(DeleteHandlerV1.java:722)
 at 
org.apache.atlas.repository.store.graph.v1.DeleteHandlerV1.deleteVertex(DeleteHandlerV1.java:865)
 at 
org.apache.atlas.repository.store.graph.v1.DeleteHandlerV1.deleteTypeVertex(DeleteHandlerV1.java:718)
 at 
org.apache.atlas.repository.store.graph.v1.DeleteHandlerV1.deleteEntities(DeleteHandlerV1.java:140)
 at 
org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.deleteVertices(AtlasEntityStoreV1.java:704)
 at 
org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.deleteById(AtlasEntityStoreV1.java:297)
 at 
org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1$$FastClassBySpringCGLIB$$80c00649.invoke()
 at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at 
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
 at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
 at 
org.apache.atlas.GraphTransactionInterceptor.invoke(GraphTransactionInterceptor.java:75)
 at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
 at 
org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
 at 
org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1$$EnhancerBySpringCGLIB$$2072786c.deleteById()
 at org.apache.atlas.web.rest.EntityREST.deleteByGuid(EntityREST.java:327) at 
sun.reflect.GeneratedMethodAccessor231.invoke(Unknown Source) at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498) at 
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
{code}
Type system:
{code:java}
Entites:
{
  "name": "DiscoveryPack",
  "superTypes": [
"DataSet"
  ],
  "typeVersion": "0.1",
  "attributeDefs": [
  ]
},
{
  "name": "DiscoveryPackTable",
  "superTypes": [
"Table"
  ],
  "typeVersion": "0.1",
  "attributeDefs": [
  ]
},
{
  "name": "Table",
  "superTypes": [
"DataSet"
  ],
  "typeVersion": "0.1",
  "attributeDefs": [
  ]
}

relationship:
{
  "endDef1": {
"type": "DiscoveryPack",
"name": "tables",
"isContainer": true,
"cardinality": "SET",
"isLegacyAttribute": false
  },
  "endDef2": {
"type": "DiscoveryPackTable",
"name": "discoveryPack",
"isContainer": false,
"cardinality": "SINGLE",
"isLegacyAttribute": false
  },
  "name": "DiscoveryPackTables",
  "propagateTags": "ONE_TO_TWO",
  "relationshipCategory": "COMPOSITION",
  "typeVersion": "1.0"
}{code}
 
 * Create a DiscoveryPack
 * Create a DiscoveryPackTable.
 * Create a relationship between them
 * Delete the DiscoveryPackTable created previously

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: Problem deleting edges with 1.0.0 - pre rc

2018-05-31 Thread Pierre Padovani
Created issue: https://issues.apache.org/jira/browse/ATLAS-2727



On Thu, May 31, 2018 at 11:47 AM, Pierre Padovani <
pierre.padov...@civitaslearning.com> wrote:

> Sorry for the spam... One last data point. It looks like the deletion of B
> is the issue. I can delete A, but this doesn't delete B.
>
> On Thu, May 31, 2018 at 11:21 AM, Pierre Padovani  civitaslearning.com> wrote:
>
>> The level we are using is a few weeks old, but I didn't see an issue that
>> fixed this in the interim.
>>
>> Problem:
>>
>> Entity A -> Composition Relationship -> Entity B
>>
>> Delete Entity A and Entity B
>>
>> I get the following exception trace:
>>
>> java.lang.IllegalArgumentException: Invalid edge label
>> r:DiscoveryPackTables: expected 2 or 3 label components but found 1 at
>> org.apache.atlas.repository.graph.AtlasEdgeLabel.(AtlasEdgeLabel.java:37)
>> at org.apache.atlas.repository.store.graph.v1.DeleteHandlerV1.g
>> etAttributeForEdge(DeleteHandlerV1.java:722) at
>> org.apache.atlas.repository.store.graph.v1.DeleteHandlerV1.
>> deleteVertex(DeleteHandlerV1.java:865) at org.apache.atlas.repository.st
>> ore.graph.v1.DeleteHandlerV1.deleteTypeVertex(DeleteHandlerV1.java:718)
>> at org.apache.atlas.repository.store.graph.v1.DeleteHandlerV1.d
>> eleteEntities(DeleteHandlerV1.java:140) at org.apache.atlas.repository.st
>> ore.graph.v1.AtlasEntityStoreV1.deleteVertices(AtlasEntityStoreV1.java:704)
>> at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV
>> 1.deleteById(AtlasEntityStoreV1.java:297) at
>> org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV
>> 1$$FastClassBySpringCGLIB$$80c00649.invoke() at
>> org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
>> at org.springframework.aop.framework.CglibAopProxy$CglibMethodI
>> nvocation.invokeJoinpoint(CglibAopProxy.java:738) at
>> org.springframework.aop.framework.ReflectiveMethodInvocation
>> .proceed(ReflectiveMethodInvocation.java:157) at
>> org.apache.atlas.GraphTransactionInterceptor.invoke(GraphTransactionInterceptor.java:75)
>> at org.springframework.aop.framework.ReflectiveMethodInvocation
>> .proceed(ReflectiveMethodInvocation.java:179) at
>> org.springframework.aop.framework.CglibAopProxy$DynamicAdvis
>> edInterceptor.intercept(CglibAopProxy.java:673) at
>> org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV
>> 1$$EnhancerBySpringCGLIB$$2072786c.deleteById() at
>> org.apache.atlas.web.rest.EntityREST.deleteByGuid(EntityREST.java:327)
>> at sun.reflect.GeneratedMethodAccessor231.invoke(Unknown Source) at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498) at
>> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invo
>> ke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.mod
>> el.method.dispatch.AbstractResourceMethodDispatchProvider$
>> TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
>> at com.sun.jersey.server.impl.model.method.dispatch.ResourceJav
>> aMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at
>> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
>> at 
>> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
>> at 
>> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
>> at 
>> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
>> at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule
>> .accept(RootResourceClassesRule.java:84) at
>> com.sun.jersey.server.impl.application.WebApplicationImpl._h
>> andleRequest(WebApplicationImpl.java:1542) at
>> com.sun.jersey.server.impl.application.WebApplicationImpl._h
>> andleRequest(WebApplicationImpl.java:1473) at
>> com.sun.jersey.server.impl.application.WebApplicationImpl.ha
>> ndleRequest(WebApplicationImpl.java:1419) at
>> com.sun.jersey.server.impl.application.WebApplicationImpl.ha
>> ndleRequest(WebApplicationImpl.java:1409) at
>> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
>> at 
>> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
>> at 
>> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at
>> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845)
>> at 
>> org.eclipse.jetty.servlet.Servl

Re: Problem deleting edges with 1.0.0 - pre rc

2018-05-31 Thread Pierre Padovani
Sorry for the spam... One last data point. It looks like the deletion of B
is the issue. I can delete A, but this doesn't delete B.

On Thu, May 31, 2018 at 11:21 AM, Pierre Padovani <
pierre.padov...@civitaslearning.com> wrote:

> The level we are using is a few weeks old, but I didn't see an issue that
> fixed this in the interim.
>
> Problem:
>
> Entity A -> Composition Relationship -> Entity B
>
> Delete Entity A and Entity B
>
> I get the following exception trace:
>
> java.lang.IllegalArgumentException: Invalid edge label
> r:DiscoveryPackTables: expected 2 or 3 label components but found 1 at
> org.apache.atlas.repository.graph.AtlasEdgeLabel.(AtlasEdgeLabel.java:37)
> at org.apache.atlas.repository.store.graph.v1.DeleteHandlerV1.
> getAttributeForEdge(DeleteHandlerV1.java:722) at
> org.apache.atlas.repository.store.graph.v1.DeleteHandlerV1.deleteVertex(DeleteHandlerV1.java:865)
> at org.apache.atlas.repository.store.graph.v1.DeleteHandlerV1.
> deleteTypeVertex(DeleteHandlerV1.java:718) at org.apache.atlas.repository.
> store.graph.v1.DeleteHandlerV1.deleteEntities(DeleteHandlerV1.java:140)
> at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.
> deleteVertices(AtlasEntityStoreV1.java:704) at
> org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.deleteById(AtlasEntityStoreV1.java:297)
> at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1$$
> FastClassBySpringCGLIB$$80c00649.invoke() at
> org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
> at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.
> invokeJoinpoint(CglibAopProxy.java:738) at org.springframework.aop.
> framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
> at org.apache.atlas.GraphTransactionInterceptor.invoke(
> GraphTransactionInterceptor.java:75) at org.springframework.aop.framework.
> ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
> at org.springframework.aop.framework.CglibAopProxy$
> DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673) at
> org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1$$
> EnhancerBySpringCGLIB$$2072786c.deleteById() at
> org.apache.atlas.web.rest.EntityREST.deleteByGuid(EntityREST.java:327) at
> sun.reflect.GeneratedMethodAccessor231.invoke(Unknown Source) at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43) at 
> java.lang.reflect.Method.invoke(Method.java:498)
> at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(
> JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.
> model.method.dispatch.AbstractResourceMethodDispatch
> Provider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
> at com.sun.jersey.server.impl.model.method.dispatch.
> ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.
> accept(HttpMethodRule.java:302) at com.sun.jersey.server.impl.
> uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.
> accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.
> uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(
> RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.
> application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
> at com.sun.jersey.server.impl.application.WebApplicationImpl._
> handleRequest(WebApplicationImpl.java:1473) at com.sun.jersey.server.impl.
> application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
> at com.sun.jersey.server.impl.application.WebApplicationImpl.
> handleRequest(WebApplicationImpl.java:1409) at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
> at com.sun.jersey.spi.container.servlet.ServletContainer.
> service(ServletContainer.java:558) at com.sun.jersey.spi.container.
> servlet.ServletContainer.service(ServletContainer.java:733) at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845) at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1712) at org.apache.atlas.web.filters.
> AuditFilter.doFilter(AuditFilter.java:80) at org.eclipse.jetty.servlet.
> ServletHandler$CachedChain.doFilter(ServletHandler.java:1699) at
> org.springframework.security.web.FilterChainProxy$
> VirtualFilterChain.doFilter(FilterChainProxy.java:317) at
> org.springframework.security.web.access.intercept.
> FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) at

Re: Problem deleting edges with 1.0.0 - pre rc

2018-05-31 Thread Pierre Padovani
I have another data point. If I bring up the relationships UI tab from the
A entity, I can double click on the B entity and be taken to it. However, I
cannot double click on A to go back to that entity. Here are the raw
entities:

A:
{
"referredEntities": {},
"entity": {
"typeName": "DiscoveryPack",
"attributes": {
"owner": null,
"concepts": "{\"Student\":{\"baseStore\":\"PS_PERSON\"}}",
"qualifiedName":
"a2fee698-32d6-4c7b-9adf-81270e927d9f<>discoveryPack",
"scoreSpecification": null,
"name": "discovery_pack_instance",
"description": "discovery_pack_instance_description",
"expectedSystemOfRecord": null,
"comparisonFilter": null
},
"guid": "b350d741-8c19-4a90-ae31-5cfbc796f440",
"status": "ACTIVE",
"createdBy": "admin",
"updatedBy": "admin",
"createTime": 1527784446903,
"updateTime": 1527784446903,
"version": 0,
"relationshipAttributes": {
"inputToProcesses": [],
"tables": [
{
"guid": "5f9b776a-d77a-4252-9d29-fa90d1a455dc",
"typeName": "DiscoveryPackTable",
"displayText": "table_test",
"relationshipGuid":
"0d0f5f3e-1c7e-495b-9f25-5c9404435ea8",
"relationshipStatus": "ACTIVE",
"relationshipAttributes": {
"typeName": "DiscoveryPackTables"
}
}
],
"discoveryProcesses": [],
"boundDataSources": [],
"extractDefinitions": [],
"meanings": [],
"outputFromProcesses": []
}
}
}

B:
{
"referredEntities": {},
"entity": {
"typeName": "DiscoveryPackTable",
"attributes": {
"owner": null,
"schema": "dbo",
"tableType": "TABLE",
"qualifiedName":
"table_test<>a2fee698-32d6-4c7b-9adf-81270e927d9f<>discoveryPack",
"name": "table_test",
"description": "discovery_pack_instance_description",
"fullName": "dbo.table_test",
"primaryKey": {
"typeName": "PfKeyDefinition",
"attributes": {
"columns": null,
"tableName": "table_test"
}
}
},
"guid": "5f9b776a-d77a-4252-9d29-fa90d1a455dc",
"status": "ACTIVE",
    "createdBy": "admin",
"updatedBy": "admin",
"createTime": 1527784446903,
"updateTime": 1527784446903,
"version": 0,
"relationshipAttributes": {
"inputToProcesses": [],
"columns": [],
"discoveryPack": {
"guid": "b350d741-8c19-4a90-ae31-5cfbc796f440",
"typeName": "DiscoveryPack",
"displayText": "discovery_pack_instance",
"relationshipGuid": "0d0f5f3e-1c7e-495b-9f25-5c9404435ea8",
"relationshipStatus": "ACTIVE",
"relationshipAttributes": {
"typeName": "DiscoveryPackTables"
}
},
"meanings": [],
"outputFromProcesses": []
}
}
}


On Thu, May 31, 2018 at 11:21 AM, Pierre Padovani <
pierre.padov...@civitaslearning.com> wrote:

> The level we are using is a few weeks old, but I didn't see an issue that
> fixed this in the interim.
>
> Problem:
>
> Entity A -> Composition Relationship -> Entity B
>
> Delete Entity A and Entity B
>
> I get the following exception trace:
>
> java.lang.IllegalArgumentException: Invalid edge label
> r:DiscoveryPackTables: expected 2 or 3 label components but found 1 at
> org.

Problem deleting edges with 1.0.0 - pre rc

2018-05-31 Thread Pierre Padovani
The level we are using is a few weeks old, but I didn't see an issue that
fixed this in the interim.

Problem:

Entity A -> Composition Relationship -> Entity B

Delete Entity A and Entity B

I get the following exception trace:

java.lang.IllegalArgumentException: Invalid edge label
r:DiscoveryPackTables: expected 2 or 3 label components but found 1 at
org.apache.atlas.repository.graph.AtlasEdgeLabel.(AtlasEdgeLabel.java:37)
at
org.apache.atlas.repository.store.graph.v1.DeleteHandlerV1.getAttributeForEdge(DeleteHandlerV1.java:722)
at
org.apache.atlas.repository.store.graph.v1.DeleteHandlerV1.deleteVertex(DeleteHandlerV1.java:865)
at
org.apache.atlas.repository.store.graph.v1.DeleteHandlerV1.deleteTypeVertex(DeleteHandlerV1.java:718)
at
org.apache.atlas.repository.store.graph.v1.DeleteHandlerV1.deleteEntities(DeleteHandlerV1.java:140)
at
org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.deleteVertices(AtlasEntityStoreV1.java:704)
at
org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.deleteById(AtlasEntityStoreV1.java:297)
at
org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1$$FastClassBySpringCGLIB$$80c00649.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at
org.apache.atlas.GraphTransactionInterceptor.invoke(GraphTransactionInterceptor.java:75)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at
org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
at
org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1$$EnhancerBySpringCGLIB$$2072786c.deleteById()
at org.apache.atlas.web.rest.EntityREST.deleteByGuid(EntityREST.java:327)
at sun.reflect.GeneratedMethodAccessor231.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
at
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845) at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1712)
at org.apache.atlas.web.filters.AuditFilter.doFilter(AuditFilter.java:80)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
at
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
at
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
at

Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with JanusGraph 0.2.0

2018-05-17 Thread Pierre Padovani


> On May 16, 2018, 6:11 p.m., Madhan Neethiraj wrote:
> > docs/src/site/twiki/Configuration.twiki
> > Lines 47 (patched)
> > <https://reviews.apache.org/r/66064/diff/5/?file=2024037#file2024037line47>
> >
> > Per earlier discussions, support for Elasticsearch will be tech-preview 
> > for Atlas 1.0.0 release. Please update the documentation to reflect this.
> > 
> > Does Elasticsearch support kerberos authentication? If yes, have you 
> > validated Atlas working with Elasticsearch in kerberized environment?

I updated the application properties to reflect the answers to the above. It 
does not support kerberos out of the box. There is a plugin that provides 
security from the company that makes Elasticsearch, and I referenced it as well 
as the recommended method for securing elasticsearch in the JanusGraph 
ducomentation.


- Pierre


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review203265
-------


On May 17, 2018, 2:39 p.m., Pierre Padovani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> ---
> 
> (Updated May 17, 2018, 2:39 p.m.)
> 
> 
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates 
> documentation.
> 
> Included with this patch is an update to the berkley-elasticsearch profile to 
> automatically download and include elasticsearch as a side application much 
> like solr is. Updates to the start/stop/conf scripts are included as well.
> 
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to 
> /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There 
> are six constants that are incorrectly named with a '.' (dot). This is not 
> supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** 
> the field names can be collectively thought of as an object. In the case of 
> the fields defined in the Constants.java file, 'type' is defined as a string 
> field, and 'type.name' is also defined as a string field. Elasticsearch sees 
> this as an error, since it cannot convert type to an object. The fix included 
> simply changes the field names from using a '.' (dot) to an '_' (underscore). 
> This should NOT affect compatibility with hbase/solr for new installs. For 
> existing installations, a reindex will be required as the field names will 
> have changed.
> 
> **Query**: There is a way we can simplify integration/unit tests for the 
> in-memory graph store by using a maven plugin that will download and run an 
> elasticsearch node. This is nothing more than a maven change, and change to 
> the atlas-application.properties to switch to elasticsearch from solr. I did 
> not implement this, but am curious if this change would be desired. If so, 
> this can be done with a separate ticket.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 3732556ae 
>   distro/pom.xml 379ff0d39 
>   distro/src/bin/atlas_config.py 9062da649 
>   distro/src/bin/atlas_start.py 61d69eb21 
>   distro/src/bin/atlas_stop.py 94c3d6d46 
>   distro/src/conf/atlas-env.sh 053cbd500 
>   distro/src/main/assemblies/standalone-package.xml dc88add56 
>   docs/src/site/twiki/Configuration.twiki 63c3fce96 
>   docs/src/site/twiki/HighAvailability.twiki 4270d0974 
>   docs/src/site/twiki/InstallationSteps.twiki dca0618e3 
>   graphdb/janus/pom.xml 016a09c33 
>   pom.xml 3469a393a 
>   webapp/pom.xml 03b84087f 
> 
> 
> Diff: https://reviews.apache.org/r/66064/diff/6/
> 
> 
> Testing
> ---
> 
> We currently use this fix with our Atlas setup in a fork of the Atlas code, 
> and have found no issues with it. Additionally, with the update to the 
> berkley-elasticsearch profile, I have extensively tested that profile to make 
> sure that management of Elasticsearch functioned correctly.
> 
> 
> Thanks,
> 
> Pierre Padovani
> 
>



Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with JanusGraph 0.2.0

2018-05-17 Thread Pierre Padovani

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/
---

(Updated May 17, 2018, 2:39 p.m.)


Review request for atlas, David Radley, Madhan Neethiraj, and Sarath 
Subramanian.


Changes
---

Updated the comments in the application properties for configuring 
elasticsearch: Indicated tech preview, and security options


Bugs: ATLAS-2478
https://issues.apache.org/jira/browse/ATLAS-2478


Repository: atlas


Description
---

This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates 
documentation.

Included with this patch is an update to the berkley-elasticsearch profile to 
automatically download and include elasticsearch as a side application much 
like solr is. Updates to the start/stop/conf scripts are included as well.

NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to 
/atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There 
are six constants that are incorrectly named with a '.' (dot). This is not 
supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the 
field names can be collectively thought of as an object. In the case of the 
fields defined in the Constants.java file, 'type' is defined as a string field, 
and 'type.name' is also defined as a string field. Elasticsearch sees this as 
an error, since it cannot convert type to an object. The fix included simply 
changes the field names from using a '.' (dot) to an '_' (underscore). This 
should NOT affect compatibility with hbase/solr for new installs. For existing 
installations, a reindex will be required as the field names will have changed.

**Query**: There is a way we can simplify integration/unit tests for the 
in-memory graph store by using a maven plugin that will download and run an 
elasticsearch node. This is nothing more than a maven change, and change to the 
atlas-application.properties to switch to elasticsearch from solr. I did not 
implement this, but am curious if this change would be desired. If so, this can 
be done with a separate ticket.


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/repository/Constants.java 3732556ae 
  distro/pom.xml 379ff0d39 
  distro/src/bin/atlas_config.py 9062da649 
  distro/src/bin/atlas_start.py 61d69eb21 
  distro/src/bin/atlas_stop.py 94c3d6d46 
  distro/src/conf/atlas-env.sh 053cbd500 
  distro/src/main/assemblies/standalone-package.xml dc88add56 
  docs/src/site/twiki/Configuration.twiki 63c3fce96 
  docs/src/site/twiki/HighAvailability.twiki 4270d0974 
  docs/src/site/twiki/InstallationSteps.twiki dca0618e3 
  graphdb/janus/pom.xml 016a09c33 
  pom.xml 3469a393a 
  webapp/pom.xml 03b84087f 


Diff: https://reviews.apache.org/r/66064/diff/6/

Changes: https://reviews.apache.org/r/66064/diff/5-6/


Testing
---

We currently use this fix with our Atlas setup in a fork of the Atlas code, and 
have found no issues with it. Additionally, with the update to the 
berkley-elasticsearch profile, I have extensively tested that profile to make 
sure that management of Elasticsearch functioned correctly.


Thanks,

Pierre Padovani



Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with JanusGraph 0.2.0

2018-05-16 Thread Pierre Padovani


> On May 7, 2018, 10:39 p.m., Madhan Neethiraj wrote:
> > pom.xml
> > Line 675 (original), 676 (patched)
> > <https://reviews.apache.org/r/66064/diff/3/?file=2017714#file2017714line676>
> >
> > Will this change end up including elasticsearch libraries even when 
> > building for solr?
> 
> Pierre Padovani wrote:
> Yes... however the new 5.x version of the Elasticsearch jars no longer 
> shade 3rd party libraries so are far safer than the earlier versions. So we 
> have a choice, include the jars, or force users who wish to use Elasticsearch 
> as the search backend to copy in the jars manually.
> 
> Madhan Neethiraj wrote:
> Pierre - I would suggest to not package libraries from multiple search 
> backends in Atlas. It should be done either via profiles or as you suggest by 
> manually copying necessary libraries.
> 
> Pierre Padovani wrote:
> Not to be obstinate about this... but have you tried to install a 
> different set of libraries into the default atlas distribution? The lib 
> directory doesn't exist, so you have to:
> 
> -start atlas
> -have it unpack itself
> -fail to start
> -stop atlas
> -copy in the libraries needed
> -start atlas
> 
> The above assumes the user isn't going to remove the solr jars... since 
> they likely don't know about them. In the end it is very likely that they 
> will end up with both search backend jars if they choose to use Elasticsearch.
> 
> IMHO - This indicates that there is a bigger distribution packaging issue 
> in place. What we should do is not include any backend or search jars in the 
> default war build. Instead include them in the zip, and provide a setup 
> script that allows the user to choose their setup which in turn unpacks and 
> copies in the require jars.
> 
> Finally, can you enlighten me as to the issue with including the jars in 
> the default distribution? I understand it is your preference, but I fail to 
> grasp what the conflict happens to be except for overall package size.
> 
> Madhan Neethiraj wrote:
> This removes existing exclusion of all elasicsearch libraries 
> (WEB-INF/lib/elasticsearch-*.jar). Can you list all libraries included in 
> Atlas packaging due to this change?
> 
> >> can you enlighten me as to the issue with including the jars in the 
> default distribution?
> Have you validated running Atlas with Solr after this change (which 
> packages elastic-search JARs)? If you already validated, it should be okay to 
> push this commit.

Here are the two jars that are included in the build:

elasticsearch-rest-client-5.6.4.jar 
elasticsearch-rest-high-level-client-5.6.4.jar

To verify that there is no conflict, I rebuilt with embedded hbase and solr, 
then copied in the above two jars. I was able to run, create an entity and 
search for it without any issues.


- Pierre


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review202599
---


On May 16, 2018, 2:29 p.m., Pierre Padovani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> ---
> 
> (Updated May 16, 2018, 2:29 p.m.)
> 
> 
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates 
> documentation.
> 
> Included with this patch is an update to the berkley-elasticsearch profile to 
> automatically download and include elasticsearch as a side application much 
> like solr is. Updates to the start/stop/conf scripts are included as well.
> 
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to 
> /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There 
> are six constants that are incorrectly named with a '.' (dot). This is not 
> supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** 
> the field names can be collectively thought of as an object. In the case of 
> the fields defined in the Constants.java file, 'type' is defined as a string 
> field, and 'type.name' is also defined as a string field. Elasticsearch sees 
> this as an error, since it cannot convert type to an object. The fix included 
> simply changes the field names from using

Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with JanusGraph 0.2.0

2018-05-16 Thread Pierre Padovani


> On May 7, 2018, 10:39 p.m., Madhan Neethiraj wrote:
> > pom.xml
> > Line 675 (original), 676 (patched)
> > <https://reviews.apache.org/r/66064/diff/3/?file=2017714#file2017714line676>
> >
> > Will this change end up including elasticsearch libraries even when 
> > building for solr?
> 
> Pierre Padovani wrote:
> Yes... however the new 5.x version of the Elasticsearch jars no longer 
> shade 3rd party libraries so are far safer than the earlier versions. So we 
> have a choice, include the jars, or force users who wish to use Elasticsearch 
> as the search backend to copy in the jars manually.
> 
> Madhan Neethiraj wrote:
> Pierre - I would suggest to not package libraries from multiple search 
> backends in Atlas. It should be done either via profiles or as you suggest by 
> manually copying necessary libraries.

Not to be obstinate about this... but have you tried to install a different set 
of libraries into the default atlas distribution? The lib directory doesn't 
exist, so you have to:

-start atlas
-have it unpack itself
-fail to start
-stop atlas
-copy in the libraries needed
-start atlas

The above assumes the user isn't going to remove the solr jars... since they 
likely don't know about them. In the end it is very likely that they will end 
up with both search backend jars if they choose to use Elasticsearch.

IMHO - This indicates that there is a bigger distribution packaging issue in 
place. What we should do is not include any backend or search jars in the 
default war build. Instead include them in the zip, and provide a setup script 
that allows the user to choose their setup which in turn unpacks and copies in 
the require jars.

Finally, can you enlighten me as to the issue with including the jars in the 
default distribution? I understand it is your preference, but I fail to grasp 
what the conflict happens to be except for overall package size.


- Pierre


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review202599
---


On May 16, 2018, 2:29 p.m., Pierre Padovani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> ---
> 
> (Updated May 16, 2018, 2:29 p.m.)
> 
> 
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates 
> documentation.
> 
> Included with this patch is an update to the berkley-elasticsearch profile to 
> automatically download and include elasticsearch as a side application much 
> like solr is. Updates to the start/stop/conf scripts are included as well.
> 
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to 
> /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There 
> are six constants that are incorrectly named with a '.' (dot). This is not 
> supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** 
> the field names can be collectively thought of as an object. In the case of 
> the fields defined in the Constants.java file, 'type' is defined as a string 
> field, and 'type.name' is also defined as a string field. Elasticsearch sees 
> this as an error, since it cannot convert type to an object. The fix included 
> simply changes the field names from using a '.' (dot) to an '_' (underscore). 
> This should NOT affect compatibility with hbase/solr for new installs. For 
> existing installations, a reindex will be required as the field names will 
> have changed.
> 
> **Query**: There is a way we can simplify integration/unit tests for the 
> in-memory graph store by using a maven plugin that will download and run an 
> elasticsearch node. This is nothing more than a maven change, and change to 
> the atlas-application.properties to switch to elasticsearch from solr. I did 
> not implement this, but am curious if this change would be desired. If so, 
> this can be done with a separate ticket.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 3732556ae 
>   distro/pom.xml 379ff0d39 
>   distro/src/bin/atlas_config.py 9062da649 
>   distro/src/bin/atlas_start.py 61d69eb21 
>   distro/src/bin/atlas_stop.py 94c3d6d46 
>   distro/src/conf/atlas-env.sh 053cbd500 
>   distro/src/main/assemblies/standalone-package.xml dc88add56 

Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with JanusGraph 0.2.0

2018-05-16 Thread Pierre Padovani


> On May 7, 2018, 11:41 p.m., Ashutosh Mestry wrote:
> > On executing ./bin/atlas_start.py displays these messages:
> > ./target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/conf/atlas-env.sh:
> >  line 65: MANAGE_EMBEDDED_CASSANDRA=${cassandra.embedded}: bad substitution
> > ./target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/conf/atlas-env.sh:
> >  line 68: MANAGE_LOCAL_ELASTICSEARCH=${elasticsearch.managed}: bad 
> > substitution
> > Exception: ('Could not find hbase-site.xml in %s. Please set env var 
> > HBASE_CONF_DIR to the hbase client conf dir', 
> > './target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/hbase/conf')
> > Traceback (most recent call last):
> >   File "./bin/atlas_start.py", line 163, in 
> >   File "./bin/atlas_start.py", line 92, in main
> > Exception: ('Could not find hbase-site.xml in %s. Please set env var 
> > HBASE_CONF_DIR to the hbase client conf dir', 
> > './target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/hbase/conf')
> 
> Pierre Padovani wrote:
> Fixed in the updated patch.
> 
> Pierre Padovani wrote:
> Any updates on this review?
> 
> David Radley wrote:
> Hi Pierre - there are outstanding issues against this entry - which 
> normally indicates there is work for the controbuter to do. These need to be 
> fixed or dropped before a subsequent review.

My bad, I had uploaded a new patch, but didn't click the 'fixed' buttons. I've 
rebased and uploaded a new version of the patch.


- Pierre


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review202603
---


On May 16, 2018, 2:29 p.m., Pierre Padovani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> ---
> 
> (Updated May 16, 2018, 2:29 p.m.)
> 
> 
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates 
> documentation.
> 
> Included with this patch is an update to the berkley-elasticsearch profile to 
> automatically download and include elasticsearch as a side application much 
> like solr is. Updates to the start/stop/conf scripts are included as well.
> 
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to 
> /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There 
> are six constants that are incorrectly named with a '.' (dot). This is not 
> supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** 
> the field names can be collectively thought of as an object. In the case of 
> the fields defined in the Constants.java file, 'type' is defined as a string 
> field, and 'type.name' is also defined as a string field. Elasticsearch sees 
> this as an error, since it cannot convert type to an object. The fix included 
> simply changes the field names from using a '.' (dot) to an '_' (underscore). 
> This should NOT affect compatibility with hbase/solr for new installs. For 
> existing installations, a reindex will be required as the field names will 
> have changed.
> 
> **Query**: There is a way we can simplify integration/unit tests for the 
> in-memory graph store by using a maven plugin that will download and run an 
> elasticsearch node. This is nothing more than a maven change, and change to 
> the atlas-application.properties to switch to elasticsearch from solr. I did 
> not implement this, but am curious if this change would be desired. If so, 
> this can be done with a separate ticket.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 3732556ae 
>   distro/pom.xml 379ff0d39 
>   distro/src/bin/atlas_config.py 9062da649 
>   distro/src/bin/atlas_start.py 61d69eb21 
>   distro/src/bin/atlas_stop.py 94c3d6d46 
>   distro/src/conf/atlas-env.sh 053cbd500 
>   distro/src/main/assemblies/standalone-package.xml dc88add56 
>   docs/src/site/twiki/Configuration.twiki 63c3fce96 
>   docs/src/site/twiki/HighAvailability.twiki 4270d0974 
>   docs/src/site/twiki/InstallationSteps.twiki dca0618e3 
>   graphdb/janus/pom.xml 016a09c33 
>   pom.xml 3469a393a 
>   webapp/pom.xml 03b84087f 
> 
> 
> Diff: https://re

Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with JanusGraph 0.2.0

2018-05-16 Thread Pierre Padovani

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/
---

(Updated May 16, 2018, 2:29 p.m.)


Review request for atlas, David Radley, Madhan Neethiraj, and Sarath 
Subramanian.


Changes
---

Rebased against master.


Bugs: ATLAS-2478
https://issues.apache.org/jira/browse/ATLAS-2478


Repository: atlas


Description
---

This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates 
documentation.

Included with this patch is an update to the berkley-elasticsearch profile to 
automatically download and include elasticsearch as a side application much 
like solr is. Updates to the start/stop/conf scripts are included as well.

NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to 
/atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There 
are six constants that are incorrectly named with a '.' (dot). This is not 
supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the 
field names can be collectively thought of as an object. In the case of the 
fields defined in the Constants.java file, 'type' is defined as a string field, 
and 'type.name' is also defined as a string field. Elasticsearch sees this as 
an error, since it cannot convert type to an object. The fix included simply 
changes the field names from using a '.' (dot) to an '_' (underscore). This 
should NOT affect compatibility with hbase/solr for new installs. For existing 
installations, a reindex will be required as the field names will have changed.

**Query**: There is a way we can simplify integration/unit tests for the 
in-memory graph store by using a maven plugin that will download and run an 
elasticsearch node. This is nothing more than a maven change, and change to the 
atlas-application.properties to switch to elasticsearch from solr. I did not 
implement this, but am curious if this change would be desired. If so, this can 
be done with a separate ticket.


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/repository/Constants.java 3732556ae 
  distro/pom.xml 379ff0d39 
  distro/src/bin/atlas_config.py 9062da649 
  distro/src/bin/atlas_start.py 61d69eb21 
  distro/src/bin/atlas_stop.py 94c3d6d46 
  distro/src/conf/atlas-env.sh 053cbd500 
  distro/src/main/assemblies/standalone-package.xml dc88add56 
  docs/src/site/twiki/Configuration.twiki 63c3fce96 
  docs/src/site/twiki/HighAvailability.twiki 4270d0974 
  docs/src/site/twiki/InstallationSteps.twiki dca0618e3 
  graphdb/janus/pom.xml 016a09c33 
  pom.xml 3469a393a 
  webapp/pom.xml 03b84087f 


Diff: https://reviews.apache.org/r/66064/diff/5/

Changes: https://reviews.apache.org/r/66064/diff/4-5/


Testing
---

We currently use this fix with our Atlas setup in a fork of the Atlas code, and 
have found no issues with it. Additionally, with the update to the 
berkley-elasticsearch profile, I have extensively tested that profile to make 
sure that management of Elasticsearch functioned correctly.


Thanks,

Pierre Padovani



Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with JanusGraph 0.2.0

2018-05-15 Thread Pierre Padovani


> On May 7, 2018, 11:41 p.m., Ashutosh Mestry wrote:
> > On executing ./bin/atlas_start.py displays these messages:
> > ./target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/conf/atlas-env.sh:
> >  line 65: MANAGE_EMBEDDED_CASSANDRA=${cassandra.embedded}: bad substitution
> > ./target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/conf/atlas-env.sh:
> >  line 68: MANAGE_LOCAL_ELASTICSEARCH=${elasticsearch.managed}: bad 
> > substitution
> > Exception: ('Could not find hbase-site.xml in %s. Please set env var 
> > HBASE_CONF_DIR to the hbase client conf dir', 
> > './target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/hbase/conf')
> > Traceback (most recent call last):
> >   File "./bin/atlas_start.py", line 163, in 
> >   File "./bin/atlas_start.py", line 92, in main
> > Exception: ('Could not find hbase-site.xml in %s. Please set env var 
> > HBASE_CONF_DIR to the hbase client conf dir', 
> > './target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/hbase/conf')
> 
> Pierre Padovani wrote:
> Fixed in the updated patch.

Any updates on this review?


- Pierre


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review202603
---


On May 8, 2018, 3 p.m., Pierre Padovani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> ---
> 
> (Updated May 8, 2018, 3 p.m.)
> 
> 
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates 
> documentation.
> 
> Included with this patch is an update to the berkley-elasticsearch profile to 
> automatically download and include elasticsearch as a side application much 
> like solr is. Updates to the start/stop/conf scripts are included as well.
> 
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to 
> /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There 
> are six constants that are incorrectly named with a '.' (dot). This is not 
> supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** 
> the field names can be collectively thought of as an object. In the case of 
> the fields defined in the Constants.java file, 'type' is defined as a string 
> field, and 'type.name' is also defined as a string field. Elasticsearch sees 
> this as an error, since it cannot convert type to an object. The fix included 
> simply changes the field names from using a '.' (dot) to an '_' (underscore). 
> This should NOT affect compatibility with hbase/solr for new installs. For 
> existing installations, a reindex will be required as the field names will 
> have changed.
> 
> **Query**: There is a way we can simplify integration/unit tests for the 
> in-memory graph store by using a maven plugin that will download and run an 
> elasticsearch node. This is nothing more than a maven change, and change to 
> the atlas-application.properties to switch to elasticsearch from solr. I did 
> not implement this, but am curious if this change would be desired. If so, 
> this can be done with a separate ticket.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 16a3aaa15 
>   distro/pom.xml 6431fd86d 
>   distro/src/bin/atlas_config.py 9062da649 
>   distro/src/bin/atlas_start.py 61d69eb21 
>   distro/src/bin/atlas_stop.py 94c3d6d46 
>   distro/src/conf/atlas-env.sh 053cbd500 
>   distro/src/main/assemblies/standalone-package.xml dc88add56 
>   docs/src/site/twiki/Configuration.twiki 63c3fce96 
>   docs/src/site/twiki/HighAvailability.twiki 4270d0974 
>   docs/src/site/twiki/InstallationSteps.twiki dca0618e3 
>   graphdb/janus/pom.xml 016a09c33 
>   pom.xml 3469a393a 
>   webapp/pom.xml 03b84087f 
> 
> 
> Diff: https://reviews.apache.org/r/66064/diff/4/
> 
> 
> Testing
> ---
> 
> We currently use this fix with our Atlas setup in a fork of the Atlas code, 
> and have found no issues with it. Additionally, with the update to the 
> berkley-elasticsearch profile, I have extensively tested that profile to make 
> sure that management of Elasticsearch functioned correctly.
> 
> 
> Thanks,
> 
> Pierre Padovani
> 
>



Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with JanusGraph 0.2.0

2018-05-08 Thread Pierre Padovani


> On May 7, 2018, 11:35 p.m., Ashutosh Mestry wrote:
> > graphdb/janus/pom.xml
> > Lines 121 (patched)
> > <https://reviews.apache.org/r/66064/diff/3/?file=2017713#file2017713line121>
> >
> > This is getting included even when the profile is not selected.

Per the comment about elasticsearch jars in the main distribution. 

The new version is safer now after Elastic removed all of the shaded jars... 
and smaller.

We either keep them, or force them to download and copy it into their 
distribution during setup/installation.


- Pierre


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review202602
-------


On May 8, 2018, 3 p.m., Pierre Padovani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> ---
> 
> (Updated May 8, 2018, 3 p.m.)
> 
> 
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates 
> documentation.
> 
> Included with this patch is an update to the berkley-elasticsearch profile to 
> automatically download and include elasticsearch as a side application much 
> like solr is. Updates to the start/stop/conf scripts are included as well.
> 
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to 
> /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There 
> are six constants that are incorrectly named with a '.' (dot). This is not 
> supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** 
> the field names can be collectively thought of as an object. In the case of 
> the fields defined in the Constants.java file, 'type' is defined as a string 
> field, and 'type.name' is also defined as a string field. Elasticsearch sees 
> this as an error, since it cannot convert type to an object. The fix included 
> simply changes the field names from using a '.' (dot) to an '_' (underscore). 
> This should NOT affect compatibility with hbase/solr for new installs. For 
> existing installations, a reindex will be required as the field names will 
> have changed.
> 
> **Query**: There is a way we can simplify integration/unit tests for the 
> in-memory graph store by using a maven plugin that will download and run an 
> elasticsearch node. This is nothing more than a maven change, and change to 
> the atlas-application.properties to switch to elasticsearch from solr. I did 
> not implement this, but am curious if this change would be desired. If so, 
> this can be done with a separate ticket.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 16a3aaa15 
>   distro/pom.xml 6431fd86d 
>   distro/src/bin/atlas_config.py 9062da649 
>   distro/src/bin/atlas_start.py 61d69eb21 
>   distro/src/bin/atlas_stop.py 94c3d6d46 
>   distro/src/conf/atlas-env.sh 053cbd500 
>   distro/src/main/assemblies/standalone-package.xml dc88add56 
>   docs/src/site/twiki/Configuration.twiki 63c3fce96 
>   docs/src/site/twiki/HighAvailability.twiki 4270d0974 
>   docs/src/site/twiki/InstallationSteps.twiki dca0618e3 
>   graphdb/janus/pom.xml 016a09c33 
>   pom.xml 3469a393a 
>   webapp/pom.xml 03b84087f 
> 
> 
> Diff: https://reviews.apache.org/r/66064/diff/4/
> 
> 
> Testing
> ---
> 
> We currently use this fix with our Atlas setup in a fork of the Atlas code, 
> and have found no issues with it. Additionally, with the update to the 
> berkley-elasticsearch profile, I have extensively tested that profile to make 
> sure that management of Elasticsearch functioned correctly.
> 
> 
> Thanks,
> 
> Pierre Padovani
> 
>



Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with JanusGraph 0.2.0

2018-05-08 Thread Pierre Padovani


> On May 7, 2018, 11:41 p.m., Ashutosh Mestry wrote:
> > On executing ./bin/atlas_start.py displays these messages:
> > ./target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/conf/atlas-env.sh:
> >  line 65: MANAGE_EMBEDDED_CASSANDRA=${cassandra.embedded}: bad substitution
> > ./target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/conf/atlas-env.sh:
> >  line 68: MANAGE_LOCAL_ELASTICSEARCH=${elasticsearch.managed}: bad 
> > substitution
> > Exception: ('Could not find hbase-site.xml in %s. Please set env var 
> > HBASE_CONF_DIR to the hbase client conf dir', 
> > './target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/hbase/conf')
> > Traceback (most recent call last):
> >   File "./bin/atlas_start.py", line 163, in 
> >   File "./bin/atlas_start.py", line 92, in main
> > Exception: ('Could not find hbase-site.xml in %s. Please set env var 
> > HBASE_CONF_DIR to the hbase client conf dir', 
> > './target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/hbase/conf')

Fixed in the updated patch.


- Pierre


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review202603
---


On May 8, 2018, 3 p.m., Pierre Padovani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> ---
> 
> (Updated May 8, 2018, 3 p.m.)
> 
> 
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates 
> documentation.
> 
> Included with this patch is an update to the berkley-elasticsearch profile to 
> automatically download and include elasticsearch as a side application much 
> like solr is. Updates to the start/stop/conf scripts are included as well.
> 
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to 
> /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There 
> are six constants that are incorrectly named with a '.' (dot). This is not 
> supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** 
> the field names can be collectively thought of as an object. In the case of 
> the fields defined in the Constants.java file, 'type' is defined as a string 
> field, and 'type.name' is also defined as a string field. Elasticsearch sees 
> this as an error, since it cannot convert type to an object. The fix included 
> simply changes the field names from using a '.' (dot) to an '_' (underscore). 
> This should NOT affect compatibility with hbase/solr for new installs. For 
> existing installations, a reindex will be required as the field names will 
> have changed.
> 
> **Query**: There is a way we can simplify integration/unit tests for the 
> in-memory graph store by using a maven plugin that will download and run an 
> elasticsearch node. This is nothing more than a maven change, and change to 
> the atlas-application.properties to switch to elasticsearch from solr. I did 
> not implement this, but am curious if this change would be desired. If so, 
> this can be done with a separate ticket.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 16a3aaa15 
>   distro/pom.xml 6431fd86d 
>   distro/src/bin/atlas_config.py 9062da649 
>   distro/src/bin/atlas_start.py 61d69eb21 
>   distro/src/bin/atlas_stop.py 94c3d6d46 
>   distro/src/conf/atlas-env.sh 053cbd500 
>   distro/src/main/assemblies/standalone-package.xml dc88add56 
>   docs/src/site/twiki/Configuration.twiki 63c3fce96 
>   docs/src/site/twiki/HighAvailability.twiki 4270d0974 
>   docs/src/site/twiki/InstallationSteps.twiki dca0618e3 
>   graphdb/janus/pom.xml 016a09c33 
>   pom.xml 3469a393a 
>   webapp/pom.xml 03b84087f 
> 
> 
> Diff: https://reviews.apache.org/r/66064/diff/4/
> 
> 
> Testing
> ---
> 
> We currently use this fix with our Atlas setup in a fork of the Atlas code, 
> and have found no issues with it. Additionally, with the update to the 
> berkley-elasticsearch profile, I have extensively tested that profile to make 
> sure that management of Elasticsearch functioned correctly.
> 
> 
> Thanks,
> 
> Pierre Padovani
> 
>



Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with JanusGraph 0.2.0

2018-05-08 Thread Pierre Padovani


> On May 7, 2018, 10:39 p.m., Madhan Neethiraj wrote:
> > pom.xml
> > Line 675 (original), 676 (patched)
> > <https://reviews.apache.org/r/66064/diff/3/?file=2017714#file2017714line676>
> >
> > Will this change end up including elasticsearch libraries even when 
> > building for solr?

Yes... however the new 5.x version of the Elasticsearch jars no longer shade 
3rd party libraries so are far safer than the earlier versions. So we have a 
choice, include the jars, or force users who wish to use Elasticsearch as the 
search backend to copy in the jars manually.


- Pierre


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review202599
-------


On May 8, 2018, 3 p.m., Pierre Padovani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> ---
> 
> (Updated May 8, 2018, 3 p.m.)
> 
> 
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates 
> documentation.
> 
> Included with this patch is an update to the berkley-elasticsearch profile to 
> automatically download and include elasticsearch as a side application much 
> like solr is. Updates to the start/stop/conf scripts are included as well.
> 
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to 
> /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There 
> are six constants that are incorrectly named with a '.' (dot). This is not 
> supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** 
> the field names can be collectively thought of as an object. In the case of 
> the fields defined in the Constants.java file, 'type' is defined as a string 
> field, and 'type.name' is also defined as a string field. Elasticsearch sees 
> this as an error, since it cannot convert type to an object. The fix included 
> simply changes the field names from using a '.' (dot) to an '_' (underscore). 
> This should NOT affect compatibility with hbase/solr for new installs. For 
> existing installations, a reindex will be required as the field names will 
> have changed.
> 
> **Query**: There is a way we can simplify integration/unit tests for the 
> in-memory graph store by using a maven plugin that will download and run an 
> elasticsearch node. This is nothing more than a maven change, and change to 
> the atlas-application.properties to switch to elasticsearch from solr. I did 
> not implement this, but am curious if this change would be desired. If so, 
> this can be done with a separate ticket.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 16a3aaa15 
>   distro/pom.xml 6431fd86d 
>   distro/src/bin/atlas_config.py 9062da649 
>   distro/src/bin/atlas_start.py 61d69eb21 
>   distro/src/bin/atlas_stop.py 94c3d6d46 
>   distro/src/conf/atlas-env.sh 053cbd500 
>   distro/src/main/assemblies/standalone-package.xml dc88add56 
>   docs/src/site/twiki/Configuration.twiki 63c3fce96 
>   docs/src/site/twiki/HighAvailability.twiki 4270d0974 
>   docs/src/site/twiki/InstallationSteps.twiki dca0618e3 
>   graphdb/janus/pom.xml 016a09c33 
>   pom.xml 3469a393a 
>   webapp/pom.xml 03b84087f 
> 
> 
> Diff: https://reviews.apache.org/r/66064/diff/4/
> 
> 
> Testing
> ---
> 
> We currently use this fix with our Atlas setup in a fork of the Atlas code, 
> and have found no issues with it. Additionally, with the update to the 
> berkley-elasticsearch profile, I have extensively tested that profile to make 
> sure that management of Elasticsearch functioned correctly.
> 
> 
> Thanks,
> 
> Pierre Padovani
> 
>



Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with JanusGraph 0.2.0

2018-05-08 Thread Pierre Padovani

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/
---

(Updated May 8, 2018, 3 p.m.)


Review request for atlas, David Radley, Madhan Neethiraj, and Sarath 
Subramanian.


Changes
---

Fixes based on review


Bugs: ATLAS-2478
https://issues.apache.org/jira/browse/ATLAS-2478


Repository: atlas


Description
---

This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates 
documentation.

Included with this patch is an update to the berkley-elasticsearch profile to 
automatically download and include elasticsearch as a side application much 
like solr is. Updates to the start/stop/conf scripts are included as well.

NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to 
/atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There 
are six constants that are incorrectly named with a '.' (dot). This is not 
supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the 
field names can be collectively thought of as an object. In the case of the 
fields defined in the Constants.java file, 'type' is defined as a string field, 
and 'type.name' is also defined as a string field. Elasticsearch sees this as 
an error, since it cannot convert type to an object. The fix included simply 
changes the field names from using a '.' (dot) to an '_' (underscore). This 
should NOT affect compatibility with hbase/solr for new installs. For existing 
installations, a reindex will be required as the field names will have changed.

**Query**: There is a way we can simplify integration/unit tests for the 
in-memory graph store by using a maven plugin that will download and run an 
elasticsearch node. This is nothing more than a maven change, and change to the 
atlas-application.properties to switch to elasticsearch from solr. I did not 
implement this, but am curious if this change would be desired. If so, this can 
be done with a separate ticket.


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/repository/Constants.java 16a3aaa15 
  distro/pom.xml 6431fd86d 
  distro/src/bin/atlas_config.py 9062da649 
  distro/src/bin/atlas_start.py 61d69eb21 
  distro/src/bin/atlas_stop.py 94c3d6d46 
  distro/src/conf/atlas-env.sh 053cbd500 
  distro/src/main/assemblies/standalone-package.xml dc88add56 
  docs/src/site/twiki/Configuration.twiki 63c3fce96 
  docs/src/site/twiki/HighAvailability.twiki 4270d0974 
  docs/src/site/twiki/InstallationSteps.twiki dca0618e3 
  graphdb/janus/pom.xml 016a09c33 
  pom.xml 3469a393a 
  webapp/pom.xml 03b84087f 


Diff: https://reviews.apache.org/r/66064/diff/4/

Changes: https://reviews.apache.org/r/66064/diff/3-4/


Testing
---

We currently use this fix with our Atlas setup in a fork of the Atlas code, and 
have found no issues with it. Additionally, with the update to the 
berkley-elasticsearch profile, I have extensively tested that profile to make 
sure that management of Elasticsearch functioned correctly.


Thanks,

Pierre Padovani



Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with JanusGraph 0.2.0

2018-05-07 Thread Pierre Padovani

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/
---

(Updated May 7, 2018, 9:28 p.m.)


Review request for atlas, David Radley, Madhan Neethiraj, and Sarath 
Subramanian.


Changes
---

Rebased against master


Bugs: ATLAS-2478
https://issues.apache.org/jira/browse/ATLAS-2478


Repository: atlas


Description
---

This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates 
documentation.

Included with this patch is an update to the berkley-elasticsearch profile to 
automatically download and include elasticsearch as a side application much 
like solr is. Updates to the start/stop/conf scripts are included as well.

NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to 
/atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There 
are six constants that are incorrectly named with a '.' (dot). This is not 
supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the 
field names can be collectively thought of as an object. In the case of the 
fields defined in the Constants.java file, 'type' is defined as a string field, 
and 'type.name' is also defined as a string field. Elasticsearch sees this as 
an error, since it cannot convert type to an object. The fix included simply 
changes the field names from using a '.' (dot) to an '_' (underscore). This 
should NOT affect compatibility with hbase/solr for new installs. For existing 
installations, a reindex will be required as the field names will have changed.

**Query**: There is a way we can simplify integration/unit tests for the 
in-memory graph store by using a maven plugin that will download and run an 
elasticsearch node. This is nothing more than a maven change, and change to the 
atlas-application.properties to switch to elasticsearch from solr. I did not 
implement this, but am curious if this change would be desired. If so, this can 
be done with a separate ticket.


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/repository/Constants.java 16a3aaa15 
  distro/pom.xml 6431fd86d 
  distro/src/bin/atlas_config.py 9062da649 
  distro/src/bin/atlas_start.py 61d69eb21 
  distro/src/bin/atlas_stop.py 94c3d6d46 
  distro/src/conf/atlas-env.sh 053cbd500 
  distro/src/main/assemblies/standalone-package.xml dc88add56 
  docs/src/site/twiki/Configuration.twiki 63c3fce96 
  docs/src/site/twiki/HighAvailability.twiki 4270d0974 
  docs/src/site/twiki/InstallationSteps.twiki dca0618e3 
  graphdb/janus/pom.xml 016a09c33 
  pom.xml 3469a393a 
  webapp/pom.xml 03b84087f 


Diff: https://reviews.apache.org/r/66064/diff/3/

Changes: https://reviews.apache.org/r/66064/diff/2-3/


Testing
---

We currently use this fix with our Atlas setup in a fork of the Atlas code, and 
have found no issues with it. Additionally, with the update to the 
berkley-elasticsearch profile, I have extensively tested that profile to make 
sure that management of Elasticsearch functioned correctly.


Thanks,

Pierre Padovani



Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with JanusGraph 0.2.0

2018-05-07 Thread Pierre Padovani


> On April 13, 2018, 12:34 a.m., Ashutosh Mestry wrote:
> > Can you please add instructions on how to use this profile? I tried the 
> > patch today and I am still having trouble getting Atlas up.
> 
> Pierre Padovani wrote:
> Let me rebase the patch and make some changes, and retest. Master has 
> moved a bit since I generated the original patch.  I'll update as soon as I 
> have it done.
> 
> Pierre Padovani wrote:
> I updated the patch four days ago, can you please retest?
> 
> Ashutosh Mestry wrote:
> I will retest and let you know. Please give me a couple of days.

Any updates on this?

Thanks!


- Pierre


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review201069
---


On April 13, 2018, 2:55 a.m., Pierre Padovani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> ---
> 
> (Updated April 13, 2018, 2:55 a.m.)
> 
> 
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates 
> documentation.
> 
> Included with this patch is an update to the berkley-elasticsearch profile to 
> automatically download and include elasticsearch as a side application much 
> like solr is. Updates to the start/stop/conf scripts are included as well.
> 
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to 
> /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There 
> are six constants that are incorrectly named with a '.' (dot). This is not 
> supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** 
> the field names can be collectively thought of as an object. In the case of 
> the fields defined in the Constants.java file, 'type' is defined as a string 
> field, and 'type.name' is also defined as a string field. Elasticsearch sees 
> this as an error, since it cannot convert type to an object. The fix included 
> simply changes the field names from using a '.' (dot) to an '_' (underscore). 
> This should NOT affect compatibility with hbase/solr for new installs. For 
> existing installations, a reindex will be required as the field names will 
> have changed.
> 
> **Query**: There is a way we can simplify integration/unit tests for the 
> in-memory graph store by using a maven plugin that will download and run an 
> elasticsearch node. This is nothing more than a maven change, and change to 
> the atlas-application.properties to switch to elasticsearch from solr. I did 
> not implement this, but am curious if this change would be desired. If so, 
> this can be done with a separate ticket.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 01e49157d 
>   distro/pom.xml 1f4c6d557 
>   distro/src/bin/atlas_config.py 9062da649 
>   distro/src/bin/atlas_start.py 61d69eb21 
>   distro/src/bin/atlas_stop.py 94c3d6d46 
>   distro/src/conf/atlas-env.sh 9213f488b 
>   distro/src/main/assemblies/standalone-package.xml dc2a66ba9 
>   docs/src/site/twiki/Configuration.twiki 63c3fce96 
>   docs/src/site/twiki/HighAvailability.twiki 4270d0974 
>   docs/src/site/twiki/InstallationSteps.twiki dca0618e3 
>   graphdb/janus/pom.xml 016a09c33 
>   pom.xml 4d2ab7391 
>   webapp/pom.xml 284f538f7 
> 
> 
> Diff: https://reviews.apache.org/r/66064/diff/2/
> 
> 
> Testing
> ---
> 
> We currently use this fix with our Atlas setup in a fork of the Atlas code, 
> and have found no issues with it. Additionally, with the update to the 
> berkley-elasticsearch profile, I have extensively tested that profile to make 
> sure that management of Elasticsearch functioned correctly.
> 
> 
> Thanks,
> 
> Pierre Padovani
> 
>



Re: Odd warning during search

2018-05-04 Thread Pierre Padovani
I don't have it of hand, but this is the code snippet with the Atlas client
we used:


  SearchParameters.FilterCriteria qualifiedNameCriteria = new
SearchParameters.FilterCriteria();
  qualifiedNameCriteria.setAttributeName(QUALIFIED_NAME);
  qualifiedNameCriteria.setOperator(SearchParameters.Operator.EQ);
  qualifiedNameCriteria.setAttributeValue(qualifiedName);
  SearchParameters searchParams = new SearchParameters();
  searchParams.setTypeName(typeName);
  searchParams.setExcludeDeletedEntities(true);
  searchParams.setEntityFilters(qualifiedNameCriteria);
  searchParams.setLimit(ATLAS_SEARCH_LIMIT);

  AtlasSearchResult searchResult = atlas.facetedSearch(searchParams);


On Fri, May 4, 2018 at 2:37 PM, Apoorv Naik <an...@hortonworks.com> wrote:

> What is the search criteria that you're working with ? If it's basically
> search , can you share the request JSON ?
>
> Sent from Nine<http://www.9folders.com/>
> ________
> From: Pierre Padovani <pierre.padov...@civitaslearning.com>
> Sent: Friday, May 4, 2018 12:34 PM
> To: dev@atlas.apache.org
> Subject: Odd warning during search
>
> We are seeing this warning message:
>
> not using index-search for attribute 'Referenceable.qualifiedName'; might
> cause poor performance
>
> I have not been able to find a configuration related to indexing/search or
> documentation that could help us resolve this.
>
> Can someone help clarify what would cause this?
>
> Thanks!
>
> Pierre Padovani
>


Odd warning during search

2018-05-04 Thread Pierre Padovani
We are seeing this warning message:

not using index-search for attribute 'Referenceable.qualifiedName'; might
cause poor performance

I have not been able to find a configuration related to indexing/search or
documentation that could help us resolve this.

Can someone help clarify what would cause this?

Thanks!

Pierre Padovani


Re: Cassandra support for Atlas v1.0

2018-04-27 Thread Pierre Padovani
I don't have an issue with #3, but we will be using it in production,
specifically with the Scylladb implementation.

Out of curiosity, what exactly are the issues?

On Fri, Apr 27, 2018 at 10:45 AM, Apoorv Naik  wrote:

> #3 sounds like a better option for now. +1
>
> On 4/27/18, 12:42 AM, "Nigel Jones"  wrote:
>
>
>
> On 2018/04/26 16:14:06, David Radley  wrote:
>
> > 3) Document it in the same way as the embedded solr and hbase build.
> That
>
> +1 for this option #3
>
> If in future someone has a need for production support I'm sure they
> will get engaged and help in testing etc :-)
>
>
>
>


Re: Review Request 66609: ATLAS-2564 - Reduce AtlasBaseClient.java logging verbosity to standard levels.

2018-04-23 Thread Pierre Padovani

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66609/
---

(Updated April 23, 2018, 5:11 p.m.)


Review request for atlas.


Bugs: ATLAS-2564
https://issues.apache.org/jira/browse/ATLAS-2564


Repository: atlas


Description (updated)
---

This patch reduces the verbosity of AtlasBaseClient.java, by changing the info 
lines to debug, and adding a single line log of the request/response without 
logging the payloads.


Diffs
-

  client/common/src/main/java/org/apache/atlas/AtlasBaseClient.java 551b7b6f5 


Diff: https://reviews.apache.org/r/66609/diff/1/


Testing (updated)
---

Built and deployed within our test environment for 2 days to verfiy the 
improvement in logging.

Any chance for a quick review?


Thanks,

Pierre Padovani



Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with JanusGraph 0.2.0

2018-04-23 Thread Pierre Padovani


> On April 13, 2018, 12:34 a.m., Ashutosh Mestry wrote:
> > Can you please add instructions on how to use this profile? I tried the 
> > patch today and I am still having trouble getting Atlas up.
> 
> Pierre Padovani wrote:
> Let me rebase the patch and make some changes, and retest. Master has 
> moved a bit since I generated the original patch.  I'll update as soon as I 
> have it done.

I updated the patch four days ago, can you please retest?


- Pierre


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review201069
---


On April 13, 2018, 2:55 a.m., Pierre Padovani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> ---
> 
> (Updated April 13, 2018, 2:55 a.m.)
> 
> 
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates 
> documentation.
> 
> Included with this patch is an update to the berkley-elasticsearch profile to 
> automatically download and include elasticsearch as a side application much 
> like solr is. Updates to the start/stop/conf scripts are included as well.
> 
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to 
> /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There 
> are six constants that are incorrectly named with a '.' (dot). This is not 
> supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** 
> the field names can be collectively thought of as an object. In the case of 
> the fields defined in the Constants.java file, 'type' is defined as a string 
> field, and 'type.name' is also defined as a string field. Elasticsearch sees 
> this as an error, since it cannot convert type to an object. The fix included 
> simply changes the field names from using a '.' (dot) to an '_' (underscore). 
> This should NOT affect compatibility with hbase/solr for new installs. For 
> existing installations, a reindex will be required as the field names will 
> have changed.
> 
> **Query**: There is a way we can simplify integration/unit tests for the 
> in-memory graph store by using a maven plugin that will download and run an 
> elasticsearch node. This is nothing more than a maven change, and change to 
> the atlas-application.properties to switch to elasticsearch from solr. I did 
> not implement this, but am curious if this change would be desired. If so, 
> this can be done with a separate ticket.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 01e49157d 
>   distro/pom.xml 1f4c6d557 
>   distro/src/bin/atlas_config.py 9062da649 
>   distro/src/bin/atlas_start.py 61d69eb21 
>   distro/src/bin/atlas_stop.py 94c3d6d46 
>   distro/src/conf/atlas-env.sh 9213f488b 
>   distro/src/main/assemblies/standalone-package.xml dc2a66ba9 
>   docs/src/site/twiki/Configuration.twiki 63c3fce96 
>   docs/src/site/twiki/HighAvailability.twiki 4270d0974 
>   docs/src/site/twiki/InstallationSteps.twiki dca0618e3 
>   graphdb/janus/pom.xml 016a09c33 
>   pom.xml 4d2ab7391 
>   webapp/pom.xml 284f538f7 
> 
> 
> Diff: https://reviews.apache.org/r/66064/diff/2/
> 
> 
> Testing
> ---
> 
> We currently use this fix with our Atlas setup in a fork of the Atlas code, 
> and have found no issues with it. Additionally, with the update to the 
> berkley-elasticsearch profile, I have extensively tested that profile to make 
> sure that management of Elasticsearch functioned correctly.
> 
> 
> Thanks,
> 
> Pierre Padovani
> 
>



Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with JanusGraph 0.2.0

2018-04-17 Thread Pierre Padovani


> On April 12, 2018, 11:49 p.m., Madhan Neethiraj wrote:
> > common/src/main/java/org/apache/atlas/repository/Constants.java
> > Line 56 (original), 56 (patched)
> > <https://reviews.apache.org/r/66064/diff/1/?file=1997842#file1997842line56>
> >
> > As you noted in the description, this would require re-indexing of 
> > existing deployments - which can be very expensive depending upon the 
> > amount of data. This should be avoided.
> > 
> > If an index software has restrictions on the keynames, consider 
> > replacing these constants with calls to instance-specific property key 
> > provider. And implement a default one with current values and ElasticSearch 
> > specific one with necessary key names.

I believe I have a solution in place, but would like feedback on implementation.


> On April 12, 2018, 11:49 p.m., Madhan Neethiraj wrote:
> > pom.xml
> > Line 674 (original), 675 (patched)
> > <https://reviews.apache.org/r/66064/diff/1/?file=1997853#file1997853line675>
> >
> > Would this change cause elasticsearch libraries to be included for Solr 
> > deployments as well? If yes, please consider alternate approach (like a 
> > separate profile) for Elasticsearch - to avoid potential issues in 
> > deployments that use Solr.
> > 
> > Same applies for line #694 below.

Only for the standard -Pdist profile. This shouldn't be an issue since the 
newer elasticsearch jars contain no shadded classes and should not conflict.


- Pierre


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review201061
---


On April 13, 2018, 2:55 a.m., Pierre Padovani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> ---
> 
> (Updated April 13, 2018, 2:55 a.m.)
> 
> 
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates 
> documentation.
> 
> Included with this patch is an update to the berkley-elasticsearch profile to 
> automatically download and include elasticsearch as a side application much 
> like solr is. Updates to the start/stop/conf scripts are included as well.
> 
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to 
> /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There 
> are six constants that are incorrectly named with a '.' (dot). This is not 
> supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** 
> the field names can be collectively thought of as an object. In the case of 
> the fields defined in the Constants.java file, 'type' is defined as a string 
> field, and 'type.name' is also defined as a string field. Elasticsearch sees 
> this as an error, since it cannot convert type to an object. The fix included 
> simply changes the field names from using a '.' (dot) to an '_' (underscore). 
> This should NOT affect compatibility with hbase/solr for new installs. For 
> existing installations, a reindex will be required as the field names will 
> have changed.
> 
> **Query**: There is a way we can simplify integration/unit tests for the 
> in-memory graph store by using a maven plugin that will download and run an 
> elasticsearch node. This is nothing more than a maven change, and change to 
> the atlas-application.properties to switch to elasticsearch from solr. I did 
> not implement this, but am curious if this change would be desired. If so, 
> this can be done with a separate ticket.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 01e49157d 
>   distro/pom.xml 1f4c6d557 
>   distro/src/bin/atlas_config.py 9062da649 
>   distro/src/bin/atlas_start.py 61d69eb21 
>   distro/src/bin/atlas_stop.py 94c3d6d46 
>   distro/src/conf/atlas-env.sh 9213f488b 
>   distro/src/main/assemblies/standalone-package.xml dc2a66ba9 
>   docs/src/site/twiki/Configuration.twiki 63c3fce96 
>   docs/src/site/twiki/HighAvailability.twiki 4270d0974 
>   docs/src/site/twiki/InstallationSteps.twiki dca0618e3 
>   graphdb/janus/pom.xml 016a09c33 
>   pom.xml 4d2ab7391 
>   webapp/pom.xml 284f538f7 
> 
> 
> Diff: https://reviews.apache.org/r/66064/diff/2/
> 
> 
> Testing
> ---
> 
> We currently use this fix with our Atlas setup in a fork of the Atlas code, 
> and have found no issues with it. Additionally, with the update to the 
> berkley-elasticsearch profile, I have extensively tested that profile to make 
> sure that management of Elasticsearch functioned correctly.
> 
> 
> Thanks,
> 
> Pierre Padovani
> 
>



Review Request 66609: ATLAS-2564 - Reduce AtlasBaseClient.java logging verbosity to standard levels.

2018-04-13 Thread Pierre Padovani

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66609/
---

Review request for atlas.


Bugs: ATLAS-2564
https://issues.apache.org/jira/browse/ATLAS-2564


Repository: atlas


Description
---

This patch reduces the verbosity of AtlasBaseClient.java, but changing the info 
lines to debug, and adding a single line log of the request/response without 
logging the payloads.


Diffs
-

  client/common/src/main/java/org/apache/atlas/AtlasBaseClient.java 551b7b6f5 


Diff: https://reviews.apache.org/r/66609/diff/1/


Testing
---

Built and deployed within our test environment for 2 days to verfiy the 
improvement in logging.


Thanks,

Pierre Padovani



[jira] [Updated] (ATLAS-2564) AtlasBaseClient.java info logging too verbose

2018-04-13 Thread Pierre Padovani (JIRA)

 [ 
https://issues.apache.org/jira/browse/ATLAS-2564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Padovani updated ATLAS-2564:
---
Attachment: ATLAS-2564.patch

> AtlasBaseClient.java info logging too verbose
> -
>
> Key: ATLAS-2564
> URL: https://issues.apache.org/jira/browse/ATLAS-2564
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 1.0.0-alpha, 1.0.0
>    Reporter: Pierre Padovani
>    Assignee: Pierre Padovani
>Priority: Major
> Fix For: 1.0.0
>
> Attachments: ATLAS-2564.patch
>
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> The AtlasBaseClient.java logs the full payload at the info level. For large 
> entities, this can fill up log files very quickly. These log lines should be 
> moved to log level debug, and a new log line put in place that provides a 
> single line summary of the operation with out printing out the 
> request/response payloads. This also solves a potential issue with PII if 
> someone decides to put PII into an entity.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (ATLAS-2564) AtlasBaseClient.java info logging too verbose

2018-04-13 Thread Pierre Padovani (JIRA)
Pierre Padovani created ATLAS-2564:
--

 Summary: AtlasBaseClient.java info logging too verbose
 Key: ATLAS-2564
 URL: https://issues.apache.org/jira/browse/ATLAS-2564
 Project: Atlas
  Issue Type: Bug
Affects Versions: 1.0.0-alpha, 1.0.0
Reporter: Pierre Padovani
Assignee: Pierre Padovani
 Fix For: 1.0.0


The AtlasBaseClient.java logs the full payload at the info level. For large 
entities, this can fill up log files very quickly. These log lines should be 
moved to log level debug, and a new log line put in place that provides a 
single line summary of the operation with out printing out the request/response 
payloads. This also solves a potential issue with PII if someone decides to put 
PII into an entity.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Closed] (ATLAS-2470) Basic support for Cassandra

2018-04-13 Thread Pierre Padovani (JIRA)

 [ 
https://issues.apache.org/jira/browse/ATLAS-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Padovani closed ATLAS-2470.
--

> Basic support for Cassandra 
> 
>
> Key: ATLAS-2470
> URL: https://issues.apache.org/jira/browse/ATLAS-2470
> Project: Atlas
>  Issue Type: Sub-task
>Affects Versions: 1.0.0
>    Reporter: Pierre Padovani
>    Assignee: Pierre Padovani
>Priority: Major
> Fix For: 1.0.0
>
> Attachments: ATLAS-2470-1.patch, ATLAS-2470-2.patch, 
> ATLAS-2470-3.patch, ATLAS-2470-5.patch, ATLAS-2470.patch
>
>
> Add the basic build support, and ability to run embedded Cassandra and solr 
> using a profile build:
> -Pdist,embedded-cassandra-solr



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (ATLAS-2473) Add Audit support w/o HBase

2018-04-13 Thread Pierre Padovani (JIRA)

 [ 
https://issues.apache.org/jira/browse/ATLAS-2473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Padovani reassigned ATLAS-2473:
--

Assignee: Pierre Padovani

> Add Audit support w/o HBase
> ---
>
> Key: ATLAS-2473
> URL: https://issues.apache.org/jira/browse/ATLAS-2473
> Project: Atlas
>  Issue Type: Sub-task
>    Reporter: Pierre Padovani
>    Assignee: Pierre Padovani
>Priority: Major
> Fix For: 1.0.0
>
>
> Currently you can only have no audit, or audit on hbase. Need to add an audit 
> handler that can talk Cassandra or find another alternative.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Resolved] (ATLAS-2547) Fix CassandraAuditRepositoryTest.java to wait for Cassandra server start

2018-04-13 Thread Pierre Padovani (JIRA)

 [ 
https://issues.apache.org/jira/browse/ATLAS-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Padovani resolved ATLAS-2547.

   Resolution: Fixed
Fix Version/s: 1.0.0

> Fix CassandraAuditRepositoryTest.java to wait for Cassandra server start
> 
>
> Key: ATLAS-2547
> URL: https://issues.apache.org/jira/browse/ATLAS-2547
> Project: Atlas
>  Issue Type: Sub-task
>    Reporter: Pierre Padovani
>    Assignee: Pierre Padovani
>Priority: Major
> Fix For: 1.0.0
>
>
> Remove hard coded Thread.sleep and instead provide a basic looping mechanism 
> to test for an active Cassandra server. Loop should abort and throw an 
> exception if no connection can be achieved within a timeout/number of loops.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Closed] (ATLAS-2473) Add Audit support w/o HBase

2018-04-13 Thread Pierre Padovani (JIRA)

 [ 
https://issues.apache.org/jira/browse/ATLAS-2473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Padovani closed ATLAS-2473.
--

> Add Audit support w/o HBase
> ---
>
> Key: ATLAS-2473
> URL: https://issues.apache.org/jira/browse/ATLAS-2473
> Project: Atlas
>  Issue Type: Sub-task
>    Reporter: Pierre Padovani
>Priority: Major
> Fix For: 1.0.0
>
>
> Currently you can only have no audit, or audit on hbase. Need to add an audit 
> handler that can talk Cassandra or find another alternative.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Closed] (ATLAS-2547) Fix CassandraAuditRepositoryTest.java to wait for Cassandra server start

2018-04-13 Thread Pierre Padovani (JIRA)

 [ 
https://issues.apache.org/jira/browse/ATLAS-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Padovani closed ATLAS-2547.
--

> Fix CassandraAuditRepositoryTest.java to wait for Cassandra server start
> 
>
> Key: ATLAS-2547
> URL: https://issues.apache.org/jira/browse/ATLAS-2547
> Project: Atlas
>  Issue Type: Sub-task
>    Reporter: Pierre Padovani
>    Assignee: Pierre Padovani
>Priority: Major
> Fix For: 1.0.0
>
>
> Remove hard coded Thread.sleep and instead provide a basic looping mechanism 
> to test for an active Cassandra server. Loop should abort and throw an 
> exception if no connection can be achieved within a timeout/number of loops.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with JanusGraph 0.2.0

2018-04-12 Thread Pierre Padovani

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/
---

(Updated April 13, 2018, 2:55 a.m.)


Review request for atlas, David Radley, Madhan Neethiraj, and Sarath 
Subramanian.


Changes
---

This update is a rebase against master. It also resolves an inconsistent 
startup of elasticsearch.

This partially resolves some of the issues found in this review, but I believe 
further discussion is needed to resolve things correctly.

Issues:

  - Berkley db is excluded from the distribution, so there is this copy hack 
that must happen. This has always been the case and falls outside the scope of 
this patch
  - The way I configure the Constants is not ideal. I am in need of guidance on 
where/how to implement this fix as my understanding of the internals of Atlas 
is limited.
  - The connection to elasticsearch is now done by REST and not by native 
client. The jars do not contain any shaded 3rd party jars, and are needed for 
the REST client. If we build/distribute a full distribution that isn't based on 
a particular profile, these jars will be required if someone wants to use 
Elasticsearch. They should be harmless due to the lack of shaded 3rd party jars.
  - I need to add some documentation around using this profile, but wish to 
hold off until we have agreement on what and how to deal with the Elasticsearch 
jars.


To Run this profile:
mvn clean install -Pdist,berkeley-elasticsearch -DskipTests
cd distro/target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT
./bin/atlas_start.py
./bin/atlas_stop.py  <-- Atlas will NOT start correctly becuase the berkley db 
jar is missing, starting and stopping will create the lib dir where the berkley 
db jar is needed.
cp 
../../../../webapp/target/atlas-webapp-1.0.0-SNAPSHOT/WEB-INF/lib/je-7.3.7.jar 
server/webapp/atlas/WEB-INF/lib
./bin/atlas_start.py


Bugs: ATLAS-2478
https://issues.apache.org/jira/browse/ATLAS-2478


Repository: atlas


Description
---

This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates 
documentation.

Included with this patch is an update to the berkley-elasticsearch profile to 
automatically download and include elasticsearch as a side application much 
like solr is. Updates to the start/stop/conf scripts are included as well.

NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to 
/atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There 
are six constants that are incorrectly named with a '.' (dot). This is not 
supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the 
field names can be collectively thought of as an object. In the case of the 
fields defined in the Constants.java file, 'type' is defined as a string field, 
and 'type.name' is also defined as a string field. Elasticsearch sees this as 
an error, since it cannot convert type to an object. The fix included simply 
changes the field names from using a '.' (dot) to an '_' (underscore). This 
should NOT affect compatibility with hbase/solr for new installs. For existing 
installations, a reindex will be required as the field names will have changed.

**Query**: There is a way we can simplify integration/unit tests for the 
in-memory graph store by using a maven plugin that will download and run an 
elasticsearch node. This is nothing more than a maven change, and change to the 
atlas-application.properties to switch to elasticsearch from solr. I did not 
implement this, but am curious if this change would be desired. If so, this can 
be done with a separate ticket.


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/repository/Constants.java 01e49157d 
  distro/pom.xml 1f4c6d557 
  distro/src/bin/atlas_config.py 9062da649 
  distro/src/bin/atlas_start.py 61d69eb21 
  distro/src/bin/atlas_stop.py 94c3d6d46 
  distro/src/conf/atlas-env.sh 9213f488b 
  distro/src/main/assemblies/standalone-package.xml dc2a66ba9 
  docs/src/site/twiki/Configuration.twiki 63c3fce96 
  docs/src/site/twiki/HighAvailability.twiki 4270d0974 
  docs/src/site/twiki/InstallationSteps.twiki dca0618e3 
  graphdb/janus/pom.xml 016a09c33 
  pom.xml 4d2ab7391 
  webapp/pom.xml 284f538f7 


Diff: https://reviews.apache.org/r/66064/diff/2/

Changes: https://reviews.apache.org/r/66064/diff/1-2/


Testing
---

We currently use this fix with our Atlas setup in a fork of the Atlas code, and 
have found no issues with it. Additionally, with the update to the 
berkley-elasticsearch profile, I have extensively tested that profile to make 
sure that management of Elasticsearch functioned correctly.


Thanks,

Pierre Padovani



Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with JanusGraph 0.2.0

2018-04-12 Thread Pierre Padovani


> On April 13, 2018, 12:34 a.m., Ashutosh Mestry wrote:
> > Can you please add instructions on how to use this profile? I tried the 
> > patch today and I am still having trouble getting Atlas up.

Let me rebase the patch and make some changes, and retest. Master has moved a 
bit since I generated the original patch.  I'll update as soon as I have it 
done.


- Pierre


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review201069
---


On April 12, 2018, 5:35 p.m., Pierre Padovani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> ---
> 
> (Updated April 12, 2018, 5:35 p.m.)
> 
> 
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates 
> documentation.
> 
> Included with this patch is an update to the berkley-elasticsearch profile to 
> automatically download and include elasticsearch as a side application much 
> like solr is. Updates to the start/stop/conf scripts are included as well.
> 
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to 
> /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There 
> are six constants that are incorrectly named with a '.' (dot). This is not 
> supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** 
> the field names can be collectively thought of as an object. In the case of 
> the fields defined in the Constants.java file, 'type' is defined as a string 
> field, and 'type.name' is also defined as a string field. Elasticsearch sees 
> this as an error, since it cannot convert type to an object. The fix included 
> simply changes the field names from using a '.' (dot) to an '_' (underscore). 
> This should NOT affect compatibility with hbase/solr for new installs. For 
> existing installations, a reindex will be required as the field names will 
> have changed.
> 
> **Query**: There is a way we can simplify integration/unit tests for the 
> in-memory graph store by using a maven plugin that will download and run an 
> elasticsearch node. This is nothing more than a maven change, and change to 
> the atlas-application.properties to switch to elasticsearch from solr. I did 
> not implement this, but am curious if this change would be desired. If so, 
> this can be done with a separate ticket.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 605742dd 
>   distro/pom.xml 0103bef6 
>   distro/src/bin/atlas_config.py e6415cf4 
>   distro/src/bin/atlas_start.py 39be6b7c 
>   distro/src/bin/atlas_stop.py 66edd904 
>   distro/src/conf/atlas-env.sh 68b24e93 
>   distro/src/main/assemblies/standalone-package.xml 1881082e 
>   docs/src/site/twiki/Configuration.twiki 63c3fce9 
>   docs/src/site/twiki/HighAvailability.twiki 4270d097 
>   docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 
>   graphdb/janus/pom.xml 143b775f 
>   pom.xml 34e1ad04 
>   webapp/pom.xml 284f538f 
> 
> 
> Diff: https://reviews.apache.org/r/66064/diff/1/
> 
> 
> Testing
> ---
> 
> We currently use this fix with our Atlas setup in a fork of the Atlas code, 
> and have found no issues with it. Additionally, with the update to the 
> berkley-elasticsearch profile, I have extensively tested that profile to make 
> sure that management of Elasticsearch functioned correctly.
> 
> 
> Thanks,
> 
> Pierre Padovani
> 
>



Re: Review needed: ATLAS-2547

2018-04-12 Thread Pierre Padovani
I could swear I had uploaded the diff to the review board. I updated the
review with the diff, and added an additional sentence around hbase/solr.

As near as I can tell, the change doesn't affect new installations, only
existing that are upgrading. The only thing that will likely need to be
done will be a full reindex to handle the field name changes made to the
constants file.

Pierre

On Thu, Apr 12, 2018 at 12:07 PM, Madhan Neethiraj <mad...@apache.org>
wrote:

> Pierre,
>
> The review is in 'Discarded' state and no source file changes.
>
> I see a note on 'backwards incompatible' introduced in this patch. It is
> not clear what the impact will be on existing deployments that use
> HBase+Solr. Can you please add details?
>
> Thanks,
> Madhan
>
>
>
>
> On 4/12/18, 9:48 AM, "Pierre Padovani" <pierre.padovani@
> civitaslearning.com> wrote:
>
> Hey all,
>
> I submitted a review to fix the Elasticsearch support several weeks
> ago,
> and it has gotten no traction.
>
> https://reviews.apache.org/r/66064/
>
> The current Elasticsearch support is severely broken, and cannot
> function
> with any of the currently support Elasticsearch versions for
> JanusGraph.
> This patch resolves those issues.
>
> Can someone please take a look at this and get it reviewed? This really
> should make the 1.0.0 cut of Atlas.
>
> Thanks!
>
> Pierre Padovani
>
>
>
>


Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with JanusGraph 0.2.0

2018-04-12 Thread Pierre Padovani

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/
---

(Updated April 12, 2018, 5:35 p.m.)


Review request for atlas, David Radley, Madhan Neethiraj, and Sarath 
Subramanian.


Changes
---

This is the original patch. I swear I had uploaded it before.


Bugs: ATLAS-2478
https://issues.apache.org/jira/browse/ATLAS-2478


Repository: atlas


Description (updated)
---

This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates 
documentation.

Included with this patch is an update to the berkley-elasticsearch profile to 
automatically download and include elasticsearch as a side application much 
like solr is. Updates to the start/stop/conf scripts are included as well.

NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to 
/atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There 
are six constants that are incorrectly named with a '.' (dot). This is not 
supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the 
field names can be collectively thought of as an object. In the case of the 
fields defined in the Constants.java file, 'type' is defined as a string field, 
and 'type.name' is also defined as a string field. Elasticsearch sees this as 
an error, since it cannot convert type to an object. The fix included simply 
changes the field names from using a '.' (dot) to an '_' (underscore). This 
should NOT affect compatibility with hbase/solr for new installs. For existing 
installations, a reindex will be required as the field names will have changed.

**Query**: There is a way we can simplify integration/unit tests for the 
in-memory graph store by using a maven plugin that will download and run an 
elasticsearch node. This is nothing more than a maven change, and change to the 
atlas-application.properties to switch to elasticsearch from solr. I did not 
implement this, but am curious if this change would be desired. If so, this can 
be done with a separate ticket.


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/repository/Constants.java 605742dd 
  distro/pom.xml 0103bef6 
  distro/src/bin/atlas_config.py e6415cf4 
  distro/src/bin/atlas_start.py 39be6b7c 
  distro/src/bin/atlas_stop.py 66edd904 
  distro/src/conf/atlas-env.sh 68b24e93 
  distro/src/main/assemblies/standalone-package.xml 1881082e 
  docs/src/site/twiki/Configuration.twiki 63c3fce9 
  docs/src/site/twiki/HighAvailability.twiki 4270d097 
  docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 
  graphdb/janus/pom.xml 143b775f 
  pom.xml 34e1ad04 
  webapp/pom.xml 284f538f 


Diff: https://reviews.apache.org/r/66064/diff/1/


Testing
---

We currently use this fix with our Atlas setup in a fork of the Atlas code, and 
have found no issues with it. Additionally, with the update to the 
berkley-elasticsearch profile, I have extensively tested that profile to make 
sure that management of Elasticsearch functioned correctly.


Thanks,

Pierre Padovani



Review needed: ATLAS-2547

2018-04-12 Thread Pierre Padovani
Hey all,

I submitted a review to fix the Elasticsearch support several weeks ago,
and it has gotten no traction.

https://reviews.apache.org/r/66064/

The current Elasticsearch support is severely broken, and cannot function
with any of the currently support Elasticsearch versions for JanusGraph.
This patch resolves those issues.

Can someone please take a look at this and get it reviewed? This really
should make the 1.0.0 cut of Atlas.

Thanks!

Pierre Padovani


Re: Atlas Startup Failure with HBase Backend

2018-04-12 Thread Pierre Padovani
Looks good!

On Wed, Apr 11, 2018 at 6:11 PM, Ashutosh Mestry <ames...@hortonworks.com>
wrote:

> Hi Pierre
>
>
>
> Attached is the patch with the change you suggested below.
>
>
>
> I have started a pre-commit build. If it succeeds, I will commit it.
>
>
>
> Best regards,
>
>
>
> *~ ashutosh*
>
> *Ashutosh Mestry* <ames...@hortonworks.com>* . Staff Software Engineer .
> Hortonworks, Inc. .  +1-310-988 0670 <%2B1-310-988%200670>*
>
> ...
>
> *No hurry, no pause. – Tim Ferriss, Life Hacker, Author*
>
>
>
> *From: *Pierre Padovani <pie...@padovani.org>
> *Date: *Monday, April 9, 2018 at 12:48 PM
> *To: *Ashutosh Mestry <ames...@hortonworks.com>, "nixon.rodrigues@
> freestoneinfotech.com" <nixon.rodrig...@freestoneinfotech.com>
> *Cc: *"dev@atlas.apache.org" <dev@atlas.apache.org>
> *Subject: *Re: Atlas Startup Failure with HBase Backend
>
>
>
> If one of you wants to update the in-flight patch, here is the code that
> will retry connecting to Cassandra for up to 9 seconds.
>
>
>
> CassandraAuditRepositoryTest.java -
>
>
>
> Add these constants to the top of the file:
>
>
>
>   private static final String TEST_CLUSTER_NAME = "Test Cluster";
>
>   private static final int CLUSTER_PORT = 9042;
>
>   private static final String CLUSTER_HOST = "localhost";
>
>
>
>   private static final int MAX_RETRIES = 9;
>
>
>
>
>
> Replace the Thread.sleep with this code:
>
>
>
> // Retry the connection until we either connect or timeout
>
> Cluster.Builder cassandraClusterBuilder = Cluster.builder();
>
> Cluster cluster =
>
> cassandraClusterBuilder.addContactPoint(CLUSTER_HOST).
> withClusterName(TEST_CLUSTER_NAME).withPort(CLUSTER_PORT)
>
> .build();
>
> int retryCount = 0;
>
>
>
> while (retryCount < MAX_RETRIES) {
>
>   try {
>
> Session cassSession = cluster.connect();
>
> if (cassSession.getState().getConnectedHosts().size() > 0) {
>
>   cassSession.close();
>
>   return;
>
> }
>
>   } catch (Exception e) {
>
> Thread.sleep(1000);
>
>   }
>
>   retryCount++;
>
> }
>
> throw new RuntimeException("Unable to connect to embedded Cassandra
> after " + MAX_RETRIES + " seconds.");
>
>   }
>
>
>
> I can generate a patch with this as well... let me know what you want to
> do.
>
>
>
> Pierre
>
>
>
> On Mon, Apr 9, 2018 at 1:59 PM, Pierre Padovani <pie...@padovani.org>
> wrote:
>
> I believe this may be environmental... this test works locally for me:
>
>
>
> Running org.apache.atlas.repository.audit.InMemoryAuditRepositoryTest
>
> Running org.apache.atlas.repository.audit.CassandraAuditRepositoryTest
>
> Running org.apache.atlas.repository.userprofile.UserProfileServiceTest
>
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.346 sec
> - in org.apache.atlas.repository.impexp.TypeAttributeDifferenceTest
>
> Running org.apache.atlas.repository.migration.RelationshipMappingTest
>
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.689 sec
> - in org.apache.atlas.repository.audit.InMemoryAuditRepositoryTest
>
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.748 sec
> - in org.apache.atlas.repository.impexp.ImportTransformsTest
>
> Running org.apache.atlas.repository.migration.HiveParititionTest
>
> Running org.apache.atlas.repository.migration.HiveStocksTest
>
> Running org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1Test
>
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.732 sec
> - in org.apache.atlas.repository.audit.CassandraAuditRepositoryTest
>
>
>
> The class 'CassandraAuditRepositoryTest' has a hard coded sleep in place
> to allow the embedded Cassandra to startup. It is very likely that the box
> this is being run on is slow enough to cause the tests to fail. The
> quickest way to unblock this, would be to increase the sleep from 1 second
> to a greater value, say 5 seconds. I would recommend we do this to unblock
> the integration tests short term.
>
>
>
> Longer term, I'll change the test case to attempt to ping the Cassandra
> server during the setup method for a few times, and either exit with an
> error if it could not startup within a period of time, or after a certain
> number of tries. I created: https://issues.apache.org/jira/browse/ATLAS-
> 2547 to track this issue.
>
>
>
> Pie

Re: Atlas Startup Failure with HBase Backend

2018-04-09 Thread Pierre Padovani
I believe this may be environmental... this test works locally for me:

Running org.apache.atlas.repository.audit.InMemoryAuditRepositoryTest
Running org.apache.atlas.repository.audit.CassandraAuditRepositoryTest
Running org.apache.atlas.repository.userprofile.UserProfileServiceTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.346 sec -
in org.apache.atlas.repository.impexp.TypeAttributeDifferenceTest
Running org.apache.atlas.repository.migration.RelationshipMappingTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.689 sec -
in org.apache.atlas.repository.audit.InMemoryAuditRepositoryTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.748 sec -
in org.apache.atlas.repository.impexp.ImportTransformsTest
Running org.apache.atlas.repository.migration.HiveParititionTest
Running org.apache.atlas.repository.migration.HiveStocksTest
Running org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1Test
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.732 sec
- in org.apache.atlas.repository.audit.CassandraAuditRepositoryTest

The class 'CassandraAuditRepositoryTest' has a hard coded sleep in place to
allow the embedded Cassandra to startup. It is very likely that the box
this is being run on is slow enough to cause the tests to fail. The
quickest way to unblock this, would be to increase the sleep from 1 second
to a greater value, say 5 seconds. I would recommend we do this to unblock
the integration tests short term.

Longer term, I'll change the test case to attempt to ping the Cassandra
server during the setup method for a few times, and either exit with an
error if it could not startup within a period of time, or after a certain
number of tries. I created: https://issues.apache.org/jira/browse/ATLAS-2547
to track this issue.

Pierre



> On Mon, Apr 9, 2018 at 1:18 PM, Nixon Rodrigues <
> nixon.rodrig...@freestoneinfotech.com> wrote:
>
>> Ashutosh, Pierre,
>>
>> Can you review below unit testcase, its failing in precommit jenkins job.
>>
>> Tests run: 7, Failures: 1, Errors: 0, Skipped: 6, Time elapsed: 17.872 sec
>> <<< FAILURE! - in org.apache.atlas.repository.au
>> dit.CassandraAuditRepositoryTest
>> setup(org.apache.atlas.repository.audit.CassandraAuditRepositoryTest)
>> Time
>> elapsed: 17.709 sec  <<< FAILURE!
>> org.apache.atlas.AtlasException: com.datastax.driver.core.excep
>> tions.NoHostAvailableException: All host(s) tried for query failed
>> (tried:
>> localhost/127.0.0.1:9042(com.datastax.driver.core.
>> exceptions.TransportException:
>> [localhost/127.0.0.1:9042] Cannot connect))
>> at com.datastax.driver.core.ControlConnection.reconnectInternal
>> (ControlConnection.java:233)
>> at com.datastax.driver.core.ControlConnection.connect(ControlCo
>> nnection.java:79)
>> at com.datastax.driver.core.Cluster$Manager.init(Cluster.
>> java:1483)
>> at com.datastax.driver.core.Cluster.init(Cluster.java:159)
>> at com.datastax.driver.core.Cluster.connectAsync(Cluster.
>> java:330)
>> at com.datastax.driver.core.Cluster.connectAsync(Cluster.
>> java:305)
>> at com.datastax.driver.core.Cluster.connect(Cluster.java:247)
>> at org.apache.atlas.repository.audit.CassandraBasedAuditReposit
>> ory.createSession(CassandraBasedAuditRepository.java:217)
>> at org.apache.atlas.repository.audit.CassandraBasedAuditReposit
>> ory.startInternal(CassandraBasedAuditRepository.java:208)
>> at org.apache.atlas.repository.audit.CassandraBasedAuditReposit
>> ory.start(CassandraBasedAuditRepository.java:196)
>> at org.apache.atlas.repository.audit.CassandraAuditRepositoryTe
>> st.setup(CassandraAuditRepositoryTest.java:48)
>>
>> On Mon, Apr 9, 2018, 11:40 PM Pierre Padovani <pie...@padovani.org>
>> wrote:
>>
>> > Hi Ashtosh,
>> >
>> > Good catch! This looks good to me.
>> >
>> > Thanks!
>> >
>> > Pierre
>> >
>> > On Mon, Apr 9, 2018 at 12:48 PM, Ashutosh Mestry <
>> ames...@hortonworks.com>
>> > wrote:
>> >
>> > > Hi
>> > >
>> > >
>> > >
>> > > Thanks for adding Cassandra support to Atlas. With this update, Atlas
>> > > fails on startup when used with *HBase* backend.
>> > >
>> > >
>> > >
>> > > Attached is the patch that addresses the problem. I verified it in an
>> > > environment with *HBase* backend. I was not able to verify it with
>> > > Cassandra as backend. Can you please review and let me know if the
>> change
>> > > is OK? Other things look fine, I think.
>> > >
>> > >
>> > >
>> > > Best regards,
>> > >
>> > >
>> > >
>> > > *~ ashutosh*
>> > >
>> > > *Ashutosh Mestry* <ames...@hortonworks.com>* . Staff Software
>> Engineer .
>> > > Hortonworks, Inc. .  +1-310-988 0670 <%2B1-310-988%200670>*
>> > >
>> > > ...
>> > >
>> > > *No hurry, no pause. – Tim Ferriss, Life Hacker, Author*
>> > >
>> > >
>> > >
>> >
>>
>>


Re: Atlas Startup Failure with HBase Backend

2018-04-09 Thread Pierre Padovani
If one of you wants to update the in-flight patch, here is the code that
will retry connecting to Cassandra for up to 9 seconds.

CassandraAuditRepositoryTest.java -

Add these constants to the top of the file:

  private static final String TEST_CLUSTER_NAME = "Test Cluster";
  private static final int CLUSTER_PORT = 9042;
  private static final String CLUSTER_HOST = "localhost";

  private static final int MAX_RETRIES = 9;


Replace the Thread.sleep with this code:

// Retry the connection until we either connect or timeout
Cluster.Builder cassandraClusterBuilder = Cluster.builder();
Cluster cluster =

cassandraClusterBuilder.addContactPoint(CLUSTER_HOST).withClusterName(TEST_CLUSTER_NAME).withPort(CLUSTER_PORT)
.build();
int retryCount = 0;

while (retryCount < MAX_RETRIES) {
  try {
Session cassSession = cluster.connect();
if (cassSession.getState().getConnectedHosts().size() > 0) {
  cassSession.close();
  return;
}
  } catch (Exception e) {
Thread.sleep(1000);
  }
  retryCount++;
}
throw new RuntimeException("Unable to connect to embedded Cassandra
after " + MAX_RETRIES + " seconds.");
  }

I can generate a patch with this as well... let me know what you want to do.

Pierre

On Mon, Apr 9, 2018 at 1:59 PM, Pierre Padovani <pie...@padovani.org> wrote:

> I believe this may be environmental... this test works locally for me:
>
> Running org.apache.atlas.repository.audit.InMemoryAuditRepositoryTest
> Running org.apache.atlas.repository.audit.CassandraAuditRepositoryTest
> Running org.apache.atlas.repository.userprofile.UserProfileServiceTest
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.346 sec
> - in org.apache.atlas.repository.impexp.TypeAttributeDifferenceTest
> Running org.apache.atlas.repository.migration.RelationshipMappingTest
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.689 sec
> - in org.apache.atlas.repository.audit.InMemoryAuditRepositoryTest
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.748 sec
> - in org.apache.atlas.repository.impexp.ImportTransformsTest
> Running org.apache.atlas.repository.migration.HiveParititionTest
> Running org.apache.atlas.repository.migration.HiveStocksTest
> Running org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1Test
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.732 sec
> - in org.apache.atlas.repository.audit.CassandraAuditRepositoryTest
>
> The class 'CassandraAuditRepositoryTest' has a hard coded sleep in place
> to allow the embedded Cassandra to startup. It is very likely that the box
> this is being run on is slow enough to cause the tests to fail. The
> quickest way to unblock this, would be to increase the sleep from 1 second
> to a greater value, say 5 seconds. I would recommend we do this to unblock
> the integration tests short term.
>
> Longer term, I'll change the test case to attempt to ping the Cassandra
> server during the setup method for a few times, and either exit with an
> error if it could not startup within a period of time, or after a certain
> number of tries. I created: https://issues.apache.org/jira/browse/ATLAS-
> 2547 to track this issue.
>
> Pierre
>
>
>
>> On Mon, Apr 9, 2018 at 1:18 PM, Nixon Rodrigues <nixon.rodrigues@
>> freestoneinfotech.com> wrote:
>>
>>> Ashutosh, Pierre,
>>>
>>> Can you review below unit testcase, its failing in precommit jenkins job.
>>>
>>> Tests run: 7, Failures: 1, Errors: 0, Skipped: 6, Time elapsed: 17.872
>>> sec
>>> <<< FAILURE! - in org.apache.atlas.repository.au
>>> dit.CassandraAuditRepositoryTest
>>> setup(org.apache.atlas.repository.audit.CassandraAuditRepositoryTest)
>>> Time
>>> elapsed: 17.709 sec  <<< FAILURE!
>>> org.apache.atlas.AtlasException: com.datastax.driver.core.excep
>>> tions.NoHostAvailableException: All host(s) tried for query failed
>>> (tried:
>>> localhost/127.0.0.1:9042(com.datastax.driver.core.exceptions
>>> .TransportException:
>>> [localhost/127.0.0.1:9042] Cannot connect))
>>> at com.datastax.driver.core.ControlConnection.reconnectInternal
>>> (ControlConnection.java:233)
>>> at com.datastax.driver.core.ControlConnection.connect(ControlCo
>>> nnection.java:79)
>>> at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:
>>> 1483)
>>> at com.datastax.driver.core.Cluster.init(Cluster.java:159)
>>> at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:
>>> 330)

[jira] [Created] (ATLAS-2547) Fix CassandraAuditRepositoryTest.java to wait for Cassandra server start

2018-04-09 Thread Pierre Padovani (JIRA)
Pierre Padovani created ATLAS-2547:
--

 Summary: Fix CassandraAuditRepositoryTest.java to wait for 
Cassandra server start
 Key: ATLAS-2547
 URL: https://issues.apache.org/jira/browse/ATLAS-2547
 Project: Atlas
  Issue Type: Sub-task
Reporter: Pierre Padovani
Assignee: Pierre Padovani


Remove hard coded Thread.sleep and instead provide a basic looping mechanism to 
test for an active Cassandra server. Loop should abort and throw an exception 
if no connection can be achieved within a timeout/number of loops.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Resolved] (ATLAS-2473) Add Audit support w/o HBase

2018-04-09 Thread Pierre Padovani (JIRA)

 [ 
https://issues.apache.org/jira/browse/ATLAS-2473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Padovani resolved ATLAS-2473.

   Resolution: Fixed
Fix Version/s: 1.0.0

> Add Audit support w/o HBase
> ---
>
> Key: ATLAS-2473
> URL: https://issues.apache.org/jira/browse/ATLAS-2473
> Project: Atlas
>  Issue Type: Sub-task
>    Reporter: Pierre Padovani
>Priority: Major
> Fix For: 1.0.0
>
>
> Currently you can only have no audit, or audit on hbase. Need to add an audit 
> handler that can talk Cassandra or find another alternative.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas

2018-03-28 Thread Pierre Padovani
at 
> > org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)
> > [ERROR] at 
> > org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)
> > [ERROR] at 
> > org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:384)
> > [ERROR] at 
> > org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$4(MiniSolrCloudCluster.java:247)
> > [ERROR] at 
> > java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > [ERROR] at 
> > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$128(ExecutorUtil.java:188)
> > [ERROR] at 
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > [ERROR] at 
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > [ERROR] at java.lang.Thread.run(Thread.java:748)
> > [ERROR] Caused by: java.lang.ClassNotFoundException: 
> > com.codahale.metrics.MetricRegistry$MetricSupplier
> > [ERROR] at 
> > java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> > [ERROR] at 
> > java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > [ERROR] at 
> > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> > [ERROR] at 
> > java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > [ERROR] ... 23 more
> > [ERROR] -> [Help 1]
> > [ERROR] 
> > [ERROR] To see the full stack trace of the errors, re-run Maven with 
> > the -e switch.
> > [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> > [ERROR] 
> > [ERROR] For more information about the errors and possible solutions, 
> > please read the following articles:
> > [ERROR] [Help 1] 
> > http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> > [ERROR] 
> > [ERROR] After correcting the problems, you can resume the build with 
> > the command
> > [ERROR]   mvn  -rf :atlas-repository
> 
> Pierre Padovani wrote:
> I just attempted a build on master using 'mvn clean package -Pdist' and 
> got the same failure. I do not think this failure is due to my patch. I'm 
> running on OS X, what are you using?
> 
> Pierre Padovani wrote:
> Ok... got this one figured out. Looks like the storm-bridge/shim is 
> pulling in an old version of the 'io.dropwizard.metrics:metrics-core'. The 
> order of classpath loading may differ on a system based on CPU speed etc. So 
> I'm guessing that this is why you didn't see it and I did. This same 
> dependency is introduced with the janus-cassandra inclusion, so I'm going to 
> fix this across the board to make sure we are only using/including one 
> version of these metric jars.
> 
> Pierre Padovani wrote:
> Hi David,
> 
> I'm getting some weirdly inconsistent results with the unit tests. 
> Sometimes they pass others,  the webapp fails to start missing the 
> org.apache.solr.cloud.MiniSolrCloudCluster. So there are still some classpath 
> issues at play. I'm going to figure this out, and I'll ping when I have the 
> tests consistently passing.

David,

I'd also like to point you at the current documentation which indicates that 
tests should be skipped...

https://atlas.apache.org/InstallationSteps.html

I would assume the docs are out of date? 

;-)

Pierre


- Pierre


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65885/#review199964
---


On March 28, 2018, 3:28 p.m., Pierre Padovani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65885/
> ---
> 
> (Updated March 28, 2018, 3:28 p.m.)
> 
> 
> Review request for atlas and David Radley.
> 
> 
> Bugs: ATLAS-2470
> https://issues.apache.org/jira/browse/ATLAS-2470
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This updates pom's to add the needed cassandra jars, and adds a dist profile 
> embedded-cassandra-solr.
> 
> 
> Diffs
> -
> 
>   addons/storm-bridge-shim/pom.xml c25fbfe5 
>   addons/storm-bridge/pom.xml 2627d92e 
>   distro/pom.xml 0103bef6 
>   distro/src/bin/at

Re: Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas

2018-03-28 Thread Pierre Padovani

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65885/
---

(Updated March 28, 2018, 3:28 p.m.)


Review request for atlas and David Radley.


Changes
---

This update resolves build issues when using the -Pdist flag. In master, the 
webapp pom.xml leverages the profiles as part of building the war. For the 
-Pdist profile, it will exclude three jars that are required for the 
integration tests to pass. Berkleyje, solr-test-framework, and jts. I've 
updated the pom to copy the three needed dependencies into the 
target/test-classes directory, and append the three jars to the webapp 
classpath. This allows the build to pass regardless of the profile used.


Bugs: ATLAS-2470
https://issues.apache.org/jira/browse/ATLAS-2470


Repository: atlas


Description
---

This updates pom's to add the needed cassandra jars, and adds a dist profile 
embedded-cassandra-solr.


Diffs (updated)
-

  addons/storm-bridge-shim/pom.xml c25fbfe5 
  addons/storm-bridge/pom.xml 2627d92e 
  distro/pom.xml 0103bef6 
  distro/src/bin/atlas_config.py e6415cf4 
  distro/src/bin/atlas_start.py 39be6b7c 
  distro/src/bin/atlas_stop.py 66edd904 
  distro/src/conf/atlas-application.properties 10fb7b86 
  distro/src/conf/atlas-env.sh 68b24e93 
  distro/src/conf/cassandra.yml.template PRE-CREATION 
  distro/src/conf/zookeeper/log4j.properties PRE-CREATION 
  distro/src/conf/zookeeper/zoo.cfg.template PRE-CREATION 
  distro/src/main/assemblies/standalone-package.xml 1881082e 
  docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 
  graphdb/janus/pom.xml 143b775f 
  pom.xml 9ddc8e18 
  repository/pom.xml b1d6b1f9 
  
repository/src/main/java/org/apache/atlas/repository/audit/AbstractStorageBasedAuditRepository.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
 cee7374e 
  
repository/src/test/java/org/apache/atlas/repository/audit/AuditRepositoryTestBase.java
 4d70b7f7 
  
repository/src/test/java/org/apache/atlas/repository/audit/CassandraAuditRepositoryTest.java
 PRE-CREATION 
  repository/src/test/resources/cassandra_test.yml PRE-CREATION 
  
server-api/src/main/java/org/apache/atlas/listener/ActiveStateChangeHandler.java
 7d282b30 
  test-tools/src/main/resources/solr/solr.xml 041f0058 
  webapp/pom.xml 284f538f 


Diff: https://reviews.apache.org/r/65885/diff/6/

Changes: https://reviews.apache.org/r/65885/diff/5-6/


Testing
---

Full build with the new embedded-cassandra-solr, and testing to make sure Atlas 
comes up and is functional.

Be aware that we have been running Cassandra backed JanusGraph for months with 
no issues.

This patch now includes support for Cassandra as an audit store. I've also 
refactored the audit unit tests, and added cassandra specific tests.


Thanks,

Pierre Padovani



maven dependency issues

2018-03-27 Thread Pierre Padovani
As part of working through my patch with David, I've run across
inconsistent versioning of transitive dependencies. For example:

httpcore:4.4.1
httpcore:4.4.6

lucene-core:4.8.1
lucene-core:4.10.0
lucene-core:7.0.0

These version variations may not always cause issues, but class load
ordering across incompatible versions can cause significant
unpredictability.

Is there a reason we have not cleaned up and enforced a single version of
each dependency?

Thanks!

Pierre Padovani


Re: Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas

2018-03-27 Thread Pierre Padovani
at 
> > org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)
> > [ERROR] at 
> > org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)
> > [ERROR] at 
> > org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:384)
> > [ERROR] at 
> > org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$4(MiniSolrCloudCluster.java:247)
> > [ERROR] at 
> > java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > [ERROR] at 
> > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$128(ExecutorUtil.java:188)
> > [ERROR] at 
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > [ERROR] at 
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > [ERROR] at java.lang.Thread.run(Thread.java:748)
> > [ERROR] Caused by: java.lang.ClassNotFoundException: 
> > com.codahale.metrics.MetricRegistry$MetricSupplier
> > [ERROR] at 
> > java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> > [ERROR] at 
> > java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > [ERROR] at 
> > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> > [ERROR] at 
> > java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > [ERROR] ... 23 more
> > [ERROR] -> [Help 1]
> > [ERROR] 
> > [ERROR] To see the full stack trace of the errors, re-run Maven with 
> > the -e switch.
> > [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> > [ERROR] 
> > [ERROR] For more information about the errors and possible solutions, 
> > please read the following articles:
> > [ERROR] [Help 1] 
> > http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> > [ERROR] 
> > [ERROR] After correcting the problems, you can resume the build with 
> > the command
> > [ERROR]   mvn  -rf :atlas-repository
> 
> Pierre Padovani wrote:
> I just attempted a build on master using 'mvn clean package -Pdist' and 
> got the same failure. I do not think this failure is due to my patch. I'm 
> running on OS X, what are you using?
> 
> Pierre Padovani wrote:
> Ok... got this one figured out. Looks like the storm-bridge/shim is 
> pulling in an old version of the 'io.dropwizard.metrics:metrics-core'. The 
> order of classpath loading may differ on a system based on CPU speed etc. So 
> I'm guessing that this is why you didn't see it and I did. This same 
> dependency is introduced with the janus-cassandra inclusion, so I'm going to 
> fix this across the board to make sure we are only using/including one 
> version of these metric jars.

Hi David,

I'm getting some weirdly inconsistent results with the unit tests. Sometimes 
they pass others,  the webapp fails to start missing the 
org.apache.solr.cloud.MiniSolrCloudCluster. So there are still some classpath 
issues at play. I'm going to figure this out, and I'll ping when I have the 
tests consistently passing.


- Pierre


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65885/#review199964
---


On March 26, 2018, 8:07 p.m., Pierre Padovani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65885/
> ---
> 
> (Updated March 26, 2018, 8:07 p.m.)
> 
> 
> Review request for atlas and David Radley.
> 
> 
> Bugs: ATLAS-2470
> https://issues.apache.org/jira/browse/ATLAS-2470
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This updates pom's to add the needed cassandra jars, and adds a dist profile 
> embedded-cassandra-solr.
> 
> 
> Diffs
> -
> 
>   addons/storm-bridge-shim/pom.xml c25fbfe5 
>   addons/storm-bridge/pom.xml 2627d92e 
>   distro/pom.xml 0103bef6 
>   distro/src/bin/atlas_config.py e6415cf4 
>   distro/src/bin/atlas_start.py 39be6b7c 
>   distro/src/bin/atlas_stop.py 66edd904 
>   distro/src/conf/atlas-application.properties 10fb7b86 
>   distro/src/conf/atlas-env.sh 68b24e93 
>   distro/src/conf/cassandra.yml.template PRE-CREATION 
>   dis

[jira] [Updated] (ATLAS-2470) Basic support for Cassandra

2018-03-26 Thread Pierre Padovani (JIRA)

 [ 
https://issues.apache.org/jira/browse/ATLAS-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Padovani updated ATLAS-2470:
---
Attachment: ATLAS-2470-5.patch

> Basic support for Cassandra 
> 
>
> Key: ATLAS-2470
> URL: https://issues.apache.org/jira/browse/ATLAS-2470
> Project: Atlas
>  Issue Type: Sub-task
>Affects Versions: 1.0.0
>    Reporter: Pierre Padovani
>    Assignee: Pierre Padovani
>Priority: Major
> Fix For: 1.0.0
>
> Attachments: ATLAS-2470-1.patch, ATLAS-2470-2.patch, 
> ATLAS-2470-3.patch, ATLAS-2470-5.patch, ATLAS-2470.patch
>
>
> Add the basic build support, and ability to run embedded Cassandra and solr 
> using a profile build:
> -Pdist,embedded-cassandra-solr



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas

2018-03-26 Thread Pierre Padovani

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65885/
---

(Updated March 26, 2018, 8:07 p.m.)


Review request for atlas and David Radley.


Changes
---

Resolved all indicated issues from review. Of special note, changes were made 
to the storm bridge/shim pom files to force the dropwizard metric jars to match 
the version used by other pieces in Atlas. This avoids random test failures due 
to classpath loading ordering.


Bugs: ATLAS-2470
https://issues.apache.org/jira/browse/ATLAS-2470


Repository: atlas


Description
---

This updates pom's to add the needed cassandra jars, and adds a dist profile 
embedded-cassandra-solr.


Diffs (updated)
-

  addons/storm-bridge-shim/pom.xml c25fbfe5 
  addons/storm-bridge/pom.xml 2627d92e 
  distro/pom.xml 0103bef6 
  distro/src/bin/atlas_config.py e6415cf4 
  distro/src/bin/atlas_start.py 39be6b7c 
  distro/src/bin/atlas_stop.py 66edd904 
  distro/src/conf/atlas-application.properties 10fb7b86 
  distro/src/conf/atlas-env.sh 68b24e93 
  distro/src/conf/cassandra.yml.template PRE-CREATION 
  distro/src/conf/zookeeper/log4j.properties PRE-CREATION 
  distro/src/conf/zookeeper/zoo.cfg.template PRE-CREATION 
  distro/src/main/assemblies/standalone-package.xml 1881082e 
  docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 
  graphdb/janus/pom.xml 143b775f 
  pom.xml 9ddc8e18 
  repository/pom.xml b1d6b1f9 
  
repository/src/main/java/org/apache/atlas/repository/audit/AbstractStorageBasedAuditRepository.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
 cee7374e 
  
repository/src/test/java/org/apache/atlas/repository/audit/AuditRepositoryTestBase.java
 4d70b7f7 
  
repository/src/test/java/org/apache/atlas/repository/audit/CassandraAuditRepositoryTest.java
 PRE-CREATION 
  repository/src/test/resources/cassandra_test.yml PRE-CREATION 
  
server-api/src/main/java/org/apache/atlas/listener/ActiveStateChangeHandler.java
 7d282b30 
  test-tools/src/main/resources/solr/solr.xml 041f0058 


Diff: https://reviews.apache.org/r/65885/diff/5/

Changes: https://reviews.apache.org/r/65885/diff/4-5/


Testing
---

Full build with the new embedded-cassandra-solr, and testing to make sure Atlas 
comes up and is functional.

Be aware that we have been running Cassandra backed JanusGraph for months with 
no issues.

This patch now includes support for Cassandra as an audit store. I've also 
refactored the audit unit tests, and added cassandra specific tests.


Thanks,

Pierre Padovani



Re: Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas

2018-03-26 Thread Pierre Padovani
at 
> > org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)
> > [ERROR] at 
> > org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)
> > [ERROR] at 
> > org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:384)
> > [ERROR] at 
> > org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$4(MiniSolrCloudCluster.java:247)
> > [ERROR] at 
> > java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > [ERROR] at 
> > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$128(ExecutorUtil.java:188)
> > [ERROR] at 
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > [ERROR] at 
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > [ERROR] at java.lang.Thread.run(Thread.java:748)
> > [ERROR] Caused by: java.lang.ClassNotFoundException: 
> > com.codahale.metrics.MetricRegistry$MetricSupplier
> > [ERROR] at 
> > java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> > [ERROR] at 
> > java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > [ERROR] at 
> > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> > [ERROR] at 
> > java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > [ERROR] ... 23 more
> > [ERROR] -> [Help 1]
> > [ERROR] 
> > [ERROR] To see the full stack trace of the errors, re-run Maven with 
> > the -e switch.
> > [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> > [ERROR] 
> > [ERROR] For more information about the errors and possible solutions, 
> > please read the following articles:
> > [ERROR] [Help 1] 
> > http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> > [ERROR] 
> > [ERROR] After correcting the problems, you can resume the build with 
> > the command
> > [ERROR]   mvn  -rf :atlas-repository
> 
> Pierre Padovani wrote:
> I just attempted a build on master using 'mvn clean package -Pdist' and 
> got the same failure. I do not think this failure is due to my patch. I'm 
> running on OS X, what are you using?

Ok... got this one figured out. Looks like the storm-bridge/shim is pulling in 
an old version of the 'io.dropwizard.metrics:metrics-core'. The order of 
classpath loading may differ on a system based on CPU speed etc. So I'm 
guessing that this is why you didn't see it and I did. This same dependency is 
introduced with the janus-cassandra inclusion, so I'm going to fix this across 
the board to make sure we are only using/including one version of these metric 
jars.


- Pierre


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65885/#review199964
---


On March 20, 2018, 12:04 a.m., Pierre Padovani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65885/
> ---
> 
> (Updated March 20, 2018, 12:04 a.m.)
> 
> 
> Review request for atlas and David Radley.
> 
> 
> Bugs: ATLAS-2470
> https://issues.apache.org/jira/browse/ATLAS-2470
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This updates pom's to add the needed cassandra jars, and adds a dist profile 
> embedded-cassandra-solr.
> 
> 
> Diffs
> -
> 
>   distro/pom.xml 0103bef6 
>   distro/src/bin/atlas_config.py e6415cf4 
>   distro/src/bin/atlas_start.py 39be6b7c 
>   distro/src/bin/atlas_stop.py 66edd904 
>   distro/src/conf/atlas-application.properties 10fb7b86 
>   distro/src/conf/atlas-env.sh 68b24e93 
>   distro/src/conf/cassandra.yml.template PRE-CREATION 
>   distro/src/conf/zookeeper/log4j.properties PRE-CREATION 
>   distro/src/conf/zookeeper/zoo.cfg.template PRE-CREATION 
>   distro/src/main/assemblies/standalone-package.xml 1881082e 
>   docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 
>   graphdb/janus/pom.xml 143b775f 
>   pom.xml 9ddc8e18 
>   repository/pom.xml b1d6b1f9 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/AbstractStorageBasedAuditRepository.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
>  cee7374e 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/AuditRepositoryTestBase.java
>  4d70b7f7 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/CassandraAuditRepositoryTest.java
>  PRE-CREATION 
>   repository/src/test/resources/cassandra_test.yml PRE-CREATION 
>   
> server-api/src/main/java/org/apache/atlas/listener/ActiveStateChangeHandler.java
>  7d282b30 
> 
> 
> Diff: https://reviews.apache.org/r/65885/diff/4/
> 
> 
> Testing
> ---
> 
> Full build with the new embedded-cassandra-solr, and testing to make sure 
> Atlas comes up and is functional.
> 
> Be aware that we have been running Cassandra backed JanusGraph for months 
> with no issues.
> 
> This patch now includes support for Cassandra as an audit store. I've also 
> refactored the audit unit tests, and added cassandra specific tests.
> 
> 
> Thanks,
> 
> Pierre Padovani
> 
>



Re: Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas

2018-03-26 Thread Pierre Padovani
WithMapping(ServletHandler.java:1130)
> > [ERROR] at 
> > org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)
> > [ERROR] at 
> > org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)
> > [ERROR] at 
> > org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)
> > [ERROR] at 
> > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
> > [ERROR] at 
> > org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)
> > [ERROR] at 
> > org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)
> > [ERROR] at 
> > org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:384)
> > [ERROR] at 
> > org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$4(MiniSolrCloudCluster.java:247)
> > [ERROR] at 
> > java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > [ERROR] at 
> > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$128(ExecutorUtil.java:188)
> > [ERROR] at 
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > [ERROR] at 
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > [ERROR] at java.lang.Thread.run(Thread.java:748)
> > [ERROR] Caused by: java.lang.ClassNotFoundException: 
> > com.codahale.metrics.MetricRegistry$MetricSupplier
> > [ERROR] at 
> > java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> > [ERROR] at 
> > java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > [ERROR] at 
> > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> > [ERROR] at 
> > java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > [ERROR] ... 23 more
> > [ERROR] -> [Help 1]
> > [ERROR] 
> > [ERROR] To see the full stack trace of the errors, re-run Maven with 
> > the -e switch.
> > [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> > [ERROR] 
> > [ERROR] For more information about the errors and possible solutions, 
> > please read the following articles:
> > [ERROR] [Help 1] 
> > http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> > [ERROR] 
> > [ERROR] After correcting the problems, you can resume the build with 
> > the command
> > [ERROR]   mvn  -rf :atlas-repository

I just attempted a build on master using 'mvn clean package -Pdist' and got the 
same failure. I do not think this failure is due to my patch. I'm running on OS 
X, what are you using?


- Pierre


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65885/#review199964
---


On March 20, 2018, 12:04 a.m., Pierre Padovani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65885/
> ---
> 
> (Updated March 20, 2018, 12:04 a.m.)
> 
> 
> Review request for atlas and David Radley.
> 
> 
> Bugs: ATLAS-2470
> https://issues.apache.org/jira/browse/ATLAS-2470
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This updates pom's to add the needed cassandra jars, and adds a dist profile 
> embedded-cassandra-solr.
> 
> 
> Diffs
> -
> 
>   distro/pom.xml 0103bef6 
>   distro/src/bin/atlas_config.py e6415cf4 
>   distro/src/bin/atlas_start.py 39be6b7c 
>   distro/src/bin/atlas_stop.py 66edd904 
>   distro/src/conf/atlas-application.properties 10fb7b86 
>   distro/src/conf/atlas-env.sh 68b24e93 
>   distro/src/conf/cassandra.yml.template PRE-CREATION 
>   distro/src/conf/zookeeper/log4j.properties PRE-CREATION 
>   distro/src/conf/zookeeper/zoo.cfg.template PRE-CREATION 
>   distro/src/main/assemblies/standalone-package.xml 1881082e 
>   docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 
>   graphdb/janus/pom.xml 143b775f 
>   pom.xml 9ddc8e18 
>   repository/pom.xml b1d6b1f9 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/AbstractStorageBasedAuditRepository.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
>  cee7374e 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/AuditRepositoryTestBase.java
>  4d70b7f7 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/CassandraAuditRepositoryTest.java
>  PRE-CREATION 
>   repository/src/test/resources/cassandra_test.yml PRE-CREATION 
>   
> server-api/src/main/java/org/apache/atlas/listener/ActiveStateChangeHandler.java
>  7d282b30 
> 
> 
> Diff: https://reviews.apache.org/r/65885/diff/4/
> 
> 
> Testing
> ---
> 
> Full build with the new embedded-cassandra-solr, and testing to make sure 
> Atlas comes up and is functional.
> 
> Be aware that we have been running Cassandra backed JanusGraph for months 
> with no issues.
> 
> This patch now includes support for Cassandra as an audit store. I've also 
> refactored the audit unit tests, and added cassandra specific tests.
> 
> 
> Thanks,
> 
> Pierre Padovani
> 
>



[jira] [Commented] (ATLAS-2471) Create Dockerfile support for Cassandra

2018-03-20 Thread Pierre Padovani (JIRA)

[ 
https://issues.apache.org/jira/browse/ATLAS-2471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406483#comment-16406483
 ] 

Pierre Padovani commented on ATLAS-2471:


Before a Dockerfile can be incorporated, the Cassandra code must be merged to 
master.

> Create Dockerfile support for Cassandra
> ---
>
> Key: ATLAS-2471
> URL: https://issues.apache.org/jira/browse/ATLAS-2471
> Project: Atlas
>  Issue Type: Sub-task
>    Reporter: Pierre Padovani
>    Assignee: Pierre Padovani
>Priority: Major
>
> Add an additional Dockerfile that builds a self contained container that runs 
> on Cassandra 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Resolved] (ATLAS-2472) Update Atlas documentation for Cassandra support

2018-03-20 Thread Pierre Padovani (JIRA)

 [ 
https://issues.apache.org/jira/browse/ATLAS-2472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Padovani resolved ATLAS-2472.

Resolution: Fixed

Added the full documentation as part of ATLAS-2470

> Update Atlas documentation for Cassandra support
> 
>
> Key: ATLAS-2472
> URL: https://issues.apache.org/jira/browse/ATLAS-2472
> Project: Atlas
>  Issue Type: Sub-task
>    Reporter: Pierre Padovani
>    Assignee: Pierre Padovani
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Closed] (ATLAS-2472) Update Atlas documentation for Cassandra support

2018-03-20 Thread Pierre Padovani (JIRA)

 [ 
https://issues.apache.org/jira/browse/ATLAS-2472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Padovani closed ATLAS-2472.
--

> Update Atlas documentation for Cassandra support
> 
>
> Key: ATLAS-2472
> URL: https://issues.apache.org/jira/browse/ATLAS-2472
> Project: Atlas
>  Issue Type: Sub-task
>    Reporter: Pierre Padovani
>    Assignee: Pierre Padovani
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas

2018-03-19 Thread Pierre Padovani

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65885/
---

(Updated March 20, 2018, 12:04 a.m.)


Review request for atlas and David Radley.


Changes
---

Fixed the injection of the logback jars in the distribution, they no longer are 
being pulled in, and logging has returned to normal.

I attempted to do a full build with tests, but the tests in repository failed. 
I do not believe the failure is due to this patch, as the same exact failure 
happens in master. The cassandra based audit tests do pass even if the 
subsequent tests are failing.

As an additional test on this patch, I applied the patch to a clean master with:

git am < ATLAS=2470-4.patch

It did give some warnings on whitespace, but properly applied the patch. I then 
built and ran Atlas with embedded-cassandra-solr and it started up with no 
errors.


Bugs: ATLAS-2470
https://issues.apache.org/jira/browse/ATLAS-2470


Repository: atlas


Description
---

This updates pom's to add the needed cassandra jars, and adds a dist profile 
embedded-cassandra-solr.


Diffs (updated)
-

  distro/pom.xml 0103bef6 
  distro/src/bin/atlas_config.py e6415cf4 
  distro/src/bin/atlas_start.py 39be6b7c 
  distro/src/bin/atlas_stop.py 66edd904 
  distro/src/conf/atlas-application.properties 10fb7b86 
  distro/src/conf/atlas-env.sh 68b24e93 
  distro/src/conf/cassandra.yml.template PRE-CREATION 
  distro/src/conf/zookeeper/log4j.properties PRE-CREATION 
  distro/src/conf/zookeeper/zoo.cfg.template PRE-CREATION 
  distro/src/main/assemblies/standalone-package.xml 1881082e 
  docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 
  graphdb/janus/pom.xml 143b775f 
  pom.xml 9ddc8e18 
  repository/pom.xml b1d6b1f9 
  
repository/src/main/java/org/apache/atlas/repository/audit/AbstractStorageBasedAuditRepository.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
 cee7374e 
  
repository/src/test/java/org/apache/atlas/repository/audit/AuditRepositoryTestBase.java
 4d70b7f7 
  
repository/src/test/java/org/apache/atlas/repository/audit/CassandraAuditRepositoryTest.java
 PRE-CREATION 
  repository/src/test/resources/cassandra_test.yml PRE-CREATION 
  
server-api/src/main/java/org/apache/atlas/listener/ActiveStateChangeHandler.java
 7d282b30 


Diff: https://reviews.apache.org/r/65885/diff/4/

Changes: https://reviews.apache.org/r/65885/diff/3-4/


Testing
---

Full build with the new embedded-cassandra-solr, and testing to make sure Atlas 
comes up and is functional.

Be aware that we have been running Cassandra backed JanusGraph for months with 
no issues.

This patch now includes support for Cassandra as an audit store. I've also 
refactored the audit unit tests, and added cassandra specific tests.


Thanks,

Pierre Padovani



Re: Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas

2018-03-19 Thread Pierre Padovani


> On March 19, 2018, 7:28 p.m., David Radley wrote:
> > docs/src/site/twiki/InstallationSteps.twiki
> > Lines 45 (patched)
> > <https://reviews.apache.org/r/65885/diff/3/?file=1982164#file1982164line45>
> >
> > I built Atlas as per these instructions. I then started atlas. Atlas 
> > started and worked - but when I looked at the application.log there was 
> > only 3 log entries.  
> > 
> > 2018-03-19 12:22:22,447 WARN  - [main:] ~ No meta.properties file under 
> > dir 
> > /Users/david/atlasreview2/distro/target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/data/kafka/kafka/meta.properties
> >  (Logging$class:87)
> > 2018-03-19 12:22:22,922 WARN  - [main:] ~ No meta.properties file under 
> > dir 
> > /Users/david/atlasreview2/distro/target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/data/kafka/kafka/meta.properties
> >  (Logging$class:87)
> > 2018-03-19 12:22:22,986 INFO  - [NotificationHookConsumer thread-0:] ~ 
> > [atlas-hook-consumer-thread]: Starting (Logging$class:72)
> > 
> > 
> > I would have expected much more logging.
> 
> Pierre Padovani wrote:
> This confused me as well. All of the logging is in the the newest 
> atlas..out file. Which makes sense when log4j is configured to 
> log to a rolling date/time stamped file. In previous builds of Atlas the 
> logging all showed up in the application.log file. This change in behavior 
> happened to me within the last few weeks. Do we know if a change was made to 
> induce this? or Was this induced by my changes?
> 
> I'll do some testing on my side.
> 
> Pierre Padovani wrote:
> Just retested off of master... something in the cassandra jars for Janus 
> must be overriding the Atlas logging configuration. I'll investigate.

Ok... the logback jars are being pulled into the distro... If you remove them 
from server/webapp/atlas/WEB-INF/lib you should have the expected behavior. 
I'll work on updating the patch to exclude those jars.


- Pierre


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65885/#review199456
---


On March 19, 2018, 11:51 a.m., Pierre Padovani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65885/
> ---
> 
> (Updated March 19, 2018, 11:51 a.m.)
> 
> 
> Review request for atlas and David Radley.
> 
> 
> Bugs: ATLAS-2470
> https://issues.apache.org/jira/browse/ATLAS-2470
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This updates pom's to add the needed cassandra jars, and adds a dist profile 
> embedded-cassandra-solr.
> 
> 
> Diffs
> -
> 
>   distro/pom.xml 0103bef6 
>   distro/src/bin/atlas_config.py e6415cf4 
>   distro/src/bin/atlas_start.py 39be6b7c 
>   distro/src/bin/atlas_stop.py 66edd904 
>   distro/src/conf/atlas-application.properties 10fb7b86 
>   distro/src/conf/atlas-env.sh 68b24e93 
>   distro/src/conf/cassandra.yml.template PRE-CREATION 
>   distro/src/conf/zookeeper/log4j.properties PRE-CREATION 
>   distro/src/conf/zookeeper/zoo.cfg.template PRE-CREATION 
>   distro/src/main/assemblies/standalone-package.xml 1881082e 
>   docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 
>   graphdb/janus/pom.xml 143b775f 
>   repository/pom.xml b1d6b1f9 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/AbstractStorageBasedAuditRepository.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
>  cee7374e 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/AuditRepositoryTestBase.java
>  4d70b7f7 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/CassandraAuditRepositoryTest.java
>  PRE-CREATION 
>   repository/src/test/resources/cassandra_test.yml PRE-CREATION 
>   
> server-api/src/main/java/org/apache/atlas/listener/ActiveStateChangeHandler.java
>  7d282b30 
> 
> 
> Diff: https://reviews.apache.org/r/65885/diff/3/
> 
> 
> Testing
> ---
> 
> Full build with the new embedded-cassandra-solr, and testing to make sure 
> Atlas comes up and is functional.
> 
> Be aware that we have been running Cassandra backed JanusGraph for months 
> with no issues.
> 
> This patch now includes support for Cassandra as an audit store. I've also 
> refactored the audit unit tests, and added cassandra specific tests.
> 
> 
> Thanks,
> 
> Pierre Padovani
> 
>



Re: Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas

2018-03-19 Thread Pierre Padovani


> On March 19, 2018, 7:28 p.m., David Radley wrote:
> > docs/src/site/twiki/InstallationSteps.twiki
> > Lines 45 (patched)
> > <https://reviews.apache.org/r/65885/diff/3/?file=1982164#file1982164line45>
> >
> > I built Atlas as per these instructions. I then started atlas. Atlas 
> > started and worked - but when I looked at the application.log there was 
> > only 3 log entries.  
> > 
> > 2018-03-19 12:22:22,447 WARN  - [main:] ~ No meta.properties file under 
> > dir 
> > /Users/david/atlasreview2/distro/target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/data/kafka/kafka/meta.properties
> >  (Logging$class:87)
> > 2018-03-19 12:22:22,922 WARN  - [main:] ~ No meta.properties file under 
> > dir 
> > /Users/david/atlasreview2/distro/target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/data/kafka/kafka/meta.properties
> >  (Logging$class:87)
> > 2018-03-19 12:22:22,986 INFO  - [NotificationHookConsumer thread-0:] ~ 
> > [atlas-hook-consumer-thread]: Starting (Logging$class:72)
> > 
> > 
> > I would have expected much more logging.
> 
> Pierre Padovani wrote:
> This confused me as well. All of the logging is in the the newest 
> atlas..out file. Which makes sense when log4j is configured to 
> log to a rolling date/time stamped file. In previous builds of Atlas the 
> logging all showed up in the application.log file. This change in behavior 
> happened to me within the last few weeks. Do we know if a change was made to 
> induce this? or Was this induced by my changes?
> 
> I'll do some testing on my side.

Just retested off of master... something in the cassandra jars for Janus must 
be overriding the Atlas logging configuration. I'll investigate.


- Pierre


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65885/#review199456
---


On March 19, 2018, 11:51 a.m., Pierre Padovani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65885/
> ---
> 
> (Updated March 19, 2018, 11:51 a.m.)
> 
> 
> Review request for atlas and David Radley.
> 
> 
> Bugs: ATLAS-2470
> https://issues.apache.org/jira/browse/ATLAS-2470
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This updates pom's to add the needed cassandra jars, and adds a dist profile 
> embedded-cassandra-solr.
> 
> 
> Diffs
> -
> 
>   distro/pom.xml 0103bef6 
>   distro/src/bin/atlas_config.py e6415cf4 
>   distro/src/bin/atlas_start.py 39be6b7c 
>   distro/src/bin/atlas_stop.py 66edd904 
>   distro/src/conf/atlas-application.properties 10fb7b86 
>   distro/src/conf/atlas-env.sh 68b24e93 
>   distro/src/conf/cassandra.yml.template PRE-CREATION 
>   distro/src/conf/zookeeper/log4j.properties PRE-CREATION 
>   distro/src/conf/zookeeper/zoo.cfg.template PRE-CREATION 
>   distro/src/main/assemblies/standalone-package.xml 1881082e 
>   docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 
>   graphdb/janus/pom.xml 143b775f 
>   repository/pom.xml b1d6b1f9 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/AbstractStorageBasedAuditRepository.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
>  cee7374e 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/AuditRepositoryTestBase.java
>  4d70b7f7 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/CassandraAuditRepositoryTest.java
>  PRE-CREATION 
>   repository/src/test/resources/cassandra_test.yml PRE-CREATION 
>   
> server-api/src/main/java/org/apache/atlas/listener/ActiveStateChangeHandler.java
>  7d282b30 
> 
> 
> Diff: https://reviews.apache.org/r/65885/diff/3/
> 
> 
> Testing
> ---
> 
> Full build with the new embedded-cassandra-solr, and testing to make sure 
> Atlas comes up and is functional.
> 
> Be aware that we have been running Cassandra backed JanusGraph for months 
> with no issues.
> 
> This patch now includes support for Cassandra as an audit store. I've also 
> refactored the audit unit tests, and added cassandra specific tests.
> 
> 
> Thanks,
> 
> Pierre Padovani
> 
>



Re: Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas

2018-03-19 Thread Pierre Padovani


> On March 19, 2018, 7:28 p.m., David Radley wrote:
> > docs/src/site/twiki/InstallationSteps.twiki
> > Lines 45 (patched)
> > <https://reviews.apache.org/r/65885/diff/3/?file=1982164#file1982164line45>
> >
> > I built Atlas as per these instructions. I then started atlas. Atlas 
> > started and worked - but when I looked at the application.log there was 
> > only 3 log entries.  
> > 
> > 2018-03-19 12:22:22,447 WARN  - [main:] ~ No meta.properties file under 
> > dir 
> > /Users/david/atlasreview2/distro/target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/data/kafka/kafka/meta.properties
> >  (Logging$class:87)
> > 2018-03-19 12:22:22,922 WARN  - [main:] ~ No meta.properties file under 
> > dir 
> > /Users/david/atlasreview2/distro/target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/data/kafka/kafka/meta.properties
> >  (Logging$class:87)
> > 2018-03-19 12:22:22,986 INFO  - [NotificationHookConsumer thread-0:] ~ 
> > [atlas-hook-consumer-thread]: Starting (Logging$class:72)
> > 
> > 
> > I would have expected much more logging.

This confused me as well. All of the logging is in the the newest atlas..out file. Which makes sense when log4j is configured to log to a rolling 
date/time stamped file. In previous builds of Atlas the logging all showed up 
in the application.log file. This change in behavior happened to me within the 
last few weeks. Do we know if a change was made to induce this? or Was this 
induced by my changes?

I'll do some testing on my side.


- Pierre


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65885/#review199456
---


On March 19, 2018, 11:51 a.m., Pierre Padovani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65885/
> ---
> 
> (Updated March 19, 2018, 11:51 a.m.)
> 
> 
> Review request for atlas and David Radley.
> 
> 
> Bugs: ATLAS-2470
> https://issues.apache.org/jira/browse/ATLAS-2470
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This updates pom's to add the needed cassandra jars, and adds a dist profile 
> embedded-cassandra-solr.
> 
> 
> Diffs
> -
> 
>   distro/pom.xml 0103bef6 
>   distro/src/bin/atlas_config.py e6415cf4 
>   distro/src/bin/atlas_start.py 39be6b7c 
>   distro/src/bin/atlas_stop.py 66edd904 
>   distro/src/conf/atlas-application.properties 10fb7b86 
>   distro/src/conf/atlas-env.sh 68b24e93 
>   distro/src/conf/cassandra.yml.template PRE-CREATION 
>   distro/src/conf/zookeeper/log4j.properties PRE-CREATION 
>   distro/src/conf/zookeeper/zoo.cfg.template PRE-CREATION 
>   distro/src/main/assemblies/standalone-package.xml 1881082e 
>   docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 
>   graphdb/janus/pom.xml 143b775f 
>   repository/pom.xml b1d6b1f9 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/AbstractStorageBasedAuditRepository.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
>  cee7374e 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/AuditRepositoryTestBase.java
>  4d70b7f7 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/CassandraAuditRepositoryTest.java
>  PRE-CREATION 
>   repository/src/test/resources/cassandra_test.yml PRE-CREATION 
>   
> server-api/src/main/java/org/apache/atlas/listener/ActiveStateChangeHandler.java
>  7d282b30 
> 
> 
> Diff: https://reviews.apache.org/r/65885/diff/3/
> 
> 
> Testing
> ---
> 
> Full build with the new embedded-cassandra-solr, and testing to make sure 
> Atlas comes up and is functional.
> 
> Be aware that we have been running Cassandra backed JanusGraph for months 
> with no issues.
> 
> This patch now includes support for Cassandra as an audit store. I've also 
> refactored the audit unit tests, and added cassandra specific tests.
> 
> 
> Thanks,
> 
> Pierre Padovani
> 
>



Re: Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas

2018-03-19 Thread Pierre Padovani


> On March 19, 2018, 12:31 a.m., David Radley wrote:
> > distro/src/conf/atlas-application.properties
> > Line 23 (original), 23 (patched)
> > <https://reviews.apache.org/r/65885/diff/2/?file=1972866#file1972866line23>
> >
> > Sorry for the delay in reviewing. Unfortunatley the patch now does not 
> > apply to master I get: 
> > git am < ../ATLAS-2470-2.patch 
> > Applying: ATLAS-2470 - JanusGraph Cassandra support
> > .git/rebase-apply/patch:1146: trailing whitespace.
> > # The number of ticks that the initial 
> > error: patch failed: distro/src/conf/atlas-application.properties:24
> > error: distro/src/conf/atlas-application.properties: patch does not 
> > apply
> > .git/rebase-apply/patch:1039: new blank line at EOF.
> > +
> > .git/rebase-apply/patch:2761: new blank line at EOF.
> > +
> > Patch failed at 0001 ATLAS-2470 - JanusGraph Cassandra support

Ok.. I uploaded a new diff after rebasing off of master. However it still 
cannot be applied for similar reasons. A quick google soved my issues...

git am --3way < ATLAS-2470-3.patch

properly applies the patch.

https://www.lullabot.com/articles/git-best-practices-upgrading-the-patch-process


- Pierre


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65885/#review199398
---


On March 19, 2018, 11:51 a.m., Pierre Padovani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65885/
> ---
> 
> (Updated March 19, 2018, 11:51 a.m.)
> 
> 
> Review request for atlas and David Radley.
> 
> 
> Bugs: ATLAS-2470
> https://issues.apache.org/jira/browse/ATLAS-2470
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This updates pom's to add the needed cassandra jars, and adds a dist profile 
> embedded-cassandra-solr.
> 
> 
> Diffs
> -
> 
>   distro/pom.xml 0103bef6 
>   distro/src/bin/atlas_config.py e6415cf4 
>   distro/src/bin/atlas_start.py 39be6b7c 
>   distro/src/bin/atlas_stop.py 66edd904 
>   distro/src/conf/atlas-application.properties 10fb7b86 
>   distro/src/conf/atlas-env.sh 68b24e93 
>   distro/src/conf/cassandra.yml.template PRE-CREATION 
>   distro/src/conf/zookeeper/log4j.properties PRE-CREATION 
>   distro/src/conf/zookeeper/zoo.cfg.template PRE-CREATION 
>   distro/src/main/assemblies/standalone-package.xml 1881082e 
>   docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 
>   graphdb/janus/pom.xml 143b775f 
>   repository/pom.xml b1d6b1f9 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/AbstractStorageBasedAuditRepository.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
>  cee7374e 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/AuditRepositoryTestBase.java
>  4d70b7f7 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/CassandraAuditRepositoryTest.java
>  PRE-CREATION 
>   repository/src/test/resources/cassandra_test.yml PRE-CREATION 
>   
> server-api/src/main/java/org/apache/atlas/listener/ActiveStateChangeHandler.java
>  7d282b30 
> 
> 
> Diff: https://reviews.apache.org/r/65885/diff/3/
> 
> 
> Testing
> ---
> 
> Full build with the new embedded-cassandra-solr, and testing to make sure 
> Atlas comes up and is functional.
> 
> Be aware that we have been running Cassandra backed JanusGraph for months 
> with no issues.
> 
> This patch now includes support for Cassandra as an audit store. I've also 
> refactored the audit unit tests, and added cassandra specific tests.
> 
> 
> Thanks,
> 
> Pierre Padovani
> 
>



[jira] [Updated] (ATLAS-2470) Basic support for Cassandra

2018-03-19 Thread Pierre Padovani (JIRA)

 [ 
https://issues.apache.org/jira/browse/ATLAS-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Padovani updated ATLAS-2470:
---
Attachment: ATLAS-2470-3.patch

> Basic support for Cassandra 
> 
>
> Key: ATLAS-2470
> URL: https://issues.apache.org/jira/browse/ATLAS-2470
> Project: Atlas
>  Issue Type: Sub-task
>Affects Versions: 1.0.0
>    Reporter: Pierre Padovani
>    Assignee: Pierre Padovani
>Priority: Major
> Fix For: 1.0.0
>
> Attachments: ATLAS-2470-1.patch, ATLAS-2470-2.patch, 
> ATLAS-2470-3.patch, ATLAS-2470.patch
>
>
> Add the basic build support, and ability to run embedded Cassandra and solr 
> using a profile build:
> -Pdist,embedded-cassandra-solr



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas

2018-03-19 Thread Pierre Padovani

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65885/
---

(Updated March 19, 2018, 11:51 a.m.)


Review request for atlas and David Radley.


Changes
---

Rebased off of master and regenerated the patch. Git is weird... a rebase 
showed no conflicts, but attempting to apply the previous version of the patch 
did.


Bugs: ATLAS-2470
https://issues.apache.org/jira/browse/ATLAS-2470


Repository: atlas


Description
---

This updates pom's to add the needed cassandra jars, and adds a dist profile 
embedded-cassandra-solr.


Diffs (updated)
-

  distro/pom.xml 0103bef6 
  distro/src/bin/atlas_config.py e6415cf4 
  distro/src/bin/atlas_start.py 39be6b7c 
  distro/src/bin/atlas_stop.py 66edd904 
  distro/src/conf/atlas-application.properties 10fb7b86 
  distro/src/conf/atlas-env.sh 68b24e93 
  distro/src/conf/cassandra.yml.template PRE-CREATION 
  distro/src/conf/zookeeper/log4j.properties PRE-CREATION 
  distro/src/conf/zookeeper/zoo.cfg.template PRE-CREATION 
  distro/src/main/assemblies/standalone-package.xml 1881082e 
  docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 
  graphdb/janus/pom.xml 143b775f 
  repository/pom.xml b1d6b1f9 
  
repository/src/main/java/org/apache/atlas/repository/audit/AbstractStorageBasedAuditRepository.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
 cee7374e 
  
repository/src/test/java/org/apache/atlas/repository/audit/AuditRepositoryTestBase.java
 4d70b7f7 
  
repository/src/test/java/org/apache/atlas/repository/audit/CassandraAuditRepositoryTest.java
 PRE-CREATION 
  repository/src/test/resources/cassandra_test.yml PRE-CREATION 
  
server-api/src/main/java/org/apache/atlas/listener/ActiveStateChangeHandler.java
 7d282b30 


Diff: https://reviews.apache.org/r/65885/diff/3/

Changes: https://reviews.apache.org/r/65885/diff/2-3/


Testing
---

Full build with the new embedded-cassandra-solr, and testing to make sure Atlas 
comes up and is functional.

Be aware that we have been running Cassandra backed JanusGraph for months with 
no issues.

This patch now includes support for Cassandra as an audit store. I've also 
refactored the audit unit tests, and added cassandra specific tests.


Thanks,

Pierre Padovani



Re: Draft board report for Apache Atlas: March 2018

2018-03-14 Thread Pierre Padovani
I don't think you intended to say:

" planning to release 1.0.0 by end of April 2010" <-- Perhaps 2018?

I am also unclear where a curated list of remaining work for a 1.0.0
release is. It would make sense to me to have that in place in order to
determine an estimated release date.

Pierre

On Wed, Mar 14, 2018 at 12:30 PM, Madhan Neethiraj <mad...@apache.org>
wrote:

> Atlas team,
>
> Please review the draft board report below and send your feedback/comments.
>
> Thanks,
> Madhan
>
>
> ## Description:
>   Apache Atlas is a scalable and extensible set of core foundational
> governance
>   services that enables enterprises to effectively and efficiently meet
> their
>   compliance requirements within Hadoop and allows integration with the
>   complete enterprise data ecosystem
>
> ## Issues:
>   There are no issues requiring board attention at this time.
>
> ## Activity:
>   - replaced Scala based DSL with ANTLR based implementation
>   - added support for propagation of classifications via relationships
>   - enhanced classifications with ability to specify validity periods
>   - added Open Connector Framework (OCF)
>   - added Open Metadata Repository Services (OMRS) API, and IGC connector
> skeleton
>   - enhanced notifications to support V2 style data structures
>   - enhanced authorization model to support fine-grained authorization
>   - updated UI to render entity relationships
>   - working on migration of data in Atlas 0.8.x (Titan graph DB) to Atlas
> 1.0 (JanusGraph)
>   - released 0.8.2 and 1.0.0-alpha versions
>   - planning to release 1.0.0 by end of April 2010
>
> ## Health report:
>   - 5 new contributors added in last 3 months: Vishal Suvagia, Pierre
> Padovani, Bosco Durai, Shrinivas Kane, Mark Ottesen
>
> ## PMC changes:
>   - Currently 33 PMC members
>   - No new PMC members added in last 3 months
>   - Last PMC member addition was on 6/21/2017
>
> ## Committer base changes:
>   - Currently 37 committers
>   - 1 new committer in last 3 months, Graham Wallis added on 1/9/2018
>   - Last addition to committer role was on 1/9/2017
>
> ## Releases:
>   1.0.0plan to release by 04/30/2018
>   0.8.2was released on 02/05/2018
>   1.0.0-alpha  was released on 01/25/2018
>   0.8.1was released on 08/29/2017
>   0.8-incubating   was released on 03/16/2017
>   0.7.1-incubating was released on 01/26/2017
>   0.7-incubating   was released on 07/09/2016
>   0.6-incubating   was released on 12/31/2015
>   0.5-incubating   was released on 07/11/2015
>
>
>
>


Review Request 66064: ATLAS-2478 Elasticsearch support broken with JanusGraph 0.2.0

2018-03-14 Thread Pierre Padovani

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/
---

Review request for atlas.


Summary (updated)
-

ATLAS-2478 Elasticsearch support broken with JanusGraph 0.2.0


Bugs: ATLAS-2478
https://issues.apache.org/jira/browse/ATLAS-2478


Repository: atlas


Description (updated)
---

This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates 
documentation.

Included with this patch is an update to the berkley-elasticsearch profile to 
automatically download and include elasticsearch as a side application much 
like solr is. Updates to the start/stop/conf scripts are included as well.

NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to 
/atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There 
are six constants that are incorrectly named with a '.' (dot). This is not 
supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the 
field names can be collectively thought of as an object. In the case of the 
fields defined in the Constants.java file, 'type' is defined as a string field, 
and 'type.name' is also defined as a string field. Elasticsearch sees this as 
an error, since it cannot convert type to an object. The fix included simply 
changes the field names from using a '.' (dot) to an '_' (underscore).

**Query**: There is a way we can simplify integration/unit tests for the 
in-memory graph store by using a maven plugin that will download and run an 
elasticsearch node. This is nothing more than a maven change, and change to the 
atlas-application.properties to switch to elasticsearch from solr. I did not 
implement this, but am curious if this change would be desired. If so, this can 
be done with a separate ticket.


Testing (updated)
---

We currently use this fix with our Atlas setup in a fork of the Atlas code, and 
have found no issues with it. Additionally, with the update to the 
berkley-elasticsearch profile, I have extensively tested that profile to make 
sure that management of Elasticsearch functioned correctly.


Thanks,

Pierre Padovani



[jira] [Updated] (ATLAS-2478) Elasticsearch support is broken for JanusGraph

2018-03-14 Thread Pierre Padovani (JIRA)

 [ 
https://issues.apache.org/jira/browse/ATLAS-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Padovani updated ATLAS-2478:
---
Attachment: ATLAS-2478.patch

> Elasticsearch support is broken for JanusGraph
> --
>
> Key: ATLAS-2478
> URL: https://issues.apache.org/jira/browse/ATLAS-2478
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 1.0.0-alpha
>    Reporter: Pierre Padovani
>Assignee: Pierre Padovani
>Priority: Major
> Fix For: 1.0.0
>
> Attachments: ATLAS-2478.patch
>
>
> With JanusGraph the Elasticsearch support moved to 5.x+. This introduced a 
> change where fields that contained '.' (dots) were not allowed unless either 
> a specific cluster wide setting was enabled AND the mapping was formatted 
> such that each of the fields that contained a '.' could be considered part of 
> an object.
> Example:
> {code:java}
> foo.x
> foo.y
> foo.z{code}
>  Elasticsearch looks at these fields as if they are truly:
> {code:java}
> foo : {
>   x,
>   y,
>   z
> }{code}
> In the file:
> /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java
> {code:java}
> /**
>  * Properties for type store graph.
>  */
> public static final String TYPE_CATEGORY_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type.category";
> public static final String VERTEX_TYPE_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type";
> public static final String TYPENAME_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type.name";
> public static final String TYPEDESCRIPTION_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type.description";
> public static final String TYPEVERSION_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type.version";
> public static final String TYPEOPTIONS_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type.options";
> {code}
> These are the only fields that cause Elasticsearch issue. As you can see a 
> field called 'type' is created, then additional fields type.name, 
> type.description etc. This will cause a mapping conflict exception in 
> Elasticsearch and it will refuse to create the mapping.
>  
> The easy fix is to simply replace the '.' with an '_' (underscore) but this 
> will be a backwards incompatible change for existing customers. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (ATLAS-2478) Elasticsearch support is broken for JanusGraph

2018-03-14 Thread Pierre Padovani (JIRA)

 [ 
https://issues.apache.org/jira/browse/ATLAS-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Padovani updated ATLAS-2478:
---
Attachment: (was: ATLAS-2478.patch)

> Elasticsearch support is broken for JanusGraph
> --
>
> Key: ATLAS-2478
> URL: https://issues.apache.org/jira/browse/ATLAS-2478
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 1.0.0-alpha
>    Reporter: Pierre Padovani
>Assignee: Pierre Padovani
>Priority: Major
> Fix For: 1.0.0
>
>
> With JanusGraph the Elasticsearch support moved to 5.x+. This introduced a 
> change where fields that contained '.' (dots) were not allowed unless either 
> a specific cluster wide setting was enabled AND the mapping was formatted 
> such that each of the fields that contained a '.' could be considered part of 
> an object.
> Example:
> {code:java}
> foo.x
> foo.y
> foo.z{code}
>  Elasticsearch looks at these fields as if they are truly:
> {code:java}
> foo : {
>   x,
>   y,
>   z
> }{code}
> In the file:
> /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java
> {code:java}
> /**
>  * Properties for type store graph.
>  */
> public static final String TYPE_CATEGORY_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type.category";
> public static final String VERTEX_TYPE_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type";
> public static final String TYPENAME_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type.name";
> public static final String TYPEDESCRIPTION_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type.description";
> public static final String TYPEVERSION_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type.version";
> public static final String TYPEOPTIONS_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type.options";
> {code}
> These are the only fields that cause Elasticsearch issue. As you can see a 
> field called 'type' is created, then additional fields type.name, 
> type.description etc. This will cause a mapping conflict exception in 
> Elasticsearch and it will refuse to create the mapping.
>  
> The easy fix is to simply replace the '.' with an '_' (underscore) but this 
> will be a backwards incompatible change for existing customers. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (ATLAS-2478) Elasticsearch support is broken for JanusGraph

2018-03-14 Thread Pierre Padovani (JIRA)

 [ 
https://issues.apache.org/jira/browse/ATLAS-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Padovani updated ATLAS-2478:
---
Attachment: ATLAS-2478.patch

> Elasticsearch support is broken for JanusGraph
> --
>
> Key: ATLAS-2478
> URL: https://issues.apache.org/jira/browse/ATLAS-2478
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 1.0.0-alpha
>    Reporter: Pierre Padovani
>Assignee: Pierre Padovani
>Priority: Major
> Fix For: 1.0.0
>
> Attachments: ATLAS-2478.patch
>
>
> With JanusGraph the Elasticsearch support moved to 5.x+. This introduced a 
> change where fields that contained '.' (dots) were not allowed unless either 
> a specific cluster wide setting was enabled AND the mapping was formatted 
> such that each of the fields that contained a '.' could be considered part of 
> an object.
> Example:
> {code:java}
> foo.x
> foo.y
> foo.z{code}
>  Elasticsearch looks at these fields as if they are truly:
> {code:java}
> foo : {
>   x,
>   y,
>   z
> }{code}
> In the file:
> /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java
> {code:java}
> /**
>  * Properties for type store graph.
>  */
> public static final String TYPE_CATEGORY_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type.category";
> public static final String VERTEX_TYPE_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type";
> public static final String TYPENAME_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type.name";
> public static final String TYPEDESCRIPTION_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type.description";
> public static final String TYPEVERSION_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type.version";
> public static final String TYPEOPTIONS_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type.options";
> {code}
> These are the only fields that cause Elasticsearch issue. As you can see a 
> field called 'type' is created, then additional fields type.name, 
> type.description etc. This will cause a mapping conflict exception in 
> Elasticsearch and it will refuse to create the mapping.
>  
> The easy fix is to simply replace the '.' with an '_' (underscore) but this 
> will be a backwards incompatible change for existing customers. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas

2018-03-09 Thread Pierre Padovani


> On March 9, 2018, 1:38 a.m., Apoorv Naik wrote:
> > distro/pom.xml
> > Lines 252 (patched)
> > <https://reviews.apache.org/r/65885/diff/1/?file=1969111#file1969111line252>
> >
> > Although harmless, it's better to even avoid download if it's not 
> > needed. It's always a good thing to avoid unnecessary jars in the final 
> > distro, makes troubleshooting easier. 
> > 
> > Also this would increase the build time, why to waste CPU/Network 
> > resources on something if it's not needed.

This was removed from the habase profile in the current patch. The directions 
for updating a patch were unclear on the wiki, and I had forgotten to update 
the patch on the review board.


> On March 9, 2018, 1:38 a.m., Apoorv Naik wrote:
> > distro/src/main/assemblies/standalone-package.xml
> > Lines 83 (patched)
> > <https://reviews.apache.org/r/65885/diff/1/?file=1969118#file1969118line83>
> >
> > ZK might be available in certain deployments and this would add 
> > unnecessary ZK jars to classpath. Potential to cause mess ups in production 
> > environments.

I'm not sure this is the case. The zk directory shows up at the root of the 
atlas directory, not within the server classpath directory. Additionally, the 
updated patch only downloads it and installs it if the embedded-cassandra-solr 
profile is used. So this particular operation is a noop unless that profile is 
used.


- Pierre


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65885/#review198893
---


On March 8, 2018, 6:53 p.m., Pierre Padovani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65885/
> ---
> 
> (Updated March 8, 2018, 6:53 p.m.)
> 
> 
> Review request for atlas and David Radley.
> 
> 
> Bugs: ATLAS-2470
> https://issues.apache.org/jira/browse/ATLAS-2470
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This updates pom's to add the needed cassandra jars, and adds a dist profile 
> embedded-cassandra-solr.
> 
> 
> Diffs
> -
> 
>   distro/pom.xml 0103bef6 
>   distro/src/bin/atlas_config.py e6415cf4 
>   distro/src/bin/atlas_start.py 39be6b7c 
>   distro/src/bin/atlas_stop.py 66edd904 
>   distro/src/conf/atlas-application.properties 355abf44 
>   distro/src/conf/atlas-env.sh 68b24e93 
>   distro/src/conf/cassandra.yml.template PRE-CREATION 
>   distro/src/conf/zookeeper/log4j.properties PRE-CREATION 
>   distro/src/conf/zookeeper/zoo.cfg.template PRE-CREATION 
>   distro/src/main/assemblies/standalone-package.xml 1881082e 
>   docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 
>   graphdb/janus/pom.xml 143b775f 
>   repository/pom.xml b1d6b1f9 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/AbstractStorageBasedAuditRepository.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
>  cee7374e 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/AuditRepositoryTestBase.java
>  4d70b7f7 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/CassandraAuditRepositoryTest.java
>  PRE-CREATION 
>   repository/src/test/resources/cassandra_test.yml PRE-CREATION 
>   
> server-api/src/main/java/org/apache/atlas/listener/ActiveStateChangeHandler.java
>  7d282b30 
> 
> 
> Diff: https://reviews.apache.org/r/65885/diff/2/
> 
> 
> Testing
> ---
> 
> Full build with the new embedded-cassandra-solr, and testing to make sure 
> Atlas comes up and is functional.
> 
> Be aware that we have been running Cassandra backed JanusGraph for months 
> with no issues.
> 
> This patch now includes support for Cassandra as an audit store. I've also 
> refactored the audit unit tests, and added cassandra specific tests.
> 
> 
> Thanks,
> 
> Pierre Padovani
> 
>



Re: Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas

2018-03-08 Thread Pierre Padovani

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65885/
---

(Updated March 8, 2018, 6:53 p.m.)


Review request for atlas and David Radley.


Bugs: ATLAS-2470
https://issues.apache.org/jira/browse/ATLAS-2470


Repository: atlas


Description
---

This updates pom's to add the needed cassandra jars, and adds a dist profile 
embedded-cassandra-solr.


Diffs
-

  distro/pom.xml 0103bef6 
  distro/src/bin/atlas_config.py e6415cf4 
  distro/src/bin/atlas_start.py 39be6b7c 
  distro/src/bin/atlas_stop.py 66edd904 
  distro/src/conf/atlas-application.properties 355abf44 
  distro/src/conf/atlas-env.sh 68b24e93 
  distro/src/conf/cassandra.yml.template PRE-CREATION 
  distro/src/conf/zookeeper/log4j.properties PRE-CREATION 
  distro/src/conf/zookeeper/zoo.cfg.template PRE-CREATION 
  distro/src/main/assemblies/standalone-package.xml 1881082e 
  docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 
  graphdb/janus/pom.xml 143b775f 
  repository/pom.xml b1d6b1f9 
  
repository/src/main/java/org/apache/atlas/repository/audit/AbstractStorageBasedAuditRepository.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
 cee7374e 
  
repository/src/test/java/org/apache/atlas/repository/audit/AuditRepositoryTestBase.java
 4d70b7f7 
  
repository/src/test/java/org/apache/atlas/repository/audit/CassandraAuditRepositoryTest.java
 PRE-CREATION 
  repository/src/test/resources/cassandra_test.yml PRE-CREATION 
  
server-api/src/main/java/org/apache/atlas/listener/ActiveStateChangeHandler.java
 7d282b30 


Diff: https://reviews.apache.org/r/65885/diff/2/


Testing (updated)
---

Full build with the new embedded-cassandra-solr, and testing to make sure Atlas 
comes up and is functional.

Be aware that we have been running Cassandra backed JanusGraph for months with 
no issues.

This patch now includes support for Cassandra as an audit store. I've also 
refactored the audit unit tests, and added cassandra specific tests.


Thanks,

Pierre Padovani



Re: Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas

2018-03-08 Thread Pierre Padovani

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65885/
---

(Updated March 8, 2018, 6:51 p.m.)


Review request for atlas and David Radley.


Bugs: ATLAS-2470
https://issues.apache.org/jira/browse/ATLAS-2470


Repository: atlas


Description
---

This updates pom's to add the needed cassandra jars, and adds a dist profile 
embedded-cassandra-solr.


Diffs (updated)
-

  distro/pom.xml 0103bef6 
  distro/src/bin/atlas_config.py e6415cf4 
  distro/src/bin/atlas_start.py 39be6b7c 
  distro/src/bin/atlas_stop.py 66edd904 
  distro/src/conf/atlas-application.properties 355abf44 
  distro/src/conf/atlas-env.sh 68b24e93 
  distro/src/conf/cassandra.yml.template PRE-CREATION 
  distro/src/conf/zookeeper/log4j.properties PRE-CREATION 
  distro/src/conf/zookeeper/zoo.cfg.template PRE-CREATION 
  distro/src/main/assemblies/standalone-package.xml 1881082e 
  docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 
  graphdb/janus/pom.xml 143b775f 
  repository/pom.xml b1d6b1f9 
  
repository/src/main/java/org/apache/atlas/repository/audit/AbstractStorageBasedAuditRepository.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
 cee7374e 
  
repository/src/test/java/org/apache/atlas/repository/audit/AuditRepositoryTestBase.java
 4d70b7f7 
  
repository/src/test/java/org/apache/atlas/repository/audit/CassandraAuditRepositoryTest.java
 PRE-CREATION 
  repository/src/test/resources/cassandra_test.yml PRE-CREATION 
  
server-api/src/main/java/org/apache/atlas/listener/ActiveStateChangeHandler.java
 7d282b30 


Diff: https://reviews.apache.org/r/65885/diff/2/

Changes: https://reviews.apache.org/r/65885/diff/1-2/


Testing
---

Full build with the new embedded-cassandra-solr, and testing to make sure Atlas 
comes up and is functional.

Be aware that we have been running Cassandra backed JanusGraph for months with 
no issues.


Thanks,

Pierre Padovani



Re: Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas

2018-03-08 Thread Pierre Padovani


> On March 8, 2018, 6:02 p.m., Apoorv Naik wrote:
> > distro/pom.xml
> > Lines 252 (patched)
> > <https://reviews.apache.org/r/65885/diff/1/?file=1969111#file1969111line252>
> >
> > Looks like this will be run for all profiles, can you make this 
> > specific to cassandra profile ?

Does it matter here? This just downloads the resource. It will only be used if 
the embedded-cassandra-solr dist profile is used, otherwise the download is 
ignored.


- Pierre


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65885/#review198890
-------


On March 2, 2018, 4:51 p.m., Pierre Padovani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65885/
> ---
> 
> (Updated March 2, 2018, 4:51 p.m.)
> 
> 
> Review request for atlas and David Radley.
> 
> 
> Bugs: ATLAS-2470
> https://issues.apache.org/jira/browse/ATLAS-2470
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This updates pom's to add the needed cassandra jars, and adds a dist profile 
> embedded-cassandra-solr.
> 
> 
> Diffs
> -
> 
>   distro/pom.xml 0103bef6 
>   distro/src/bin/atlas_config.py e6415cf4 
>   distro/src/bin/atlas_start.py 39be6b7c 
>   distro/src/bin/atlas_stop.py 66edd904 
>   distro/src/conf/cassandra.yml PRE-CREATION 
>   distro/src/conf/zookeeper/log4j.properties PRE-CREATION 
>   distro/src/conf/zookeeper/zoo.cfg PRE-CREATION 
>   distro/src/main/assemblies/standalone-package.xml 1881082e 
>   docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 
>   graphdb/janus/pom.xml 143b775f 
> 
> 
> Diff: https://reviews.apache.org/r/65885/diff/1/
> 
> 
> Testing
> ---
> 
> Full build with the new embedded-cassandra-solr, and testing to make sure 
> Atlas comes up and is functional.
> 
> Be aware that we have been running Cassandra backed JanusGraph for months 
> with no issues.
> 
> 
> Thanks,
> 
> Pierre Padovani
> 
>



[jira] [Commented] (ATLAS-2470) Basic support for Cassandra

2018-03-08 Thread Pierre Padovani (JIRA)

[ 
https://issues.apache.org/jira/browse/ATLAS-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391565#comment-16391565
 ] 

Pierre Padovani commented on ATLAS-2470:


[~davidrad]

Added an updated patch that includes:
 * Audit support for Cassandra as a storage backend
 * Refactored audit unit tests for audit to support adding a Cassandra unit test
 * Updated audit unit tests to test the V2 audit code for both in memory and 
Cassandra 
 * Updates to application.properties and distro pom.xml

> Basic support for Cassandra 
> 
>
> Key: ATLAS-2470
> URL: https://issues.apache.org/jira/browse/ATLAS-2470
> Project: Atlas
>  Issue Type: Sub-task
>Affects Versions: 1.0.0
>    Reporter: Pierre Padovani
>    Assignee: Pierre Padovani
>Priority: Major
> Fix For: 1.0.0
>
> Attachments: ATLAS-2470-1.patch, ATLAS-2470-2.patch, ATLAS-2470.patch
>
>
> Add the basic build support, and ability to run embedded Cassandra and solr 
> using a profile build:
> -Pdist,embedded-cassandra-solr



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (ATLAS-2470) Basic support for Cassandra

2018-03-08 Thread Pierre Padovani (JIRA)

 [ 
https://issues.apache.org/jira/browse/ATLAS-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Padovani updated ATLAS-2470:
---
Attachment: ATLAS-2470-2.patch

> Basic support for Cassandra 
> 
>
> Key: ATLAS-2470
> URL: https://issues.apache.org/jira/browse/ATLAS-2470
> Project: Atlas
>  Issue Type: Sub-task
>Affects Versions: 1.0.0
>    Reporter: Pierre Padovani
>    Assignee: Pierre Padovani
>Priority: Major
> Fix For: 1.0.0
>
> Attachments: ATLAS-2470-1.patch, ATLAS-2470-2.patch, ATLAS-2470.patch
>
>
> Add the basic build support, and ability to run embedded Cassandra and solr 
> using a profile build:
> -Pdist,embedded-cassandra-solr



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (ATLAS-2478) Elasticsearch support is broken for JanusGraph

2018-03-05 Thread Pierre Padovani (JIRA)

[ 
https://issues.apache.org/jira/browse/ATLAS-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386101#comment-16386101
 ] 

Pierre Padovani commented on ATLAS-2478:


[~davidrad] Could I get your thoughts on this issue please? Include anyone else 
you think might have some input.

 

Thanks!

> Elasticsearch support is broken for JanusGraph
> --
>
> Key: ATLAS-2478
> URL: https://issues.apache.org/jira/browse/ATLAS-2478
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 1.0.0-alpha
>    Reporter: Pierre Padovani
>Assignee: Pierre Padovani
>Priority: Major
> Fix For: 1.0.0
>
>
> With JanusGraph the Elasticsearch support moved to 5.x+. This introduced a 
> change where fields that contained '.' (dots) were not allowed unless either 
> a specific cluster wide setting was enabled AND the mapping was formatted 
> such that each of the fields that contained a '.' could be considered part of 
> an object.
> Example:
> {code:java}
> foo.x
> foo.y
> foo.z{code}
>  Elasticsearch looks at these fields as if they are truly:
> {code:java}
> foo : {
>   x,
>   y,
>   z
> }{code}
> In the file:
> /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java
> {code:java}
> /**
>  * Properties for type store graph.
>  */
> public static final String TYPE_CATEGORY_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type.category";
> public static final String VERTEX_TYPE_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type";
> public static final String TYPENAME_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type.name";
> public static final String TYPEDESCRIPTION_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type.description";
> public static final String TYPEVERSION_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type.version";
> public static final String TYPEOPTIONS_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type.options";
> {code}
> These are the only fields that cause Elasticsearch issue. As you can see a 
> field called 'type' is created, then additional fields type.name, 
> type.description etc. This will cause a mapping conflict exception in 
> Elasticsearch and it will refuse to create the mapping.
>  
> The easy fix is to simply replace the '.' with an '_' (underscore) but this 
> will be a backwards incompatible change for existing customers. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (ATLAS-2478) Elasticsearch support is broken for JanusGraph

2018-03-05 Thread Pierre Padovani (JIRA)
Pierre Padovani created ATLAS-2478:
--

 Summary: Elasticsearch support is broken for JanusGraph
 Key: ATLAS-2478
 URL: https://issues.apache.org/jira/browse/ATLAS-2478
 Project: Atlas
  Issue Type: Bug
  Components:  atlas-core
Affects Versions: 1.0.0-alpha
Reporter: Pierre Padovani
Assignee: Pierre Padovani
 Fix For: 1.0.0


With JanusGraph the Elasticsearch support moved to 5.x+. This introduced a 
change where fields that contained '.' (dots) were not allowed unless either a 
specific cluster wide setting was enabled AND the mapping was formatted such 
that each of the fields that contained a '.' could be considered part of an 
object.

Example:
{code:java}
foo.x
foo.y
foo.z{code}
 Elasticsearch looks at these fields as if they are truly:
{code:java}
foo : {
  x,
  y,
  z
}{code}
In the file:

/atlas/common/src/main/java/org/apache/atlas/repository/Constants.java
{code:java}
/**
 * Properties for type store graph.
 */
public static final String TYPE_CATEGORY_PROPERTY_KEY = 
INTERNAL_PROPERTY_KEY_PREFIX + "type.category";
public static final String VERTEX_TYPE_PROPERTY_KEY = 
INTERNAL_PROPERTY_KEY_PREFIX + "type";
public static final String TYPENAME_PROPERTY_KEY = INTERNAL_PROPERTY_KEY_PREFIX 
+ "type.name";
public static final String TYPEDESCRIPTION_PROPERTY_KEY = 
INTERNAL_PROPERTY_KEY_PREFIX + "type.description";
public static final String TYPEVERSION_PROPERTY_KEY = 
INTERNAL_PROPERTY_KEY_PREFIX + "type.version";
public static final String TYPEOPTIONS_PROPERTY_KEY = 
INTERNAL_PROPERTY_KEY_PREFIX + "type.options";

{code}
These are the only fields that cause Elasticsearch issue. As you can see a 
field called 'type' is created, then additional fields type.name, 
type.description etc. This will cause a mapping conflict exception in 
Elasticsearch and it will refuse to create the mapping.

 

The easy fix is to simply replace the '.' with an '_' (underscore) but this 
will be a backwards incompatible change for existing customers. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (ATLAS-2472) Update Atlas documentation for Cassandra support

2018-03-05 Thread Pierre Padovani (JIRA)

 [ 
https://issues.apache.org/jira/browse/ATLAS-2472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Padovani reassigned ATLAS-2472:
--

Assignee: Pierre Padovani

> Update Atlas documentation for Cassandra support
> 
>
> Key: ATLAS-2472
> URL: https://issues.apache.org/jira/browse/ATLAS-2472
> Project: Atlas
>  Issue Type: Sub-task
>    Reporter: Pierre Padovani
>    Assignee: Pierre Padovani
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (ATLAS-2471) Create Dockerfile support for Cassandra

2018-03-05 Thread Pierre Padovani (JIRA)

 [ 
https://issues.apache.org/jira/browse/ATLAS-2471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Padovani reassigned ATLAS-2471:
--

Assignee: Pierre Padovani

> Create Dockerfile support for Cassandra
> ---
>
> Key: ATLAS-2471
> URL: https://issues.apache.org/jira/browse/ATLAS-2471
> Project: Atlas
>  Issue Type: Sub-task
>    Reporter: Pierre Padovani
>    Assignee: Pierre Padovani
>Priority: Major
>
> Add an additional Dockerfile that builds a self contained container that runs 
> on Cassandra 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas

2018-03-05 Thread Pierre Padovani


> On March 4, 2018, 11:07 a.m., David Radley wrote:
> > My review comments are from my initial look at the code; I hope to try 
> > running this patch to verify it works for me
> 
> Pierre Padovani wrote:
> I'll update the patch with the above changes today or tomorrow as I have 
> time.
> 
> Pierre Padovani wrote:
> I've updated the ATLAS-2470 with a new patch that should fix all of these 
> issues.

Hi David, I think I'm missing something. The instructions on the 'Using GIT 
with Atlas' page are unclear on patch revisions. Do I create another review, or 
do I somehow update this review?


- Pierre


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65885/#review198593
---


On March 2, 2018, 4:51 p.m., Pierre Padovani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65885/
> ---
> 
> (Updated March 2, 2018, 4:51 p.m.)
> 
> 
> Review request for atlas and David Radley.
> 
> 
> Bugs: ATLAS-2470
> https://issues.apache.org/jira/browse/ATLAS-2470
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This updates pom's to add the needed cassandra jars, and adds a dist profile 
> embedded-cassandra-solr.
> 
> 
> Diffs
> -
> 
>   distro/pom.xml 0103bef6 
>   distro/src/bin/atlas_config.py e6415cf4 
>   distro/src/bin/atlas_start.py 39be6b7c 
>   distro/src/bin/atlas_stop.py 66edd904 
>   distro/src/conf/cassandra.yml PRE-CREATION 
>   distro/src/conf/zookeeper/log4j.properties PRE-CREATION 
>   distro/src/conf/zookeeper/zoo.cfg PRE-CREATION 
>   distro/src/main/assemblies/standalone-package.xml 1881082e 
>   docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 
>   graphdb/janus/pom.xml 143b775f 
> 
> 
> Diff: https://reviews.apache.org/r/65885/diff/1/
> 
> 
> Testing
> ---
> 
> Full build with the new embedded-cassandra-solr, and testing to make sure 
> Atlas comes up and is functional.
> 
> Be aware that we have been running Cassandra backed JanusGraph for months 
> with no issues.
> 
> 
> Thanks,
> 
> Pierre Padovani
> 
>



[jira] [Updated] (ATLAS-2470) Basic support for Cassandra

2018-03-05 Thread Pierre Padovani (JIRA)

 [ 
https://issues.apache.org/jira/browse/ATLAS-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Padovani updated ATLAS-2470:
---
Attachment: ATLAS-2470-1.patch

> Basic support for Cassandra 
> 
>
> Key: ATLAS-2470
> URL: https://issues.apache.org/jira/browse/ATLAS-2470
> Project: Atlas
>  Issue Type: Sub-task
>Affects Versions: 1.0.0
>    Reporter: Pierre Padovani
>    Assignee: Pierre Padovani
>Priority: Major
> Fix For: 1.0.0
>
> Attachments: ATLAS-2470-1.patch, ATLAS-2470.patch
>
>
> Add the basic build support, and ability to run embedded Cassandra and solr 
> using a profile build:
> -Pdist,embedded-cassandra-solr



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas

2018-03-04 Thread Pierre Padovani


> On March 4, 2018, 11:07 a.m., David Radley wrote:
> > My review comments are from my initial look at the code; I hope to try 
> > running this patch to verify it works for me
> 
> Pierre Padovani wrote:
> I'll update the patch with the above changes today or tomorrow as I have 
> time.

I've updated the ATLAS-2470 with a new patch that should fix all of these 
issues.


- Pierre


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65885/#review198593
---


On March 2, 2018, 4:51 p.m., Pierre Padovani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65885/
> ---
> 
> (Updated March 2, 2018, 4:51 p.m.)
> 
> 
> Review request for atlas and David Radley.
> 
> 
> Bugs: ATLAS-2470
> https://issues.apache.org/jira/browse/ATLAS-2470
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This updates pom's to add the needed cassandra jars, and adds a dist profile 
> embedded-cassandra-solr.
> 
> 
> Diffs
> -
> 
>   distro/pom.xml 0103bef6 
>   distro/src/bin/atlas_config.py e6415cf4 
>   distro/src/bin/atlas_start.py 39be6b7c 
>   distro/src/bin/atlas_stop.py 66edd904 
>   distro/src/conf/cassandra.yml PRE-CREATION 
>   distro/src/conf/zookeeper/log4j.properties PRE-CREATION 
>   distro/src/conf/zookeeper/zoo.cfg PRE-CREATION 
>   distro/src/main/assemblies/standalone-package.xml 1881082e 
>   docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 
>   graphdb/janus/pom.xml 143b775f 
> 
> 
> Diff: https://reviews.apache.org/r/65885/diff/1/
> 
> 
> Testing
> ---
> 
> Full build with the new embedded-cassandra-solr, and testing to make sure 
> Atlas comes up and is functional.
> 
> Be aware that we have been running Cassandra backed JanusGraph for months 
> with no issues.
> 
> 
> Thanks,
> 
> Pierre Padovani
> 
>



[jira] [Commented] (ATLAS-2470) Basic support for Cassandra

2018-03-04 Thread Pierre Padovani (JIRA)

[ 
https://issues.apache.org/jira/browse/ATLAS-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16385407#comment-16385407
 ] 

Pierre Padovani commented on ATLAS-2470:


Added a new patch version with changes from the code review.

> Basic support for Cassandra 
> 
>
> Key: ATLAS-2470
> URL: https://issues.apache.org/jira/browse/ATLAS-2470
> Project: Atlas
>  Issue Type: Sub-task
>Affects Versions: 1.0.0
>    Reporter: Pierre Padovani
>    Assignee: Pierre Padovani
>Priority: Major
> Fix For: 1.0.0
>
> Attachments: ATLAS-2470-1.patch, ATLAS-2470.patch
>
>
> Add the basic build support, and ability to run embedded Cassandra and solr 
> using a profile build:
> -Pdist,embedded-cassandra-solr



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (ATLAS-2470) Basic support for Cassandra

2018-03-04 Thread Pierre Padovani (JIRA)

 [ 
https://issues.apache.org/jira/browse/ATLAS-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Padovani updated ATLAS-2470:
---
Attachment: ATLAS-2470-1.patch

> Basic support for Cassandra 
> 
>
> Key: ATLAS-2470
> URL: https://issues.apache.org/jira/browse/ATLAS-2470
> Project: Atlas
>  Issue Type: Sub-task
>Affects Versions: 1.0.0
>    Reporter: Pierre Padovani
>    Assignee: Pierre Padovani
>Priority: Major
> Fix For: 1.0.0
>
> Attachments: ATLAS-2470-1.patch, ATLAS-2470.patch
>
>
> Add the basic build support, and ability to run embedded Cassandra and solr 
> using a profile build:
> -Pdist,embedded-cassandra-solr



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas

2018-03-04 Thread Pierre Padovani


> On March 4, 2018, 11:04 a.m., David Radley wrote:
> > distro/pom.xml
> > Lines 258 (patched)
> > <https://reviews.apache.org/r/65885/diff/1/?file=1969111#file1969111line258>
> >
> > I see that you are downloading Zookeeper. Many components require 
> > zookeeper - so it may already exist on the system. I think for production 
> > you would want to run with an external Zookeeper. 
> > 
> > I suggest we allow this as a build option. I realise this jira is for 
> > the embedded Cssandra build only - but I could envisage you needing 
> > non-embedded cassandra builds as well; I thought I would suggest this here 
> > - if this is a requirement for you, you  may decide you want to address 
> > this in a separate Jira.

Yes, this was meant only for the embedded side of things, which would typically 
only happen for a development installation or a Docker image. My intention is 
to address setup and config for cassandra in a production environment in the 
documentation. I have another task specific to updating the documentation 
around this. My hope was to get this merged to master first, then I can get the 
Dockerfile built out that supports this configuration in another task.


> On March 4, 2018, 11:04 a.m., David Radley wrote:
> > distro/pom.xml
> > Lines 302 (patched)
> > <https://reviews.apache.org/r/65885/diff/1/?file=1969111#file1969111line302>
> >
> > for consistency should we have  as well?

You cannot just have cassandra without a solr/elasticsearch install unless you 
are using DSE (the Datastax Enterprise with integrates with solr). Based on 
cursory reading over on JanusGraph, they do not recommend production 
deployments of JanusGraph with Cassandra embedded as the performance profiles 
are not entirely predictable. Again I would address this in the main 
documentation task.


> On March 4, 2018, 11:04 a.m., David Radley wrote:
> > distro/src/bin/atlas_start.py
> > Lines 121 (patched)
> > <https://reviews.apache.org/r/65885/diff/1/?file=1969113#file1969113line121>
> >
> > For an embedded hbase build, it will use the hbase zk and the embeded 
> > solr. I assume this line ( and the matching stop) should not be called for 
> > the embedded hbase build (which is not using the downloaded zk).

I could swear I tested this, and it required I spin up a zookeeper to get it to 
work. I just reset to master and retested (after fixing the atlas_config.py) 
and it worked without a local ZK start. So I'll remove this from the profile.


> On March 4, 2018, 11:04 a.m., David Radley wrote:
> > docs/src/site/twiki/InstallationSteps.twiki
> > Lines 38 (patched)
> > <https://reviews.apache.org/r/65885/diff/1/?file=1969119#file1969119line38>
> >
> > We did not have to do with before this change for embedded hbase solr. 
> > I think these notes should not apply to the embedded hbase solr profile.

Yep, as stated before I'll remove it, and find a way to only do it for a 
cassandra deploy.


- Pierre


-------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65885/#review198584
---


On March 2, 2018, 4:51 p.m., Pierre Padovani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65885/
> ---
> 
> (Updated March 2, 2018, 4:51 p.m.)
> 
> 
> Review request for atlas and David Radley.
> 
> 
> Bugs: ATLAS-2470
> https://issues.apache.org/jira/browse/ATLAS-2470
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This updates pom's to add the needed cassandra jars, and adds a dist profile 
> embedded-cassandra-solr.
> 
> 
> Diffs
> -
> 
>   distro/pom.xml 0103bef6 
>   distro/src/bin/atlas_config.py e6415cf4 
>   distro/src/bin/atlas_start.py 39be6b7c 
>   distro/src/bin/atlas_stop.py 66edd904 
>   distro/src/conf/cassandra.yml PRE-CREATION 
>   distro/src/conf/zookeeper/log4j.properties PRE-CREATION 
>   distro/src/conf/zookeeper/zoo.cfg PRE-CREATION 
>   distro/src/main/assemblies/standalone-package.xml 1881082e 
>   docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 
>   graphdb/janus/pom.xml 143b775f 
> 
> 
> Diff: https://reviews.apache.org/r/65885/diff/1/
> 
> 
> Testing
> ---
> 
> Full build with the new embedded-cassandra-solr, and testing to make sure 
> Atlas comes up and is functional.
> 
> Be aware that we have been running Cassandra backed JanusGraph for months 
> with no issues.
> 
> 
> Thanks,
> 
> Pierre Padovani
> 
>



Re: Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas

2018-03-04 Thread Pierre Padovani


> On March 4, 2018, 11:07 a.m., David Radley wrote:
> > My review comments are from my initial look at the code; I hope to try 
> > running this patch to verify it works for me

I'll update the patch with the above changes today or tomorrow as I have time.


- Pierre


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65885/#review198593
---


On March 2, 2018, 4:51 p.m., Pierre Padovani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65885/
> ---
> 
> (Updated March 2, 2018, 4:51 p.m.)
> 
> 
> Review request for atlas and David Radley.
> 
> 
> Bugs: ATLAS-2470
> https://issues.apache.org/jira/browse/ATLAS-2470
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This updates pom's to add the needed cassandra jars, and adds a dist profile 
> embedded-cassandra-solr.
> 
> 
> Diffs
> -
> 
>   distro/pom.xml 0103bef6 
>   distro/src/bin/atlas_config.py e6415cf4 
>   distro/src/bin/atlas_start.py 39be6b7c 
>   distro/src/bin/atlas_stop.py 66edd904 
>   distro/src/conf/cassandra.yml PRE-CREATION 
>   distro/src/conf/zookeeper/log4j.properties PRE-CREATION 
>   distro/src/conf/zookeeper/zoo.cfg PRE-CREATION 
>   distro/src/main/assemblies/standalone-package.xml 1881082e 
>   docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 
>   graphdb/janus/pom.xml 143b775f 
> 
> 
> Diff: https://reviews.apache.org/r/65885/diff/1/
> 
> 
> Testing
> ---
> 
> Full build with the new embedded-cassandra-solr, and testing to make sure 
> Atlas comes up and is functional.
> 
> Be aware that we have been running Cassandra backed JanusGraph for months 
> with no issues.
> 
> 
> Thanks,
> 
> Pierre Padovani
> 
>



Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas

2018-03-02 Thread Pierre Padovani

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65885/
---

Review request for atlas and David Radley.


Bugs: ATLAS-2470
https://issues.apache.org/jira/browse/ATLAS-2470


Repository: atlas


Description
---

This updates pom's to add the needed cassandra jars, and adds a dist profile 
embedded-cassandra-solr.


Diffs
-

  distro/pom.xml 0103bef6 
  distro/src/bin/atlas_config.py e6415cf4 
  distro/src/bin/atlas_start.py 39be6b7c 
  distro/src/bin/atlas_stop.py 66edd904 
  distro/src/conf/cassandra.yml PRE-CREATION 
  distro/src/conf/zookeeper/log4j.properties PRE-CREATION 
  distro/src/conf/zookeeper/zoo.cfg PRE-CREATION 
  distro/src/main/assemblies/standalone-package.xml 1881082e 
  docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 
  graphdb/janus/pom.xml 143b775f 


Diff: https://reviews.apache.org/r/65885/diff/1/


Testing
---

Full build with the new embedded-cassandra-solr, and testing to make sure Atlas 
comes up and is functional.

Be aware that we have been running Cassandra backed JanusGraph for months with 
no issues.


Thanks,

Pierre Padovani



[jira] [Updated] (ATLAS-2470) Basic support for Cassandra

2018-03-02 Thread Pierre Padovani (JIRA)

 [ 
https://issues.apache.org/jira/browse/ATLAS-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Padovani updated ATLAS-2470:
---
Attachment: ATLAS-2470.patch

> Basic support for Cassandra 
> 
>
> Key: ATLAS-2470
> URL: https://issues.apache.org/jira/browse/ATLAS-2470
> Project: Atlas
>  Issue Type: Sub-task
>    Reporter: Pierre Padovani
>    Assignee: Pierre Padovani
>Priority: Major
> Attachments: ATLAS-2470.patch
>
>
> Add the basic build support, and ability to run embedded Cassandra and solr 
> using a profile build:
> -Pdist,embedded-cassandra-solr



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (ATLAS-2473) Add Audit support w/o HBase

2018-03-01 Thread Pierre Padovani (JIRA)
Pierre Padovani created ATLAS-2473:
--

 Summary: Add Audit support w/o HBase
 Key: ATLAS-2473
 URL: https://issues.apache.org/jira/browse/ATLAS-2473
 Project: Atlas
  Issue Type: Sub-task
Reporter: Pierre Padovani


Currently you can only have no audit, or audit on hbase. Need to add an audit 
handler that can talk Cassandra or find another alternative.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (ATLAS-2470) Basic support for Cassandra

2018-03-01 Thread Pierre Padovani (JIRA)

 [ 
https://issues.apache.org/jira/browse/ATLAS-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Padovani reassigned ATLAS-2470:
--

Assignee: Pierre Padovani

> Basic support for Cassandra 
> 
>
> Key: ATLAS-2470
> URL: https://issues.apache.org/jira/browse/ATLAS-2470
> Project: Atlas
>  Issue Type: Sub-task
>    Reporter: Pierre Padovani
>    Assignee: Pierre Padovani
>Priority: Major
>
> Add the basic build support, and ability to run embedded Cassandra and solr 
> using a profile build:
> -Pdist,embedded-cassandra-solr



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


  1   2   >