[jira] [Created] (ATLAS-2044) InMemory filtering after index query

[Apoorv Naik (JIRA)]

Apoorv Naik created ATLAS-2044:
--

 Summary: InMemory filtering after index query
 Key: ATLAS-2044
 URL: https://issues.apache.org/jira/browse/ATLAS-2044
 Project: Atlas
  Issue Type: Bug
Reporter: Apoorv Naik
Assignee: Apoorv Naik


Currently the indexing and querying is done using a StandardTokenParser which 
tokenizes the string on punctuations, delimiters as a result the following test 
string will pass the equality match for the actual stored string

test string: user
stored val: /foo/bar/user or /foo/bar:user

This patch adds another layer of filtering after the index query which filters 
the vertices to ensure the correctness of results.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 61561: ATLAS-1218: Atlas says it is started but does not accept REST requests

[Richard Ding via Review Board]

> On Aug. 11, 2017, 11:11 a.m., David Radley wrote:
> > distro/src/bin/atlas_config.py
> > Lines 71 (patched)
> > 
> >
> > I am uncomfortable duplicating the default port values here. I suggest 
> > moving this logic to the java class that already has the default port values

The default ports are hard-coded in Java class:
```
appPort = StringUtils.isEmpty(enableTLSFlag) || enableTLSFlag.equals("true") ?
configuration.getInt(ATLAS_SERVER_HTTPS_PORT, 21443) :
configuration.getInt(ATLAS_SERVER_HTTP_PORT, 21000);
```
It's also difficult to access Java class from python code.


> On Aug. 11, 2017, 11:11 a.m., David Radley wrote:
> > distro/src/bin/atlas_config.py
> > Lines 454 (patched)
> > 
> >
> > is this going to work if -port was specified on the atlas_start command?

Good catch! The new patch handles -port option.


> On Aug. 11, 2017, 11:11 a.m., David Radley wrote:
> > distro/src/bin/atlas_config.py
> > Lines 473 (patched)
> > 
> >
> > I was hoping we would be able to be notified when Atlas was started or 
> > the startup time reduced, by understanding why there is a delay and 
> > minimizing it.
> > 
> > Once we understand that and decide that polling is the appropriate 
> > resolution - I suggest a 1 second poll - as we can do nothing with Atlas 
> > until it is  started.

Good point. Right now the atlas_start.py script just launches a process 
asynchronously without knowning when the Atlas server will complete the 
initialization (e.g. creating hbase tables, solr indexes and typedefs). This 
solution is to let the startup script ping the endpoint of the web server. 

I'll change the ping interval to 1 sec.


> On Aug. 11, 2017, 11:11 a.m., David Radley wrote:
> > distro/src/bin/atlas_config.py
> > Lines 475 (patched)
> > 
> >
> > I am not expert on all the ways Atlas can be started, does this work 
> > with OAuth and does Knox work with this?

The script just ping the Atlas server endpoint without performing the login. It 
should work with SSO solutions. It also assumes that the script runs on the 
same host as the Atlas server and is able to access directly to the server 
endpoint.


> On Aug. 11, 2017, 11:11 a.m., David Radley wrote:
> > distro/src/bin/atlas_config.py
> > Lines 477 (patched)
> > 
> >
> > If we have failures during startup, how can we tell  the polling to 
> > cease?

Just like _wait_for_shutdown_, the polling has a timeout setting (5 mins). The 
timeout is the only way to stop the polling. There is no callbacks from the 
server. Maybe we can try to tail and grep the log file to see there is any 
failures during startup.


- Richard


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61561/#review182676
---


On Aug. 10, 2017, 5:47 p.m., Richard Ding wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61561/
> ---
> 
> (Updated Aug. 10, 2017, 5:47 p.m.)
> 
> 
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-1218
> https://issues.apache.org/jira/browse/ATLAS-1218
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This patch implements a wait_for_startup function modeled after 
> wait_for_shutdown function.
> 
> 
> Diffs
> -
> 
>   distro/src/bin/atlas_config.py 1be9ca2b 
>   distro/src/bin/atlas_start.py a6a3455c 
> 
> 
> Diff: https://reviews.apache.org/r/61561/diff/1/
> 
> 
> Testing
> ---
> 
> Here are the output of atlas_start and atlas_stop script:
> ```
> starting atlas on port 21000
> 
> Apache Atlas Server started!!!
> ```
> and
> ```
> stopping atlas
> Apache Atlas Server stopped!!!
> ```
> 
> 
> Thanks,
> 
> Richard Ding
> 
>

[jira] [Commented] (ATLAS-1716) Export API is successfully exporting metadata when deleted entity is given as starting point.

[Ashutosh Mestry (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126235#comment-16126235
 ] 

Ashutosh Mestry commented on ATLAS-1716:


[~ayubkhan] This behavior is by design. We don't have plans to change this.

> Export API is successfully exporting metadata when deleted entity is given as 
> starting point.
> -
>
> Key: ATLAS-1716
> URL: https://issues.apache.org/jira/browse/ATLAS-1716
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 0.8-incubating
>Reporter: Ayub Pathan
>Assignee: Ashutosh Mestry
>Priority: Critical
> Fix For: 0.9-incubating
>
>
> export API should not honor the metadata export operation when deleted entity 
> is given as starting point. But this is not the case, the API is exporting 
> the deleted entity metadata as well. Is this a new requiremen?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Committer process

[David Radley]

Hi Madhan,
That makes sense. I will correct the wiki,thanks,  David

-- 
David Radley - Apache Atlas committer

From:Madhan Neethiraj 
To:David Radley 
Cc:"dev@atlas.apache.org" 
Date:14/08/2017 15:40
Subject:Re: Committer process
Sent by:Madhan Neethiraj 



David,



> It says that a commits should update the release log, but I notice that
this

> has not occurred recently. Is this still part of the process?

We stopped updating release log, as it caused merge issues while
cherry-picking commits to other branches (like 0.8 branch). And with
appropriate use of git while dealing with patches (like preserve author
information, commit message), maintaining release log file is not necessary.



Thanks,

Madhan





From: David Radley 
Date: Monday, August 14, 2017 at 4:58 AM
To: Madhan Neethiraj 
Cc: "dev@atlas.apache.org" 
Subject: Committer process



Hi Madhan,

I see that on the Apache Atlas wiki there are steps to commit a patch here:
https://cwiki.apache.org/confluence/display/ATLAS/Developer+Resources#DeveloperResources-CommitSteps

It says that a commits should update the release log, but I notice that
this has not occurred recently. Is this still part of the process?

Based on your previous suggestion:
https://cwiki.apache.org/confluence/display/ATLAS/Using+Git+with+Atlas
I assume the committer needs to issue a git commit --patch - so that the
original message and authorship are maintained. In this case, we do not
include the release log update.

I am looking to update the wiki with the current practices.


all the best,

David Radley - Apache Atlas committer

[jira] [Commented] (ATLAS-2012) Docker - image & hub - for Atlas

[Nigel Jones (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-2012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16125800#comment-16125800
 ] 

Nigel Jones commented on ATLAS-2012:


Hi Richard, 
 Thanks for posting that patch. I gave it a go under WIndows 10 x64 (with 
docker installed!) and the build fell over at :

Step 6/21 : RUN mkdir -p /usr/java
 ---> Running in b9952e21fdd7
 ---> 5b1b164a474c
Removing intermediate container b9952e21fdd7
Step 7/21 : RUN tar xzf *jdk* --strip-components 1 -C /usr/java
 ---> Running in 98080c6e748e
tar (child): *jdk*: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
The command '/bin/sh -c tar xzf *jdk* --strip-components 1 -C /usr/java' 
returned a non-zero code: 2

C:\src\atlas\dev-support\atlas-docker>dir
 Volume in drive C is Windows
 Volume Serial Number is CE3F-3B05

 Directory of C:\src\atlas\dev-support\atlas-docker

14/08/2017  13:56  .
14/08/2017  13:56  ..
14/08/2017  13:55 8,534,562 apache-maven-3.5.0-bin (1).tar.gz
14/08/2017  13:51 2,208 Dockerfile
14/08/2017  13:55   207,382,584 jdk-8u144-windows-x64 (1).exe
14/08/2017  13:51 2,857 README.md
   4 File(s)215,922,211 bytes
   2 Dir(s)  31,488,966,656 bytes free

Just taking a look now, but I also had another thought -- "openjdk" could be a 
better option to use instead of oracle java (pretty much the same anyway), and 
has the benefit of being installing via apt. Similarly maven3 is available at 
3.3.9 - at least with Ubuntu 16.10

I updated the patch to install jdk/maven via APT, and my build progressed well 
and completed just fine. I was able to login to the Atlas UI. I did get the 
hbase shell though didn't need it. Attaching my modified patch

> Docker - image & hub - for Atlas
> 
>
> Key: ATLAS-2012
> URL: https://issues.apache.org/jira/browse/ATLAS-2012
> Project: Atlas
>  Issue Type: New Feature
>Reporter: Nigel Jones
>Assignee: Péter Gergő Barna
> Attachments: atlas_docker.patch
>
>
> Docker is increasingly become a standard way of easily running components in 
> a flexible manner, whether for development, production, or test
> I feel there are a few things we can do with docker that will aid Atlas's 
> appeal
> 1. We could provide a simple example of how to create a docker image from the 
> Atlas build. This could be published on the wiki & it would make it easier 
> for developers to use Atlas within a docker environment - mostly by 
> addressing how to configure & start up
> 2. We could automatically generate a docker image as part of the build 
> process. This builds on #1 by automating the creation of the image & making 
> it "just part of the build"
> 3. We could publish the docker image from #2, for releases, to dockerhub, 
> making it near trivial for any developer to easily pull down and experiment 
> with Atlas. Full source of course would be provided/documented so that a user 
> could customize as needed for their environment plus of course improve what 
> is provided with the core project
> 4. We could work with other teams especially ranger, to allow similar efforts 
> in other teams to easily work together & be orchestrated 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 61568: ATLAS-2003: Add Javadoc format to class summaries

[David Radley]

> On Aug. 11, 2017, 9:22 a.m., David Radley wrote:
> > common/src/main/java/org/apache/atlas/security/InMemoryJAASConfiguration.java
> > Lines 54 (patched)
> > 
> >
> > Hi Richard, this looks good - please could you generate the swagger and 
> > make sure that it tolerates the markup you have added.
> 
> Richard Ding wrote:
> The following mvn command runs successfully after applying the patch: 
> >mvn -DskipTests -DskipEnunciate=false clean package 
> 
> The swagger docs are generated in the directory:
> >webapp/target/api/v2/apidocs

Hi Richard - I built it and it looks good. I will commit it once the build 
break has been fixed, all the best, David.


- David


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61568/#review182672
---


On Aug. 10, 2017, 6:15 p.m., Richard Ding wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61568/
> ---
> 
> (Updated Aug. 10, 2017, 6:15 p.m.)
> 
> 
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-2003
> https://issues.apache.org/jira/browse/ATLAS-2003
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This patch added Javadoc format to the following class summaries: 
> ```
> InMemoryJAASConfiguration.java 
> TitanGraphQuery.java   
> AtlasRelationshipDef.java  
> NotificationInterface.java 
> ExpandAndsOptimization.java 
> HBaseBasedAuditRepository.java   
> ```
> 
> 
> Diffs
> -
> 
>   
> common/src/main/java/org/apache/atlas/security/InMemoryJAASConfiguration.java 
> c65048fa 
>   
> graphdb/common/src/main/java/org/apache/atlas/repository/graphdb/titan/query/TitanGraphQuery.java
>  c38f6cc8 
>   intg/src/main/java/org/apache/atlas/model/typedef/AtlasRelationshipDef.java 
> c17e875d 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  ef8ee275 
>   
> repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandAndsOptimization.java
>  7cf97110 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
>  06e518ee 
> 
> 
> Diff: https://reviews.apache.org/r/61568/diff/1/
> 
> 
> Testing
> ---
> 
> Verified with Javadoc build.
> 
> 
> Thanks,
> 
> Richard Ding
> 
>

[jira] [Updated] (ATLAS-2038) Integration Test cases for AtlasAuthentication and Authorization Filter

[Nixon Rodrigues (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-2038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nixon Rodrigues updated ATLAS-2038:
---
Summary: Integration Test cases for AtlasAuthentication and Authorization 
Filter  (was: Unit Test cases for AtlasAuthentication and Authorization Filter)

> Integration Test cases for AtlasAuthentication and Authorization Filter
> ---
>
> Key: ATLAS-2038
> URL: https://issues.apache.org/jira/browse/ATLAS-2038
> Project: Atlas
>  Issue Type: Bug
>  Components: atlas-webui
>Affects Versions: 0.8-incubating
>Reporter: Nixon Rodrigues
>Assignee: Nixon Rodrigues
> Fix For: 0.9-incubating, 0.8.1-incubating
>
> Attachments: ATLAS-2038.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [ANNOUNCE] New Committer: David Radley

[Nigel Jones]

On 2017-08-13 18:37, Madhan Neethiraj  wrote: 

> Apache Atlas PMC has asked David Radley to become a committer and we are 
> pleased to announce that he has accepted.

Many congratulations David

[jira] [Updated] (ATLAS-2043) Do not allow an ClassificationDef update that adds entityTypes

[David Radley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-2043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Radley updated ATLAS-2043:

Description: Do not allow an ClassificationDef update that adds 
entityTypes. If we allowed this then we would need to do a lot of potentially 
non-performant queries to check that there are no classification instances that 
are not compatible with the restricted set of entityTypes  (was: Do not allow 
an ClassificaitonDef update that adds entityTypes. If we allowed this then we 
would need to do a lot of potentially non-performant queries to check that 
there are no classification instances that are not compatible with the 
restricted set of entityTypes)

> Do not allow an ClassificationDef update that adds entityTypes 
> ---
>
> Key: ATLAS-2043
> URL: https://issues.apache.org/jira/browse/ATLAS-2043
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: David Radley
>Assignee: David Radley
>
> Do not allow an ClassificationDef update that adds entityTypes. If we allowed 
> this then we would need to do a lot of potentially non-performant queries to 
> check that there are no classification instances that are not compatible with 
> the restricted set of entityTypes



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (ATLAS-2043) Do not allow an ClassificaitonDef update that adds entityTypes

[David Radley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-2043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Radley reassigned ATLAS-2043:
---

Assignee: David Radley

> Do not allow an ClassificaitonDef update that adds entityTypes 
> ---
>
> Key: ATLAS-2043
> URL: https://issues.apache.org/jira/browse/ATLAS-2043
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: David Radley
>Assignee: David Radley
>
> Do not allow an ClassificaitonDef update that adds entityTypes. If we allowed 
> this then we would need to do a lot of potentially non-performant queries to 
> check that there are no classification instances that are not compatible with 
> the restricted set of entityTypes



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (ATLAS-2043) Do not allow an ClassificaitonDef update that adds entityTypes

[David Radley (JIRA)]

David Radley created ATLAS-2043:
---

 Summary: Do not allow an ClassificaitonDef update that adds 
entityTypes 
 Key: ATLAS-2043
 URL: https://issues.apache.org/jira/browse/ATLAS-2043
 Project: Atlas
  Issue Type: Sub-task
Reporter: David Radley


Do not allow an ClassificaitonDef update that adds entityTypes. If we allowed 
this then we would need to do a lot of potentially non-performant queries to 
check that there are no classification instances that are not compatible with 
the restricted set of entityTypes



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)