Re: Seeking advice on Atlas XSS vulnerabilities

2020-06-15 Thread Bolke de Bruin
Hi Keval

I'm happy that you will tackle the UI part of it. However, these are not mere 
UI issues and need to be tackled server side IMHO. This opinion is supported by 
what the experts say[1]

Can you elaborate on how you will approach this? I have not seen a discussion. 

Cheers
Bolke

[1] 
https://security.stackexchange.com/questions/19524/current-best-practices-to-prevent-persistent-xss-attacks/19536


Sent from my iPhone

> On 15 Jun 2020, at 19:47, Keval Bhatt  wrote:
> 
> Hi Bolke,
> 
> Thanks for the details on the UI issues in rendering entity name and
> incorrect value storied for user-defined attributes. I am working on
> updating UI to address these issues, will send a patch shortly.
> 
> If you see further issues that require a server-side update, can you please
> add specific use case details?
> 
> Thanks
> 
>> On Fri, Jun 12, 2020 at 2:15 PM Bolke de Bruin  wrote:
>> 
>> Hi Madhan,
>> 
>> A second order XSS (I.e. stored) can be executed against Atlas quite
>> easily. If you create an entity with a name like
>> 
>> “>
>> 
>> You will get an alert when clicking “edit” and you will get artifacts all
>> over the place stemming from the closing ‘“>’. And there are probably other
>> areas where I can do this. Given the fact I can almost store arbitrary
>> sizes into Atlas it does not take a lot of imagination to think of a rogue
>> service adding an entity (or something else) with a malicious payload and
>> then just wait for a user to pick it up.
>> 
>> This is not an “XSS in the UI”. OWASP states that “ DOM Based XSS (or as
>> it is called in some texts, “type-0 XSS” or “CLIENT SIDE XSS”) is an XSS
>> attack wherein the attack payload is executed as a result of modifying the
>> DOM “environment” in the victim’s browser used by the original client side
>> script, so that the client side code runs in an “unexpected” manner. That
>> is, the page itself (the HTTP response that is) does not change, but the
>> client side code contained in the page executes differently due to the
>> malicious modifications that have occurred in the DOM environment. This is
>> in contrast to other XSS attacks (stored or reflected), wherein the attack
>> payload is placed in the response page (due to a server side flaw).
>> 
>> As shown above I was able to make the server return malicious code by
>> storing data and have it returned at a different time. This makes it a
>> Stored XSS. OWASP (
>> https://cheatsheetseries.owasp.org/cheatsheets/Cross_Site_Scripting_Prevention_Cheat_Sheet.html)
>> states that to prevent stored XSS it should be addressed server side.
>> 
>> Note also that the UI deals with escaping differently at different places.
>> For example the “user labels” are being escaped (but not properly
>> unescaped) on the client side. Entity definitions allow anything. Server
>> side validation is not done, so I can easily circumvent the UI and edit the
>> request.
>> 
>> I assume that you agree with the fact that bare HTML and JavaScript do not
>> really have a place in Atlas definitions or a need? Why not escape data on
>> output and make that configurable if really needed? And lets improve the
>> CSP headers while we are at it (this is being looked at as you mentioned)?
>> 
>> Cheers
>> Bolke
>> 
>> Verstuurd vanaf mijn iPad
>> 
>>>> Op 11 jun. 2020 om 02:30 heeft Madhan Neethiraj 
>> het volgende geschreven:
>>> Melinda,
>>> 
>>> As I said earlier, I don't agree on server side escaping HTML characters
>> in its input and output. Escaping must be handled by components that are
>> prone to issues in dealing with unescaped data. We are discussing potential
>> XSS in UI while dealing with data from user/server, which requires UI
>> updates to perform necessary escapes. As I said earlier, Keval is already
>> looking into this.
>>> 
>>> And, I haven't heard yet (from you/Bolke/anyone else) on any server side
>> issues that require Atlas server to escape HTML characters in data it
>> receives/responds. I strongly suggest to not treat this as a release
>> blocker. If anyone thinks it is, please come out with clear use
>> cases/examples quickly.
>>> 
>>> Regards,
>>> Madhan
>>> 
>>> 
>>> 
>>> On 6/10/20, 3:22 PM, "Melinda Crane"  wrote:
>>> 
>>>   Dear Madhan and Bolke,
>>> 
>>> 
>>>   Are there any updates on this topic? The discussion so far has been
>> pretty
>>>   exciting to see and hear - having the sanitization feature as an

Re: Seeking advice on Atlas XSS vulnerabilities

2020-06-12 Thread Bolke de Bruin
Hi Madhan,

A second order XSS (I.e. stored) can be executed against Atlas quite easily. If 
you create an entity with a name like

“>

You will get an alert when clicking “edit” and you will get artifacts all over 
the place stemming from the closing ‘“>’. And there are probably other areas 
where I can do this. Given the fact I can almost store arbitrary sizes into 
Atlas it does not take a lot of imagination to think of a rogue service adding 
an entity (or something else) with a malicious payload and then just wait for a 
user to pick it up.

This is not an “XSS in the UI”. OWASP states that “ DOM Based XSS (or as it is 
called in some texts, “type-0 XSS” or “CLIENT SIDE XSS”) is an XSS attack 
wherein the attack payload is executed as a result of modifying the DOM 
“environment” in the victim’s browser used by the original client side script, 
so that the client side code runs in an “unexpected” manner. That is, the page 
itself (the HTTP response that is) does not change, but the client side code 
contained in the page executes differently due to the malicious modifications 
that have occurred in the DOM environment. This is in contrast to other XSS 
attacks (stored or reflected), wherein the attack payload is placed in the 
response page (due to a server side flaw).

As shown above I was able to make the server return malicious code by storing 
data and have it returned at a different time. This makes it a Stored XSS. 
OWASP 
(https://cheatsheetseries.owasp.org/cheatsheets/Cross_Site_Scripting_Prevention_Cheat_Sheet.html)
 states that to prevent stored XSS it should be addressed server side. 

Note also that the UI deals with escaping differently at different places. For 
example the “user labels” are being escaped (but not properly unescaped) on the 
client side. Entity definitions allow anything. Server side validation is not 
done, so I can easily circumvent the UI and edit the request.

I assume that you agree with the fact that bare HTML and JavaScript do not 
really have a place in Atlas definitions or a need? Why not escape data on 
output and make that configurable if really needed? And lets improve the CSP 
headers while we are at it (this is being looked at as you mentioned)?

Cheers
Bolke

Verstuurd vanaf mijn iPad

>> Op 11 jun. 2020 om 02:30 heeft Madhan Neethiraj  het 
>> volgende geschreven:
> Melinda,
> 
> As I said earlier, I don't agree on server side escaping HTML characters in 
> its input and output. Escaping must be handled by components that are prone 
> to issues in dealing with unescaped data. We are discussing potential XSS in 
> UI while dealing with data from user/server, which requires UI updates to 
> perform necessary escapes. As I said earlier, Keval is already looking into 
> this.
> 
> And, I haven't heard yet (from you/Bolke/anyone else) on any server side 
> issues that require Atlas server to escape HTML characters in data it 
> receives/responds. I strongly suggest to not treat this as a release blocker. 
> If anyone thinks it is, please come out with clear use cases/examples quickly.
> 
> Regards,
> Madhan
> 
> 
> 
> On 6/10/20, 3:22 PM, "Melinda Crane"  wrote:
> 
>Dear Madhan and Bolke,
> 
> 
>Are there any updates on this topic? The discussion so far has been pretty
>exciting to see and hear - having the sanitization feature as an optional
>config like Bolke mentioned would be real useful to us (since we know our
>data should never contain html characters), and it seems like it would be
>useful for other companies down the road who have stricter security
>requirements. Where the sanitization would happen is of course completely
>up to you folks!
> 
> 
>Cheers,
> 
>Melinda Crane
> 
>>On Tue, Jun 9, 2020 at 10:46 AM Madhan Neethiraj  
>> wrote:
>> 
>> Bolke,
>> 
>>>1. Filtering input on arrival where the user cannot manipulate it
>> anymore (i.e. server)
>> It's not clear what you expect the server to do here? Should HTML
>> characters be escaped in all inputs, before saving the data in its store?
>> Can you give few examples of server-side manipulation due to unescaped HTML
>> characters?
>> 
>>>2. Encode data on output
>> I think asking the server side to escape all HTML characters in its
>> response is a very bad idea. Consider forcing such a requirement on a RDBMS
>> - wouldn't this mandate every client to un-escape every column value
>> returned for queries? Can you add references to applications that perform
>> HTML escape of all its REST API responses?.
>> 
>>>3. Set correct Http content headers (e.g. application/json)
>> This is already being done. Do you find this missing for any specific
>> usecases?
>> 
>&g

Re: Seeking advice on Atlas XSS vulnerabilities

2020-06-11 Thread Bolke de Bruin
I'm working on a POC that shows the need for server side escaping due to second 
order XSS (stored XSS).

That's taking a bit of time. 

B.

Sent from my iPhone

> On 11 Jun 2020, at 02:30, Madhan Neethiraj  wrote:
> 
> Melinda,
> 
> As I said earlier, I don't agree on server side escaping HTML characters in 
> its input and output. Escaping must be handled by components that are prone 
> to issues in dealing with unescaped data. We are discussing potential XSS in 
> UI while dealing with data from user/server, which requires UI updates to 
> perform necessary escapes. As I said earlier, Keval is already looking into 
> this.
> 
> And, I haven't heard yet (from you/Bolke/anyone else) on any server side 
> issues that require Atlas server to escape HTML characters in data it 
> receives/responds. I strongly suggest to not treat this as a release blocker. 
> If anyone thinks it is, please come out with clear use cases/examples quickly.
> 
> Regards,
> Madhan
> 
> 
> 
> On 6/10/20, 3:22 PM, "Melinda Crane"  wrote:
> 
>Dear Madhan and Bolke,
> 
> 
>Are there any updates on this topic? The discussion so far has been pretty
>exciting to see and hear - having the sanitization feature as an optional
>config like Bolke mentioned would be real useful to us (since we know our
>data should never contain html characters), and it seems like it would be
>useful for other companies down the road who have stricter security
>requirements. Where the sanitization would happen is of course completely
>up to you folks!
> 
> 
>Cheers,
> 
>Melinda Crane
> 
>>On Tue, Jun 9, 2020 at 10:46 AM Madhan Neethiraj  
>> wrote:
>> 
>> Bolke,
>> 
>>>1. Filtering input on arrival where the user cannot manipulate it
>> anymore (i.e. server)
>> It's not clear what you expect the server to do here? Should HTML
>> characters be escaped in all inputs, before saving the data in its store?
>> Can you give few examples of server-side manipulation due to unescaped HTML
>> characters?
>> 
>>>2. Encode data on output
>> I think asking the server side to escape all HTML characters in its
>> response is a very bad idea. Consider forcing such a requirement on a RDBMS
>> - wouldn't this mandate every client to un-escape every column value
>> returned for queries? Can you add references to applications that perform
>> HTML escape of all its REST API responses?.
>> 
>>>3. Set correct Http content headers (e.g. application/json)
>> This is already being done. Do you find this missing for any specific
>> usecases?
>> 
>>>4. Set correct Content Security Policy.
>> This is being looked into.
>> 
>> Regards,
>> Madhan
>> 
>> 
>> On 6/9/20, 12:49 AM, "Bolke de Bruin"  wrote:
>> 
>>Hi Madhan,
>> 
>>I don’t think solving this on the front-end is sufficient and the
>> defense should be more in-depth. The front-end is after all client side and
>> can be manipulated by the user. Typically one solves XSS vulnerabilities by:
>> 
>>1. Filtering input on arrival where the user cannot manipulate it
>> anymore (i.e. server)
>>2. Encode data on output
>>3. Set correct Http content headers (e.g. application/json)
>>4. Set correct Content Security Policy.
>> 
>>Your proposed solution does not fit any of those criteria. If you are
>> worried about compatibility I suggest making it a server side configuration
>> variable (e.g. atlas.server.insecure_escaping) which should default to
>> “false” imho and alternative could be to enable configurable escaping but
>> that’s more complex.
>> 
>>Note that in this case also #4 CSP headers seem to be very loose
>> (unsafe-inline, unsafe-eval) which add to the attack surface:
>> 
>> 
>> https://github.com/apache/atlas/blob/master/webapp/src/main/java/org/apache/atlas/web/filters/HeadersUtil.java#L44
>> 
>>Regards,
>>Bolke
>> 
>>Verstuurd vanaf mijn iPad
>> 
>>> Op 9 jun. 2020 om 06:04 heeft Madhan Neethiraj 
>> het volgende geschreven:
>>> 
>>> Melinda,
>>> 
>>> Thank you for adding details. I think the frontend must take steps
>> to prevent JavaScript execution while handling values in REST API responses.
>>> 
>>> Server side escaping all HTML characters in all responses wouldn’t
>> be the right approach, as browser is not the only consumer of the
>> responses. Consider applications that parse response from Atlas REST APIs;
>> all

Re: [VOTE] Release Apache Atlas version 2.1.0 - rc1

2020-06-09 Thread Bolke de Bruin
Hi Madhan,

These are indeed related to the issues Melinda brought up and which are now 
confirmed.

It think that warrants another RC with those items fixed.

Cheers
Bolke

Verstuurd vanaf mijn iPad

> Op 8 jun. 2020 om 20:41 heeft Madhan Neethiraj  het 
> volgende geschreven:
> 
> Bolke,
> 
> Can you send specific security concerns that need to addressed in 2.1.0 
> release? If this is about questions from Melinda Crane, I asked for Melinda 
> details to better understand the questions. I suggest we know the details 
> before treating them as release blockers.
> 
> Thanks,
> Madhan
> 
> On 6/8/20, 9:56 AM, "Bolke de Bruin"  wrote:
> 
>This is my -1 (non-binding) which I will revert to +1 when the security 
> concerns have been addressed/clarified. 
> 
>Given that Atlas has pretty slow releases I think it smart to address them 
> now. 
> 
>Cheers
>Bolke
> 
>Sent from my iPhone
> 
>> On 8 Jun 2020, at 18:18, Sharmadha Sainath  
>> wrote:
>> 
>> +1 for Apache Atlas 2.1 RC1
>> 
>> 
>>  - Validated MD5 and SHA512 checksum of source.
>>  - Brought up Atlas using embedded HBase and Solr.
>>  - Verified DSL , Basic Search , Quick search and Suggestions
>>  - Verified Basic search using system attributes filters (multiple
>>  classification search , labels , custom attributes , business metadata
>>  search)
>>  - Added labels , user defined properties (custom attributes) , business
>>  metadata to entities and verified the entity properties and audits.
>>  - Purged Soft Deleted entity and verified purge audits.
>>  - Saved the Favorite searches and executed them.
>> 
>> Thanks,
>> Sharmadha S.
>> 
>> 
>> 
>>> On Mon, Jun 8, 2020 at 6:23 PM Nixon Rodrigues <
>>> nixon.rodrig...@freestoneinfotech.com> wrote:
>>> 
>>> Thanks Madhan for putting 2.1 RC1 together for the vote.
>>> 
>>> +1 for Apache Atlas 2.1 RC1
>>> 
>>> Validated the following items.
>>> 
>>>  1. verified the md5/sha checksum of source bundle.
>>>  2. packaging with -Pdist,embedded-hbase-solr is good.
>>>  3. Unit testcase passes.
>>>  4. Run quickstart and all the types and entities created are loading
>>>  correctly on the UI.
>>>  5. Create tags, glossary, term and its apply correctly on entities.
>>>  6. Created business metadata with attributes and applied it to entities.
>>>  7. tested basic and quick search flow.
>>>  8. Verified statistics page details
>>> 
>>> 
>>> Thanks & Regards
>>> Nixon
>>> 
>>> 
>>> 
>>> On Mon, Jun 8, 2020 at 11:17 AM Sarath Subramanian 
>>> wrote:
>>> 
>>>> +1 for Apache Atlas 2.1 RC1
>>>> 
>>>> Validated the following:
>>>> 
>>>>  - No binaries in source
>>>>  - Verified source checksum - md5 and sha512
>>>>  - Verified signature
>>>>  - Build was successful on the source (mvn clean -DskipTests package
>>>>  -Pdist,embedded-hbase-solr)
>>>>  - Ran quickStart and validated types and entities were created
>>>>  successfully.
>>>>  - Validated tag propagation works
>>>>  - Added/removed labels to an entity and verified audit is getting
>>>>  updated correctly
>>>>  - Added user-defined properties and business metadata attributes.
>>>>  - Deleted few entities and purged those entities - validated if purge
>>>>  audit recorded correctly.
>>>>  - Basic sanity tests - basic search (with system attributes), advanced
>>>>  search, quick search.
>>>> 
>>>> 
>>>> 
>>>> Thanks,
>>>> Sarath
>>>> 
>>>> 
>>>> 
>>>> On Sat, Jun 6, 2020 at 10:22 AM Madhan Neethiraj 
>>>> wrote:
>>>> 
>>>>> (resending with plain-text format)
>>>>> 
>>>>> Atlas team,
>>>>> 
>>>>> Apache Atlas 2.1.0 rc1, with following fixes since rc0, is now
>>> available
>>>>> for vote within dev community.
>>>>> 
>>>>>   ATLAS-3770: UI(Classic): Active and Deleted hyperlinks for certain
>>>>> entities throwing error on click
>>>>>   ATLAS-3766: Stats modal not close issue #2
>>>>>   ATLAS-3674: ZipFileMigationImporter: Migration status display fix.
>>>>> Part 2
>>>>> 
>>>>> Links to the release artifa

Re: Seeking advice on Atlas XSS vulnerabilities

2020-06-09 Thread Bolke de Bruin
Hi Madhan,

I don’t think solving this on the front-end is sufficient and the defense 
should be more in-depth. The front-end is after all client side and can be 
manipulated by the user. Typically one solves XSS vulnerabilities by:

1. Filtering input on arrival where the user cannot manipulate it anymore (i.e. 
server)
2. Encode data on output
3. Set correct Http content headers (e.g. application/json)
4. Set correct Content Security Policy.

Your proposed solution does not fit any of those criteria. If you are worried 
about compatibility I suggest making it a server side configuration variable 
(e.g. atlas.server.insecure_escaping) which should default to “false” imho and 
alternative could be to enable configurable escaping but that’s more complex.

Note that in this case also #4 CSP headers seem to be very loose 
(unsafe-inline, unsafe-eval) which add to the attack surface:  

https://github.com/apache/atlas/blob/master/webapp/src/main/java/org/apache/atlas/web/filters/HeadersUtil.java#L44

Regards,
Bolke

Verstuurd vanaf mijn iPad

> Op 9 jun. 2020 om 06:04 heeft Madhan Neethiraj  het 
> volgende geschreven:
> 
> Melinda,
> 
> Thank you for adding details. I think the frontend must take steps to prevent 
> JavaScript execution while handling values in REST API responses.
> 
> Server side escaping all HTML characters in all responses wouldn’t be the 
> right approach, as browser is not the only consumer of the responses. 
> Consider applications that parse response from Atlas REST APIs; all such 
> applications must be updated to deal with escaped HTML characters in 
> responses. This will likely break existing Atlas integrations.
> 
> Hope this helps.
> 
> Regards,
> Madhan
> 
> On 6/8/20, 6:31 PM, "Melinda Crane"  wrote:
> 
>Apologies, looks like the second half of the email I just sent got
>formatted weirdly (I was copying from a failed earlier email attempt that
>got rejected for attaching images). Just to make sure it doesn’t get 
> buried:
> 
>Of course this specific case can be patched on the frontend, but doing some
>kind of HTML sanitization escaping into the whatever-is-handling-the-ajax
>(EntityREST or AtlasEntityStoreV2, maybe?) to blanket-catch HTML characters
>sounds safer.
> 
>Hope this helps express my concerns.
>From, Melinda Crane
> 
> 
>>On Mon, Jun 8, 2020 at 9:13 PM Melinda Crane  wrote:
>> 
>> Dear Madhan,
>> 
>> Thank you kindly for your response and for offering to look into potential
>> solutions. I've been trying to step through how the ajax calls are handled,
>> I don't really see the AtlasJsonProvider being used much at all actually
>> for that. My concern is that the data from the ajax calls aren't being
>> sanitized on the backend. I see there is a ton of sanitization happening on
>> the frontend (lots of calls to _.escape()), but they're not all captured.
>> For example.
>> 
>> In CommonViewFunction, in the propertyTable
>> 
>>  function,
>> URLs are not escaped. So I can create an entity that has a key value of
>> 
>> https://www.google.com/search?\;>> height='\''1'\'' onload='\''window.alert(\"boo\");'\''>
>> 
>> and my script will successfully get successfully injected into my browser
>> when I try to view it. Looks like I can't attach images here but here's the
>> elements:
>> 
>> 
>>
>>> class="hide-empty-value">
>>
>>newSchema
>>
>>
>>N/A
>>
>>
>>
>>
>>classField
>>
>>
>>> href="https://www.google.com/search?;>
>>> height="1" onload="window.alert(boo);">
>>"https://www.google.com/search
>> ?"
>>> height="1" onload="window.alert(boo);">
>>...
>> 
>> ^You can see the iframe-script combo here.
>> 
>> Of course this specific case can be patched on the frontend, but doing
>> some kind of HTML sanitization escaping on the
>> whatever-is-handling-the-ajax (EntityREST or AtlasEntityStoreV2, maybe?) to
>> blanket-catch HTML characters sounds safer.
>> 
>> Hope this helps express my concerns.
>> From, Melinda Crane
>> 
>> 
>>> On Sat, Jun 6, 2020 at 2:48 PM Madhan Neethiraj  wrote:
>>> 
>>> Melinda,
>>> 
>>> Thank you for reaching out to Apache Atlas community.
>>> 
>>> As you noted, AtlasJsonProvider is used to deserialize/serialize REST API
>>> requests and responses. In addition, methods in AtlasJson are used in to
>>> convert to/from Json. It will help 

Re: [VOTE] Release Apache Atlas version 2.1.0 - rc1

2020-06-08 Thread Bolke de Bruin
This is my -1 (non-binding) which I will revert to +1 when the security 
concerns have been addressed/clarified. 

Given that Atlas has pretty slow releases I think it smart to address them now. 

Cheers
Bolke

Sent from my iPhone

> On 8 Jun 2020, at 18:18, Sharmadha Sainath  
> wrote:
> 
> +1 for Apache Atlas 2.1 RC1
> 
> 
>   - Validated MD5 and SHA512 checksum of source.
>   - Brought up Atlas using embedded HBase and Solr.
>   - Verified DSL , Basic Search , Quick search and Suggestions
>   - Verified Basic search using system attributes filters (multiple
>   classification search , labels , custom attributes , business metadata
>   search)
>   - Added labels , user defined properties (custom attributes) , business
>   metadata to entities and verified the entity properties and audits.
>   - Purged Soft Deleted entity and verified purge audits.
>   - Saved the Favorite searches and executed them.
> 
> Thanks,
> Sharmadha S.
> 
> 
> 
>> On Mon, Jun 8, 2020 at 6:23 PM Nixon Rodrigues <
>> nixon.rodrig...@freestoneinfotech.com> wrote:
>> 
>> Thanks Madhan for putting 2.1 RC1 together for the vote.
>> 
>> +1 for Apache Atlas 2.1 RC1
>> 
>> Validated the following items.
>> 
>>   1. verified the md5/sha checksum of source bundle.
>>   2. packaging with -Pdist,embedded-hbase-solr is good.
>>   3. Unit testcase passes.
>>   4. Run quickstart and all the types and entities created are loading
>>   correctly on the UI.
>>   5. Create tags, glossary, term and its apply correctly on entities.
>>   6. Created business metadata with attributes and applied it to entities.
>>   7. tested basic and quick search flow.
>>   8. Verified statistics page details
>> 
>> 
>> Thanks & Regards
>> Nixon
>> 
>> 
>> 
>> On Mon, Jun 8, 2020 at 11:17 AM Sarath Subramanian 
>> wrote:
>> 
>>> +1 for Apache Atlas 2.1 RC1
>>> 
>>> Validated the following:
>>> 
>>>   - No binaries in source
>>>   - Verified source checksum - md5 and sha512
>>>   - Verified signature
>>>   - Build was successful on the source (mvn clean -DskipTests package
>>>   -Pdist,embedded-hbase-solr)
>>>   - Ran quickStart and validated types and entities were created
>>>   successfully.
>>>   - Validated tag propagation works
>>>   - Added/removed labels to an entity and verified audit is getting
>>>   updated correctly
>>>   - Added user-defined properties and business metadata attributes.
>>>   - Deleted few entities and purged those entities - validated if purge
>>>   audit recorded correctly.
>>>   - Basic sanity tests - basic search (with system attributes), advanced
>>>   search, quick search.
>>> 
>>> 
>>> 
>>> Thanks,
>>> Sarath
>>> 
>>> 
>>> 
>>> On Sat, Jun 6, 2020 at 10:22 AM Madhan Neethiraj 
>>> wrote:
>>> 
 (resending with plain-text format)
 
 Atlas team,
 
 Apache Atlas 2.1.0 rc1, with following fixes since rc0, is now
>> available
 for vote within dev community.
 
ATLAS-3770: UI(Classic): Active and Deleted hyperlinks for certain
 entities throwing error on click
ATLAS-3766: Stats modal not close issue #2
ATLAS-3674: ZipFileMigationImporter: Migration status display fix.
 Part 2
 
 Links to the release artifacts are given below. Please review and vote.
 
 The vote will be open for at least 72 hours or until necessary votes
>> are
 reached.
  [ ] +1 Approve
  [ ] +0 No opinion
  [ ] -1 Disapprove (and reason why)
 
 Thanks,
 Madhan
 
 
 List of all issues addressed in this release:
 https://issues.apache.org/jira/issues/?jql=project=ATLAS AND
 status=Resolved AND fixVersion=2.1.0 ORDER BY key DESC
 
 Git tag for the release:
 https://github.com/apache/atlas/tree/release-2.1.0-rc1
 Sources for the release:
 
>>> 
>> https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc1/apache-atlas-2.1.0-sources.tar.gz
 
 Source release verification:
  PGP Signature:
 
>>> 
>> https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc1/apache-atlas-2.1.0-sources.tar.gz.asc
  SHA512 Hash:
 
>>> 
>> https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc1/apache-atlas-2.1.0-sources.tar.gz.sha512
  MD5 Hash:
 
>>> 
>> https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc1/apache-atlas-2.1.0-sources.tar.gz.md5
 
 Keys to verify the signature of the release artifacts are available at:
 https://dist.apache.org/repos/dist/release/atlas/KEYS
 
 New features/enhancements:
  - Quick Search: provides a simpler search experience with type-ahead
 suggestions
  - Business Metadata: enables augmenting entity-types with additional
 attributes, search entities using these attributes
  - Labels: ability to add/remove labels on entities, and search
>> entities
 using labels
  - Custom Attributes: ability to add entity instance specific custom
 attributes i.e. attributes not defined in entity-def or business
>> metadata
  - Entity Purge: added REST APIs to 

Re: [VOTE] Release Apache Atlas version 2.1.0 - rc1

2020-06-06 Thread Bolke de Bruin
Hi Madhan,

Melinda Crane of Snapchat raised some concerns over XSS issues that have gone 
unanswered.

Particularly:

1. the CSP allows unsafe-inline and unsafe-eval
2. the backend JSON content provider doesn't appear to do any sort of force 
escaping on HTML sensitive characters
webapp/src/main/java/org/apache/atlas/web/filters/HeadersUtil.java:44

I think it would be great those concerns could be addressed or debunked before 
release?

Cheers
Bolke

Sent from my iPhone

> On 6 Jun 2020, at 19:22, Madhan Neethiraj  wrote:
> 
> (resending with plain-text format)
> 
> Atlas team,
>  
> Apache Atlas 2.1.0 rc1, with following fixes since rc0, is now available for 
> vote within dev community.
>  
> ATLAS-3770: UI(Classic): Active and Deleted hyperlinks for certain 
> entities throwing error on click
> ATLAS-3766: Stats modal not close issue #2
> ATLAS-3674: ZipFileMigationImporter: Migration status display fix. Part 2
>  
> Links to the release artifacts are given below. Please review and vote. 
>  
> The vote will be open for at least 72 hours or until necessary votes are 
> reached.
>   [ ] +1 Approve
>   [ ] +0 No opinion
>   [ ] -1 Disapprove (and reason why)
>  
> Thanks,
> Madhan
>  
>  
> List of all issues addressed in this release: 
> https://issues.apache.org/jira/issues/?jql=project=ATLAS AND status=Resolved 
> AND fixVersion=2.1.0 ORDER BY key DESC
>  
> Git tag for the release: 
> https://github.com/apache/atlas/tree/release-2.1.0-rc1
> Sources for the release: 
> https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc1/apache-atlas-2.1.0-sources.tar.gz
>  
> Source release verification:
>   PGP Signature: 
> https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc1/apache-atlas-2.1.0-sources.tar.gz.asc
>   SHA512 Hash:   
> https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc1/apache-atlas-2.1.0-sources.tar.gz.sha512
>   MD5 Hash:  
> https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc1/apache-atlas-2.1.0-sources.tar.gz.md5
>  
> Keys to verify the signature of the release artifacts are available at: 
> https://dist.apache.org/repos/dist/release/atlas/KEYS
>  
> New features/enhancements:
>   - Quick Search: provides a simpler search experience with type-ahead 
> suggestions
>   - Business Metadata: enables augmenting entity-types with additional 
> attributes, search entities using these attributes
>   - Labels: ability to add/remove labels on entities, and search entities 
> using labels
>   - Custom Attributes: ability to add entity instance specific custom 
> attributes i.e. attributes not defined in entity-def or business metadata
>   - Entity Purge: added REST APIs to purge deleted entities
>  
> Enhancements:
>   - Search: ability to find entities by more than one classification
>   - Performance: improvements in lineage retrieval and 
> classification-propagation
>   - Notification: ability to process notifications from multiple Kafka topics
>   - Hive Hook: tracks process-executions via hive_process_execution entities
>   - Hive Hook: captures DDL operations via hive_db_ddl and hive_table_ddl 
> entities
>   - Notification: introduced shell entities to record references to 
> non-existing entities in notifications
>   - Spark: added model to capture Spark entities, processes and relationships
>   - AWS S3: introduced updated model to capture AWS S3 entities and 
> relationships
>   - ADLS-Gen2: introduced model to capture Azure Data Lake Storage Gen2 
> entities and relationships
>   - Dependencies: JanusGraph 0.5.1, Tinkerpop 3.4.6, Spring Framework 4.3.20
>   - Authorization: updated to cover new features, like: business metadata, 
> labels, purge
>   - UI: multiple UI improvements, including a beta UI
>  
> 
> 
> 
> 
> 


Re: Apache Atlas 2.1 release

2020-05-22 Thread Bolke de Bruin
Awesome! Do you have any idea what is in scope or out scope of this release? Is 
it a branch of master or an update of 2.0 for example with cherry picks from 
master?

Sent from my iPhone

> On 17 May 2020, at 16:32, Nixon Rodrigues 
>  wrote:
> 
> Madhan,
> 
> Thanks for volunteering this release, the release plan looks good to me.
> +1 for the release
> 
> Regards
> Nixon
> 
> 
>> On Sun, May 17, 2020 at 11:47 AM Madhan Neethiraj  wrote:
>> 
>> Atlas community,
>> 
>> 
>> 
>> In the past 3 months since I sent the email on Apache Atlas 2.1 release,
>> the community has added 100+ commits to improve various areas like search,
>> UI, performance, authorization, bulk import of glossary &
>> business-metadata, support for ADLS-Ge2 and AWS-S3 entity types, updated
>> JanusGraph version. Thanks to everyone who contributed to Apache Atlas -
>> making it a feature rich, enterprise-ready, open-source governance tool
>> backed by a thriving community.
>> 
>> 
>> 
>> Let’s started on Apache Atlas 2.1 release! I volunteer to be the
>> release-manager for this release and propose the following timeline:
>> 05/18, Mon: branch-2.0 goes into lockdown mode to prepare for 2.1 release.
>> During lockdown period, only commits for release-blocker issues must be
>> merged in branch-2.0. Master branch will remain open for all commits
>> 05/18 – 05/22, Mon – Fri: community to validate branch-2.0 builds and
>> address release-blocker issues
>> 05/25, Mon: release candidate out for vote by the community
>> 05/29, Fri: release Apache Atlas 2.1, assuming successful completion of
>> votes
>> 
>> 
>> Please review and let me know your comments/suggestions.
>> 
>> 
>> 
>> Thanks,
>> 
>> Madhan
>> 
>> 
>> 
>> From: Madhan Neethiraj 
>> Date: Tuesday, February 18, 2020 at 4:10 PM
>> To: "dev@atlas.apache.org" , "u...@atlas.apache.org"
>> 
>> Subject: Apache Atlas 2.1 release
>> 
>> 
>> 
>> Atlas community,
>> 
>> 
>> 
>> Over past months the dev community has been busy in enhancing Apache Atlas
>> with new features, improvements and fixes. Here are few
>> features/enhancements since last major release, Apache Atlas 2.0:
>> 
>>- added quick-search feature, to provide a simpler search experience
>> with type-ahead suggestions
>> 
>>- introduced Namespaces feature, which allows grouping of attributes
>> to be applied to multiple entity-types
>> 
>>- introduced labels on entity instances, and search for entities using
>> the label
>> 
>>- enhancement to support entity instance specific custom attributes
>> 
>>- enhanced search to find entities by more than one classification
>> 
>>- introduced shell/incomplete entities to handle notifications
>> referencing entities that don’t (yet) exist in Atlas
>> 
>>- added REST APIs to purge deleted entities
>> 
>>- performance improvements in lineage retrieval and tag-propagation
>> 
>>- updated Atlas server to process notifications from multiple Kafka
>> topics
>> 
>>- updated Hive hook to track process executions, via
>> hive_process_execution entities
>> 
>>- updated Hive hook to capture DDL operations, via hive_db_ddl and
>> hive_table_ddl entities
>> 
>>- added models for Spark; introduced new models for AWS S3
>> 
>>- updated versions of dependent libraries/components: JanusGraph,
>> Jackson parser, Spring Framework,
>> 
>>- updated authorization model to cover new features/APIs, like
>> add/remove labels, purge entities, update namespace attributes
>> 
>> 
>> 
>> With significant improvements in place, it is time for the next
>> maintenance release of Apache Atlas!
>> 
>> 
>> 
>> I propose to release Apache Atlas 2.1 by early next month. Please review
>> and send your comments.
>> 
>> 
>> 
>> Regards,
>> 
>> Madhan
>> 
>> 


Friendly request for review on Atlas patches

2020-05-06 Thread Bolke de Bruin
Hi Guys,

My team has been creating quite a few patches for Atlas that are pending in
the review board or have pending discussions in Jira. Would you mind
reviewing them or provide feedback? I do think they are relevant for the
larger community:

ATLAS-3776 <https://issues.apache.org/jira/browse/ATLAS-3776> -
EntitySearchProcessor fails on queries with a sort by attribute
https://reviews.apache.org/r/72459/ - patch available including unit tests

ATLAS-3654 - Allow Atlas to run in standalone http mode
https://reviews.apache.org/r/72441/ - patch available

ATLAS-3758 <https://issues.apache.org/jira/browse/ATLAS-3758> -
Support sort parameters for FreeTextSearchProcessor
https://reviews.apache.org/r/72440/ - patch available

ATLAS-3755 <https://reviews.apache.org/r/72438/bugs/ATLAS-3755/> -
Support setting system attributes if policy allows it
https://reviews.apache.org/r/72438/ - under discussion, patch available

We observe that there is only a small number of committers active. Can
we do anything to help?

Kind regards

Bolke


--
Bolke de Bruin
bdbr...@gmail.com


Re: Review Request 72440: Support sort params for FreeTextSearchProcessor

2020-05-06 Thread Bolke de Bruin

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72440/#review220648
---



Can you please update the tests?

- Bolke de Bruin


On April 29, 2020, 10:26 a.m., Damian Warszawski wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72440/
> ---
> 
> (Updated April 29, 2020, 10:26 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Bolke de Bruin, Madhan Neethiraj, 
> and Sarath Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> No way to sort results by specified attribute while freetext search is 
> enabled. In our case we would like to enforce ordering by introducing custom 
> attribute definition e.g. popularity score from 
> https://github.com/dwarszawski/amundsen-atlas-types/blob/master/amundsenatlastypes/schema/01_2_table_schema.json
> 
> 
> Reference to jira https://issues.apache.org/jira/browse/ATLAS-3758
> Patched applied against master branch.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java
>  1a7bf6b16 
>   
> repository/src/main/java/org/apache/atlas/discovery/FreeTextSearchProcessor.java
>  92b5eb4d2 
>   repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 
> 11eb7ca49 
> 
> 
> Diff: https://reviews.apache.org/r/72440/diff/1/
> 
> 
> Testing
> ---
> 
> Patch was applied on our dev env with custom entity definitions and 
> successfully verified if order is applied as specified in the search query.
> 
> 
> Thanks,
> 
> Damian Warszawski
> 
>



[jira] [Comment Edited] (ATLAS-3755) Allow system attributes to be updated when policy allows

2020-05-03 Thread Bolke de Bruin (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098286#comment-17098286
 ] 

Bolke de Bruin edited comment on ATLAS-3755 at 5/3/20, 8:20 AM:


[~madhan] I have made a couple of updates:

1. I removed the defaults in the AtlasEntity to ensure that at json 
deserialisation no defaults are set. This was a change very limited in scope as 
the deserialiser uses the default constructor and the default constructor isn't 
used anywhere else. All other constructors use "init()" that do set the default 
values.
2. The update of the system attributes is now behind a feature flag 
"atlas.store.system_attribute.enable". This defaults to false and disables 
updating/creation of system attributes if not in "import" mode.
3. Access requests are done in one go having virtually no impact on processing 
time
4. Unit tests have been updated to verify the feature flags
5. Removed the side effect in preCreateOrUpdate that was setting system 
attributes, all this is now contained in createOrUpdate

*Use case*
We intend to receive about 0.5 million entity updates per day from other meta 
data systems aside from the ~2 million events per day we receive. We use Kafka 
to handle the backpressure hence the need to have it managed by policy. Also 
Glossary (which relies on the entity system) needs to expose homeId and other 
system attributes as well as Glossary items can be created other meta data 
systems. This needs to be enabled in a follow PR.


was (Author: bolke):
[~madhan] I have made a couple of updates:

1. I removed the defaults in the AtlasEntity to ensure that at json 
deserialisation no defaults are set. This was a change very limited in scope as 
the deserialiser uses the default constructor and the default constructor isn't 
used anywhere else. All other constructors use "init()" that do set the default 
values.
2. The update of the system attributes is now behind a feature flag 
"atlas.store.system_attribute.enable". This defaults to false and disables 
updating/creation of system attributes if not in "import" mode.
3. Access requests are done in one go having virtually no impact on processing 
time
4. Unit tests have been updated to verify the feature flags
5. Removed the side effect in preCreateOrUpdate that was setting system 
attributes, all this is now contained in createOrUpdate

*Use case*
We intend to receive about 0.5 million entity updates per day from other meta 
data systems. We use Kafka to handle the backpressure hence the need to have it 
managed by policy. Also Glossary (which relies on the entity system) needs to 
expose homeId and other system attributes as well as Glossary items can be 
created other meta data systems. This needs to be enabled in a follow PR.

> Allow system attributes to be updated when policy allows
> 
>
> Key: ATLAS-3755
> URL: https://issues.apache.org/jira/browse/ATLAS-3755
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Bolke de Bruin
>Assignee: Bolke de Bruin
>Priority: Critical
> Attachments: 
> 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, 
> 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, 
> 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, 
> feature.patch, feature.patch
>
>
> Atlas does not operate in a isolated environment, this is one of the reasons 
> the "homeId" system attribute was introduced. Unfortunately system attributes 
> can only be updated when importing. This means any integration with other 
> services is significantly limited (Kafka, Rest API will not work). (See also 
> ATLAS-3754)
> To resolve this I propose to make it possible to update the system attributes 
> when policy allows it. This introduces new 
> AtlasPrivilege.ENTITY_UPDATE_SYSTEM_ATTRIBUTE and 
> AtlasPrivilege.ENTITY_CREATE_SYSTEM_ATTRIBUTE next to 
> AtlasPrivilege.ENTITY_UPDATE_ATTRIBUTE and 
> AtlasPrivilege.ENTITY_CREATE_ATTRIBUTE rather than just checking on the 
> entity level. In certain places we will then drop the requirement for an 
> import to be active as this can now happen through other channels as well.
> This allows operators to specify policies that allow granular controls over 
> attributes and system attributes.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (ATLAS-3755) Allow system attributes to be updated when policy allows

2020-05-03 Thread Bolke de Bruin (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098286#comment-17098286
 ] 

Bolke de Bruin commented on ATLAS-3755:
---

[~madhan] I have made a couple of updates:

1. I removed the defaults in the AtlasEntity to ensure that at json 
deserialisation no defaults are set. This was a change very limited in scope as 
the deserialiser uses the default constructor and the default constructor isn't 
used anywhere else. All other constructors use "init()" that do set the default 
values.
2. The update of the system attributes is now behind a feature flag 
"atlas.store.system_attribute.enable". This defaults to false and disables 
updating/creation of system attributes if not in "import" mode.
3. Access requests are done in one go having virtually no impact on processing 
time
4. Unit tests have been updated to verify the feature flags
5. Removed the side effect in preCreateOrUpdate that was setting system 
attributes, all this is now contained in createOrUpdate

*Use case*
We intend to receive about 0.5 million entity updates per day from other meta 
data systems. We use Kafka to handle the backpressure hence the need to have it 
managed by policy. Also Glossary (which relies on the entity system) needs to 
expose homeId and other system attributes as well as Glossary items can be 
created other meta data systems. This needs to be enabled in a follow PR.

> Allow system attributes to be updated when policy allows
> 
>
> Key: ATLAS-3755
> URL: https://issues.apache.org/jira/browse/ATLAS-3755
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>    Affects Versions: 2.0.0, 2.1.0
>        Reporter: Bolke de Bruin
>Assignee: Bolke de Bruin
>Priority: Critical
> Attachments: 
> 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, 
> 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, 
> 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, 
> feature.patch, feature.patch
>
>
> Atlas does not operate in a isolated environment, this is one of the reasons 
> the "homeId" system attribute was introduced. Unfortunately system attributes 
> can only be updated when importing. This means any integration with other 
> services is significantly limited (Kafka, Rest API will not work). (See also 
> ATLAS-3754)
> To resolve this I propose to make it possible to update the system attributes 
> when policy allows it. This introduces new 
> AtlasPrivilege.ENTITY_UPDATE_SYSTEM_ATTRIBUTE and 
> AtlasPrivilege.ENTITY_CREATE_SYSTEM_ATTRIBUTE next to 
> AtlasPrivilege.ENTITY_UPDATE_ATTRIBUTE and 
> AtlasPrivilege.ENTITY_CREATE_ATTRIBUTE rather than just checking on the 
> entity level. In certain places we will then drop the requirement for an 
> import to be active as this can now happen through other channels as well.
> This allows operators to specify policies that allow granular controls over 
> attributes and system attributes.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3755) Allow system attributes to be updated when policy allows

2020-05-03 Thread Bolke de Bruin (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin updated ATLAS-3755:
--
Attachment: feature.patch

> Allow system attributes to be updated when policy allows
> 
>
> Key: ATLAS-3755
> URL: https://issues.apache.org/jira/browse/ATLAS-3755
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0, 2.1.0
>    Reporter: Bolke de Bruin
>    Assignee: Bolke de Bruin
>Priority: Critical
> Attachments: 
> 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, 
> 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, 
> 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, 
> feature.patch, feature.patch
>
>
> Atlas does not operate in a isolated environment, this is one of the reasons 
> the "homeId" system attribute was introduced. Unfortunately system attributes 
> can only be updated when importing. This means any integration with other 
> services is significantly limited (Kafka, Rest API will not work). (See also 
> ATLAS-3754)
> To resolve this I propose to make it possible to update the system attributes 
> when policy allows it. This introduces new 
> AtlasPrivilege.ENTITY_UPDATE_SYSTEM_ATTRIBUTE and 
> AtlasPrivilege.ENTITY_CREATE_SYSTEM_ATTRIBUTE next to 
> AtlasPrivilege.ENTITY_UPDATE_ATTRIBUTE and 
> AtlasPrivilege.ENTITY_CREATE_ATTRIBUTE rather than just checking on the 
> entity level. In certain places we will then drop the requirement for an 
> import to be active as this can now happen through other channels as well.
> This allows operators to specify policies that allow granular controls over 
> attributes and system attributes.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: Review Request 72438: Allow system attributes to be updated when policy allows

2020-05-03 Thread Bolke de Bruin

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72438/
---

(Updated May 3, 2020, 7:43 a.m.)


Review request for atlas, Ashutosh Mestry, Bolke de Bruin, Madhan Neethiraj, 
Nixon Rodrigues, and Sarath Subramanian.


Changes
---

Improved unit tests
Made into feature flag


Bugs: ATLAS-3755
https://issues.apache.org/jira/browse/ATLAS-3755


Repository: atlas


Description
---

Atlas does not operate in a isolated environment, this is one of the reasons 
the "homeId" system attribute was introduced. Unfortunately system attributes 
can only be updated when importing. This means any integration with other 
services is significantly limited (Kafka, Rest API will not work). (See also 
ATLAS-3754)
To resolve this I propose to make it possible to update the system attributes 
when policy allows it. This introduces new 
AtlasPrivilege.ENTITY_UPDATE_SYSTEM_ATTRIBUTE and 
AtlasPrivilege.ENTITY_CREATE_SYSTEM_ATTRIBUTE next to 
AtlasPrivilege.ENTITY_UPDATE_ATTRIBUTE and 
AtlasPrivilege.ENTITY_CREATE_ATTRIBUTE rather than just checking on the entity 
level. In certain places we will then drop the requirement for an import to be 
active as this can now happen through other channels as well.
This allows operators to specify policies that allow granular controls over 
attributes and system attributes.


Diffs (updated)
-

  
authorization/src/main/java/org/apache/atlas/authorize/AtlasEntityAccessRequest.java
 6d49d54b1 
  
authorization/src/main/java/org/apache/atlas/authorize/simple/AtlasSimpleAuthorizer.java
 734991691 
  
authorization/src/main/java/org/apache/atlas/authorize/simple/AtlasSimpleAuthzPolicy.java
 d19112885 
  authorization/src/main/resources/atlas-simple-authz-policy.json 6b2001279 
  intg/src/main/java/org/apache/atlas/ApplicationProperties.java 1f1f3771b 
  intg/src/main/java/org/apache/atlas/model/instance/AtlasEntity.java 4d8c94894 
  intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 3962c3c42 
  intg/src/main/java/org/apache/atlas/type/Constants.java 3fc13056e 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
 379150b7b 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
 36bee301d 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityMutationContext.java
 deb743eea 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/IDBasedEntityResolver.java
 3b9694851 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java
 38228a8ec 


Diff: https://reviews.apache.org/r/72438/diff/5/

Changes: https://reviews.apache.org/r/72438/diff/4-5/


Testing
---

- Manually tested
- Unit test updated


Thanks,

Bolke de Bruin



Re: Review Request 72459: EntitySearchProcessor is failing on graph query with sortBy attribute.

2020-05-02 Thread Bolke de Bruin

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72459/#review220587
---




repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java
Line 203 (original), 203 (patched)
<https://reviews.apache.org/r/72459/#comment309064>

can you add a test please?


- Bolke de Bruin


On May 1, 2020, 10:01 p.m., Damian Warszawski wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72459/
> ---
> 
> (Updated May 1, 2020, 10:01 p.m.)
> 
> 
> Review request for atlas, Bolke de Bruin, Madhan Neethiraj, and Nixon 
> Rodrigues.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> EntitySearchProcessor fails when doing search by classification and specify 
> orderBy attribute. The issue is that for graph query you cannot refer to 
> attribute by name but need to provide absolute path to entity attribute e.g. 
> 
>  
> 
> ```
> 
> { "attributes": [ "description", "comment", "popularityScore" ], 
> "classification": "customer_NON_PII", "excludeDeletedEntities": "False", 
> "limit": "", "offset": 100, "sortBy": "Table.popularityScore", "sortOrder": 
> "DESCENDING", "typeName": "hive_table" }
> ```
> 
> this query fails with following exception:
> 
>  
> 
> ```
> 
> {"exception":{"message":"Provided key does not exist: 
> hive_table.popularityScore","class":"java.lang.IllegalArgumentException","stacktrace":"java.lang.IllegalArgumentException:
>  Provided key does not exist: hive_table.popularityScore\n\tat 
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:163)\n\tat
>  org.janusgraph.graphdb.query.graph.GraphCentricQueryBuilder.
> orderBy(GraphCentricQueryBuilder.java:160)
> 
> ```
> 
>  
> 
> When specify full reference to attribute e.g. 
> 
>  
> 
> ```
> 
> { "attributes": [ "description", "comment", "popularityScore" ], 
> "classification": "customer_NON_PII", "excludeDeletedEntities": "False", 
> "limit": "", "offset": 100, "sortBy": "Table.popularityScore", "sortOrder": 
> "DESCENDING", "typeName": "hive_table" }
> ```
> 
> it fails on validation stage
> 
>  
> 
> ```
> 
> {"exception":{"message":"Attribute Table.popularityScore not found for type 
> Table","class":"org.apache.atlas.exception.AtlasBaseException","stacktrace":"org.apache.atlas.exception.AtlasBaseException:
>  Attribute Table.popularityScore not found for type Table\n\tat 
> org.apache.atlas.discovery.SearchContext.validateAttributes(SearchContext.java:288)
> 
> ```
> 
> Reference to JIRA https://issues.apache.org/jira/browse/ATLAS-3776
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java
>  1a7bf6b16 
> 
> 
> Diff: https://reviews.apache.org/r/72459/diff/1/
> 
> 
> Testing
> ---
> 
> tested on our dev env.
> 
> 
> Thanks,
> 
> Damian Warszawski
> 
>



[jira] [Commented] (ATLAS-3755) Allow system attributes to be updated when policy allows

2020-04-30 Thread Bolke de Bruin (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17096435#comment-17096435
 ] 

Bolke de Bruin commented on ATLAS-3755:
---

[~madhan] I have updated the patch to only use `entity-update` and 
`entity-create` and do the access request in one go. This makes the access 
request backwards incompatible and it would require an update to the Ranger 
plugin. I suggest putting the check against the attributes behind a feature 
flag so it remains backwards compatible and it would not affect anyone who 
isn't using access controls on attributes or system attributes. 

let me know what you think.

> Allow system attributes to be updated when policy allows
> 
>
> Key: ATLAS-3755
> URL: https://issues.apache.org/jira/browse/ATLAS-3755
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0, 2.1.0
>    Reporter: Bolke de Bruin
>    Assignee: Bolke de Bruin
>Priority: Critical
> Attachments: 
> 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, 
> 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, 
> 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, feature.patch
>
>
> Atlas does not operate in a isolated environment, this is one of the reasons 
> the "homeId" system attribute was introduced. Unfortunately system attributes 
> can only be updated when importing. This means any integration with other 
> services is significantly limited (Kafka, Rest API will not work). (See also 
> ATLAS-3754)
> To resolve this I propose to make it possible to update the system attributes 
> when policy allows it. This introduces new 
> AtlasPrivilege.ENTITY_UPDATE_SYSTEM_ATTRIBUTE and 
> AtlasPrivilege.ENTITY_CREATE_SYSTEM_ATTRIBUTE next to 
> AtlasPrivilege.ENTITY_UPDATE_ATTRIBUTE and 
> AtlasPrivilege.ENTITY_CREATE_ATTRIBUTE rather than just checking on the 
> entity level. In certain places we will then drop the requirement for an 
> import to be active as this can now happen through other channels as well.
> This allows operators to specify policies that allow granular controls over 
> attributes and system attributes.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3755) Allow system attributes to be updated when policy allows

2020-04-30 Thread Bolke de Bruin (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin updated ATLAS-3755:
--
Attachment: feature.patch

> Allow system attributes to be updated when policy allows
> 
>
> Key: ATLAS-3755
> URL: https://issues.apache.org/jira/browse/ATLAS-3755
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0, 2.1.0
>    Reporter: Bolke de Bruin
>    Assignee: Bolke de Bruin
>Priority: Critical
> Attachments: 
> 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, 
> 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, 
> 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, feature.patch
>
>
> Atlas does not operate in a isolated environment, this is one of the reasons 
> the "homeId" system attribute was introduced. Unfortunately system attributes 
> can only be updated when importing. This means any integration with other 
> services is significantly limited (Kafka, Rest API will not work). (See also 
> ATLAS-3754)
> To resolve this I propose to make it possible to update the system attributes 
> when policy allows it. This introduces new 
> AtlasPrivilege.ENTITY_UPDATE_SYSTEM_ATTRIBUTE and 
> AtlasPrivilege.ENTITY_CREATE_SYSTEM_ATTRIBUTE next to 
> AtlasPrivilege.ENTITY_UPDATE_ATTRIBUTE and 
> AtlasPrivilege.ENTITY_CREATE_ATTRIBUTE rather than just checking on the 
> entity level. In certain places we will then drop the requirement for an 
> import to be active as this can now happen through other channels as well.
> This allows operators to specify policies that allow granular controls over 
> attributes and system attributes.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: Review Request 72438: Allow system attributes to be updated when policy allows

2020-04-30 Thread Bolke de Bruin

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72438/
---

(Updated April 30, 2020, 11:17 a.m.)


Review request for atlas, Ashutosh Mestry, Bolke de Bruin, Madhan Neethiraj, 
Nixon Rodrigues, and Sarath Subramanian.


Changes
---

- Simplified access request model


Bugs: ATLAS-3755
https://issues.apache.org/jira/browse/ATLAS-3755


Repository: atlas


Description
---

Atlas does not operate in a isolated environment, this is one of the reasons 
the "homeId" system attribute was introduced. Unfortunately system attributes 
can only be updated when importing. This means any integration with other 
services is significantly limited (Kafka, Rest API will not work). (See also 
ATLAS-3754)
To resolve this I propose to make it possible to update the system attributes 
when policy allows it. This introduces new 
AtlasPrivilege.ENTITY_UPDATE_SYSTEM_ATTRIBUTE and 
AtlasPrivilege.ENTITY_CREATE_SYSTEM_ATTRIBUTE next to 
AtlasPrivilege.ENTITY_UPDATE_ATTRIBUTE and 
AtlasPrivilege.ENTITY_CREATE_ATTRIBUTE rather than just checking on the entity 
level. In certain places we will then drop the requirement for an import to be 
active as this can now happen through other channels as well.
This allows operators to specify policies that allow granular controls over 
attributes and system attributes.


Diffs (updated)
-

  
authorization/src/main/java/org/apache/atlas/authorize/AtlasEntityAccessRequest.java
 6d49d54b1 
  
authorization/src/main/java/org/apache/atlas/authorize/simple/AtlasSimpleAuthorizer.java
 734991691 
  
authorization/src/main/java/org/apache/atlas/authorize/simple/AtlasSimpleAuthzPolicy.java
 d19112885 
  authorization/src/main/resources/atlas-simple-authz-policy.json 6b2001279 
  intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 3962c3c42 
  intg/src/main/java/org/apache/atlas/type/Constants.java 3fc13056e 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
 379150b7b 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
 36bee301d 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityMutationContext.java
 deb743eea 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/IDBasedEntityResolver.java
 3b9694851 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java
 38228a8ec 


Diff: https://reviews.apache.org/r/72438/diff/4/

Changes: https://reviews.apache.org/r/72438/diff/3-4/


Testing
---

- Manually tested
- Unit test updated


Thanks,

Bolke de Bruin



[jira] [Comment Edited] (ATLAS-3755) Allow system attributes to be updated when policy allows

2020-04-29 Thread Bolke de Bruin (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17095795#comment-17095795
 ] 

Bolke de Bruin edited comment on ATLAS-3755 at 4/29/20, 7:09 PM:
-

[~madhan] I don't think that would work for the KafkaConsumer, or does it? 
Please note that also Glossary needs to be updated to contain all system 
attributes as it is missing at least "homeId". In addition I don't think the 
risk is that high that system attributes get updated inadvertent. The default 
authorization model denies access to these attributes. Next to that it would 
require the incoming message to include those system properties. In this case 
you could argue that it should not be consumed at all if not allowed as it is a 
bad behaving client. We would like to integrate Atlas with another metadata 
system, which would actually make the occurrence much more frequent in around 
50% of the updates. Given the fact that system attributes are part of the 
vertex there is not much additional cost as far as I can see in doing this 
during entity updates.

On your second point. I'm not strongly bound to them so I can merge them. I do 
think there might be cases that you would like to allow an update but disallow 
a create.

Authorization per attribute allows end-users to edit a particular attribute 
(say description) without allowing editing of all properties of the entities. 
This is actually a very common use case as you would like users to be able to 
enrich the metadata without adjusting some core attributes or system generated 
attributes. I understand your point about performance. What I could do is to 
submit an ArrayList and create a RangerCollectionResourceMatcher that requires 
all items in the submitted array to be matches (as opposed to 
RangerDefaultResourceMatcher) that should resolve the issue of CPU cycles and 
audit logs.

What do you think?


was (Author: bolke):
[~madhan] I don't think that would work for the KafkaConsumer, or does it? 
Please note that also Glossary needs to be updated to contain all system 
attributes as it is missing at least "homeId". In addition I don't think the 
risk is that high that system attributes get updated inadvertent. The default 
authorization model denies access to these attributes. Next to that it would 
require the incoming message to include those system properties. In this case 
you could argue that it should not be consumed at all if not allowed as it is a 
bad behaving client. We would like to integrate Atlas with another metadata 
system, which would actually make the occurrence much more frequent in around 
50% of the updates. Given the fact that system attributes are part of the 
vertex there is not much additional cost as far as I can see in doing this 
during entity updates.

On your second point. I'm not strongly bound to them so I can merge them. I do 
think there might be cases that you would like to allow an update but disallow 
a create.

Authorization per attribute allows end-users to edit a particular attribute 
(say description) without allowing editing of all properties of the entities. 
This is actually a very common use case as you would like users to be able to 
enrich the metadata without adjusting some core attributes or system generated 
attributes. I understand your point about performance. What I could do is to 
submit an Array in string format (e.g. "attribute1;attribute2;attribute4") and 
create a RangerArrayResourceMatcher that allows matching on 1 item which should 
resolve the issue of CPU cycles and audit logs.

What do you think?

> Allow system attributes to be updated when policy allows
> 
>
> Key: ATLAS-3755
> URL: https://issues.apache.org/jira/browse/ATLAS-3755
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>    Affects Versions: 2.0.0, 2.1.0
>        Reporter: Bolke de Bruin
>Assignee: Bolke de Bruin
>Priority: Critical
> Attachments: 
> 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, 
> 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, 
> 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch
>
>
> Atlas does not operate in a isolated environment, this is one of the reasons 
> the "homeId" system attribute was introduced. Unfortunately system attributes 
> can only be updated when importing. This means any integration with other 
> services is significantly limited (Kafka, Rest API will not work). (See also 
> ATLAS-3754)
> To resolve this I propose to make it possible to update the system attributes 
> when policy allows it. This introduces new 
> AtlasPrivilege.ENTITY_UPDATE_S

[jira] [Commented] (ATLAS-3755) Allow system attributes to be updated when policy allows

2020-04-29 Thread Bolke de Bruin (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17095795#comment-17095795
 ] 

Bolke de Bruin commented on ATLAS-3755:
---

[~madhan] I don't think that would work for the KafkaConsumer, or does it? 
Please note that also Glossary needs to be updated to contain all system 
attributes as it is missing at least "homeId". In addition I don't think the 
risk is that high that system attributes get updated inadvertent. The default 
authorization model denies access to these attributes. Next to that it would 
require the incoming message to include those system properties. In this case 
you could argue that it should not be consumed at all if not allowed as it is a 
bad behaving client. We would like to integrate Atlas with another metadata 
system, which would actually make the occurrence much more frequent in around 
50% of the updates. Given the fact that system attributes are part of the 
vertex there is not much additional cost as far as I can see in doing this 
during entity updates.

On your second point. I'm not strongly bound so I can merge them. I do think 
there might be cases that you would like to allow an update but disallow a 
create.

Authorization per attribute allows end-users to edit a particular attribute 
(say description) without allowing editing of all properties of the entities. 
This is actually a very common use case as you would like users to be able to 
enrich the metadata without adjusting some core attributes or system generated 
attributes. I understand your point about performance. What I could do is to 
submit an Array in string format (e.g. "attribute1;attribute2;attribute4") and 
create a RangerArrayResourceMatcher that allows matching on 1 item which should 
resolve the issue of CPU cycles and audit logs.

What do you think?

> Allow system attributes to be updated when policy allows
> 
>
> Key: ATLAS-3755
> URL: https://issues.apache.org/jira/browse/ATLAS-3755
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0, 2.1.0
>    Reporter: Bolke de Bruin
>Assignee: Bolke de Bruin
>Priority: Critical
> Attachments: 
> 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, 
> 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, 
> 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch
>
>
> Atlas does not operate in a isolated environment, this is one of the reasons 
> the "homeId" system attribute was introduced. Unfortunately system attributes 
> can only be updated when importing. This means any integration with other 
> services is significantly limited (Kafka, Rest API will not work). (See also 
> ATLAS-3754)
> To resolve this I propose to make it possible to update the system attributes 
> when policy allows it. This introduces new 
> AtlasPrivilege.ENTITY_UPDATE_SYSTEM_ATTRIBUTE and 
> AtlasPrivilege.ENTITY_CREATE_SYSTEM_ATTRIBUTE next to 
> AtlasPrivilege.ENTITY_UPDATE_ATTRIBUTE and 
> AtlasPrivilege.ENTITY_CREATE_ATTRIBUTE rather than just checking on the 
> entity level. In certain places we will then drop the requirement for an 
> import to be active as this can now happen through other channels as well.
> This allows operators to specify policies that allow granular controls over 
> attributes and system attributes.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Comment Edited] (ATLAS-3755) Allow system attributes to be updated when policy allows

2020-04-29 Thread Bolke de Bruin (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17095795#comment-17095795
 ] 

Bolke de Bruin edited comment on ATLAS-3755 at 4/29/20, 6:51 PM:
-

[~madhan] I don't think that would work for the KafkaConsumer, or does it? 
Please note that also Glossary needs to be updated to contain all system 
attributes as it is missing at least "homeId". In addition I don't think the 
risk is that high that system attributes get updated inadvertent. The default 
authorization model denies access to these attributes. Next to that it would 
require the incoming message to include those system properties. In this case 
you could argue that it should not be consumed at all if not allowed as it is a 
bad behaving client. We would like to integrate Atlas with another metadata 
system, which would actually make the occurrence much more frequent in around 
50% of the updates. Given the fact that system attributes are part of the 
vertex there is not much additional cost as far as I can see in doing this 
during entity updates.

On your second point. I'm not strongly bound to them so I can merge them. I do 
think there might be cases that you would like to allow an update but disallow 
a create.

Authorization per attribute allows end-users to edit a particular attribute 
(say description) without allowing editing of all properties of the entities. 
This is actually a very common use case as you would like users to be able to 
enrich the metadata without adjusting some core attributes or system generated 
attributes. I understand your point about performance. What I could do is to 
submit an Array in string format (e.g. "attribute1;attribute2;attribute4") and 
create a RangerArrayResourceMatcher that allows matching on 1 item which should 
resolve the issue of CPU cycles and audit logs.

What do you think?


was (Author: bolke):
[~madhan] I don't think that would work for the KafkaConsumer, or does it? 
Please note that also Glossary needs to be updated to contain all system 
attributes as it is missing at least "homeId". In addition I don't think the 
risk is that high that system attributes get updated inadvertent. The default 
authorization model denies access to these attributes. Next to that it would 
require the incoming message to include those system properties. In this case 
you could argue that it should not be consumed at all if not allowed as it is a 
bad behaving client. We would like to integrate Atlas with another metadata 
system, which would actually make the occurrence much more frequent in around 
50% of the updates. Given the fact that system attributes are part of the 
vertex there is not much additional cost as far as I can see in doing this 
during entity updates.

On your second point. I'm not strongly bound so I can merge them. I do think 
there might be cases that you would like to allow an update but disallow a 
create.

Authorization per attribute allows end-users to edit a particular attribute 
(say description) without allowing editing of all properties of the entities. 
This is actually a very common use case as you would like users to be able to 
enrich the metadata without adjusting some core attributes or system generated 
attributes. I understand your point about performance. What I could do is to 
submit an Array in string format (e.g. "attribute1;attribute2;attribute4") and 
create a RangerArrayResourceMatcher that allows matching on 1 item which should 
resolve the issue of CPU cycles and audit logs.

What do you think?

> Allow system attributes to be updated when policy allows
> 
>
> Key: ATLAS-3755
> URL: https://issues.apache.org/jira/browse/ATLAS-3755
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Bolke de Bruin
>Assignee: Bolke de Bruin
>Priority: Critical
> Attachments: 
> 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, 
> 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, 
> 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch
>
>
> Atlas does not operate in a isolated environment, this is one of the reasons 
> the "homeId" system attribute was introduced. Unfortunately system attributes 
> can only be updated when importing. This means any integration with other 
> services is significantly limited (Kafka, Rest API will not work). (See also 
> ATLAS-3754)
> To resolve this I propose to make it possible to update the system attributes 
> when policy allows it. This introduces new 
> AtlasPrivilege.ENTIT

[jira] [Updated] (ATLAS-3755) Allow system attributes to be updated when policy allows

2020-04-29 Thread Bolke de Bruin (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin updated ATLAS-3755:
--
Attachment: 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch

> Allow system attributes to be updated when policy allows
> 
>
> Key: ATLAS-3755
> URL: https://issues.apache.org/jira/browse/ATLAS-3755
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0, 2.1.0
>    Reporter: Bolke de Bruin
>    Assignee: Bolke de Bruin
>Priority: Critical
> Attachments: 
> 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, 
> 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, 
> 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch
>
>
> Atlas does not operate in a isolated environment, this is one of the reasons 
> the "homeId" system attribute was introduced. Unfortunately system attributes 
> can only be updated when importing. This means any integration with other 
> services is significantly limited (Kafka, Rest API will not work). (See also 
> ATLAS-3754)
> To resolve this I propose to make it possible to update the system attributes 
> when policy allows it. This introduces new 
> AtlasPrivilege.ENTITY_UPDATE_SYSTEM_ATTRIBUTE and 
> AtlasPrivilege.ENTITY_CREATE_SYSTEM_ATTRIBUTE next to 
> AtlasPrivilege.ENTITY_UPDATE_ATTRIBUTE and 
> AtlasPrivilege.ENTITY_CREATE_ATTRIBUTE rather than just checking on the 
> entity level. In certain places we will then drop the requirement for an 
> import to be active as this can now happen through other channels as well.
> This allows operators to specify policies that allow granular controls over 
> attributes and system attributes.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: Review Request 72438: Allow system attributes to be updated when policy allows

2020-04-29 Thread Bolke de Bruin

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72438/
---

(Updated April 29, 2020, 9:49 a.m.)


Review request for atlas, Ashutosh Mestry, Bolke de Bruin, Madhan Neethiraj, 
Nixon Rodrigues, and Sarath Subramanian.


Changes
---

Fix some minor issues


Bugs: ATLAS-3755
https://issues.apache.org/jira/browse/ATLAS-3755


Repository: atlas


Description
---

Atlas does not operate in a isolated environment, this is one of the reasons 
the "homeId" system attribute was introduced. Unfortunately system attributes 
can only be updated when importing. This means any integration with other 
services is significantly limited (Kafka, Rest API will not work). (See also 
ATLAS-3754)
To resolve this I propose to make it possible to update the system attributes 
when policy allows it. This introduces new 
AtlasPrivilege.ENTITY_UPDATE_SYSTEM_ATTRIBUTE and 
AtlasPrivilege.ENTITY_CREATE_SYSTEM_ATTRIBUTE next to 
AtlasPrivilege.ENTITY_UPDATE_ATTRIBUTE and 
AtlasPrivilege.ENTITY_CREATE_ATTRIBUTE rather than just checking on the entity 
level. In certain places we will then drop the requirement for an import to be 
active as this can now happen through other channels as well.
This allows operators to specify policies that allow granular controls over 
attributes and system attributes.


Diffs (updated)
-

  authorization/src/main/java/org/apache/atlas/authorize/AtlasPrivilege.java 
7287b3dd7 
  authorization/src/main/resources/atlas-simple-authz-policy.json 6b2001279 
  intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 3962c3c42 
  intg/src/main/java/org/apache/atlas/type/Constants.java 3fc13056e 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
 379150b7b 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
 36bee301d 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityMutationContext.java
 deb743eea 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/IDBasedEntityResolver.java
 3b9694851 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java
 38228a8ec 


Diff: https://reviews.apache.org/r/72438/diff/3/

Changes: https://reviews.apache.org/r/72438/diff/2-3/


Testing
---

- Manually tested
- Unit test updated


Thanks,

Bolke de Bruin



Re: Review Request 72438: Allow system attributes to be updated when policy allows

2020-04-27 Thread Bolke de Bruin

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72438/
---

(Updated April 27, 2020, 5:25 p.m.)


Review request for atlas, Ashutosh Mestry, Bolke de Bruin, Madhan Neethiraj, 
Nixon Rodrigues, and Sarath Subramanian.


Changes
---

Unit tests added


Bugs: ATLAS-3755
https://issues.apache.org/jira/browse/ATLAS-3755


Repository: atlas


Description
---

Atlas does not operate in a isolated environment, this is one of the reasons 
the "homeId" system attribute was introduced. Unfortunately system attributes 
can only be updated when importing. This means any integration with other 
services is significantly limited (Kafka, Rest API will not work). (See also 
ATLAS-3754)
To resolve this I propose to make it possible to update the system attributes 
when policy allows it. This introduces new 
AtlasPrivilege.ENTITY_UPDATE_SYSTEM_ATTRIBUTE and 
AtlasPrivilege.ENTITY_CREATE_SYSTEM_ATTRIBUTE next to 
AtlasPrivilege.ENTITY_UPDATE_ATTRIBUTE and 
AtlasPrivilege.ENTITY_CREATE_ATTRIBUTE rather than just checking on the entity 
level. In certain places we will then drop the requirement for an import to be 
active as this can now happen through other channels as well.
This allows operators to specify policies that allow granular controls over 
attributes and system attributes.


Diffs (updated)
-

  authorization/src/main/java/org/apache/atlas/authorize/AtlasPrivilege.java 
7287b3dd7 
  authorization/src/main/resources/atlas-simple-authz-policy.json 6b2001279 
  intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 3962c3c42 
  intg/src/main/java/org/apache/atlas/type/Constants.java 3fc13056e 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
 379150b7b 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
 36bee301d 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityMutationContext.java
 deb743eea 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/IDBasedEntityResolver.java
 3b9694851 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java
 38228a8ec 


Diff: https://reviews.apache.org/r/72438/diff/2/

Changes: https://reviews.apache.org/r/72438/diff/1-2/


Testing (updated)
---

- Manually tested
- Unit test updated


Thanks,

Bolke de Bruin



[jira] [Updated] (ATLAS-3755) Allow system attributes to be updated when policy allows

2020-04-27 Thread Bolke de Bruin (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin updated ATLAS-3755:
--
Attachment: 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch

> Allow system attributes to be updated when policy allows
> 
>
> Key: ATLAS-3755
> URL: https://issues.apache.org/jira/browse/ATLAS-3755
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0, 2.1.0
>    Reporter: Bolke de Bruin
>    Assignee: Bolke de Bruin
>Priority: Critical
> Attachments: 
> 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, 
> 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch
>
>
> Atlas does not operate in a isolated environment, this is one of the reasons 
> the "homeId" system attribute was introduced. Unfortunately system attributes 
> can only be updated when importing. This means any integration with other 
> services is significantly limited (Kafka, Rest API will not work). (See also 
> ATLAS-3754)
> To resolve this I propose to make it possible to update the system attributes 
> when policy allows it. This introduces new 
> AtlasPrivilege.ENTITY_UPDATE_SYSTEM_ATTRIBUTE and 
> AtlasPrivilege.ENTITY_CREATE_SYSTEM_ATTRIBUTE next to 
> AtlasPrivilege.ENTITY_UPDATE_ATTRIBUTE and 
> AtlasPrivilege.ENTITY_CREATE_ATTRIBUTE rather than just checking on the 
> entity level. In certain places we will then drop the requirement for an 
> import to be active as this can now happen through other channels as well.
> This allows operators to specify policies that allow granular controls over 
> attributes and system attributes.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3755) Allow system attributes to be updated when policy allows

2020-04-27 Thread Bolke de Bruin (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin updated ATLAS-3755:
--
External issue URL: https://reviews.apache.org/r/72438/

> Allow system attributes to be updated when policy allows
> 
>
> Key: ATLAS-3755
> URL: https://issues.apache.org/jira/browse/ATLAS-3755
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0, 2.1.0
>    Reporter: Bolke de Bruin
>    Assignee: Bolke de Bruin
>Priority: Critical
> Attachments: 
> 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch
>
>
> Atlas does not operate in a isolated environment, this is one of the reasons 
> the "homeId" system attribute was introduced. Unfortunately system attributes 
> can only be updated when importing. This means any integration with other 
> services is significantly limited (Kafka, Rest API will not work). (See also 
> ATLAS-3754)
> To resolve this I propose to make it possible to update the system attributes 
> when policy allows it. This introduces new 
> AtlasPrivilege.ENTITY_UPDATE_SYSTEM_ATTRIBUTE and 
> AtlasPrivilege.ENTITY_CREATE_SYSTEM_ATTRIBUTE next to 
> AtlasPrivilege.ENTITY_UPDATE_ATTRIBUTE and 
> AtlasPrivilege.ENTITY_CREATE_ATTRIBUTE rather than just checking on the 
> entity level. In certain places we will then drop the requirement for an 
> import to be active as this can now happen through other channels as well.
> This allows operators to specify policies that allow granular controls over 
> attributes and system attributes.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Review Request 72438: Allow system attributes to be updated when policy allows

2020-04-27 Thread Bolke de Bruin

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72438/
---

Review request for atlas, Ashutosh Mestry, Bolke de Bruin, Madhan Neethiraj, 
Nixon Rodrigues, and Sarath Subramanian.


Bugs: ATLAS-3755
https://issues.apache.org/jira/browse/ATLAS-3755


Repository: atlas


Description
---

Atlas does not operate in a isolated environment, this is one of the reasons 
the "homeId" system attribute was introduced. Unfortunately system attributes 
can only be updated when importing. This means any integration with other 
services is significantly limited (Kafka, Rest API will not work). (See also 
ATLAS-3754)
To resolve this I propose to make it possible to update the system attributes 
when policy allows it. This introduces new 
AtlasPrivilege.ENTITY_UPDATE_SYSTEM_ATTRIBUTE and 
AtlasPrivilege.ENTITY_CREATE_SYSTEM_ATTRIBUTE next to 
AtlasPrivilege.ENTITY_UPDATE_ATTRIBUTE and 
AtlasPrivilege.ENTITY_CREATE_ATTRIBUTE rather than just checking on the entity 
level. In certain places we will then drop the requirement for an import to be 
active as this can now happen through other channels as well.
This allows operators to specify policies that allow granular controls over 
attributes and system attributes.


Diffs
-

  authorization/src/main/java/org/apache/atlas/authorize/AtlasPrivilege.java 
7287b3dd7 
  authorization/src/main/resources/atlas-simple-authz-policy.json 6b2001279 
  intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 3962c3c42 
  intg/src/main/java/org/apache/atlas/type/Constants.java 3fc13056e 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
 379150b7b 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
 36bee301d 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityMutationContext.java
 deb743eea 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/IDBasedEntityResolver.java
 3b9694851 


Diff: https://reviews.apache.org/r/72438/diff/1/


Testing
---

- Manual testing only
- Unit tests have not been updated yet


Thanks,

Bolke de Bruin



[jira] [Updated] (ATLAS-3755) Allow system attributes to be updated when policy allows

2020-04-27 Thread Bolke de Bruin (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin updated ATLAS-3755:
--
Attachment: 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch

> Allow system attributes to be updated when policy allows
> 
>
> Key: ATLAS-3755
> URL: https://issues.apache.org/jira/browse/ATLAS-3755
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0, 2.1.0
>    Reporter: Bolke de Bruin
>    Assignee: Bolke de Bruin
>Priority: Critical
> Attachments: 
> 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch
>
>
> Atlas does not operate in a isolated environment, this is one of the reasons 
> the "homeId" system attribute was introduced. Unfortunately system attributes 
> can only be updated when importing. This means any integration with other 
> services is significantly limited (Kafka, Rest API will not work). (See also 
> ATLAS-3754)
> To resolve this I propose to make it possible to update the system attributes 
> when policy allows it. This introduces new 
> AtlasPrivilege.ENTITY_UPDATE_SYSTEM_ATTRIBUTE and 
> AtlasPrivilege.ENTITY_CREATE_SYSTEM_ATTRIBUTE next to 
> AtlasPrivilege.ENTITY_UPDATE_ATTRIBUTE and 
> AtlasPrivilege.ENTITY_CREATE_ATTRIBUTE rather than just checking on the 
> entity level. In certain places we will then drop the requirement for an 
> import to be active as this can now happen through other channels as well.
> This allows operators to specify policies that allow granular controls over 
> attributes and system attributes.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (ATLAS-3755) Allow system attributes to be updated when policy allows

2020-04-27 Thread Bolke de Bruin (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17093423#comment-17093423
 ] 

Bolke de Bruin commented on ATLAS-3755:
---

cc [~madhan]

> Allow system attributes to be updated when policy allows
> 
>
> Key: ATLAS-3755
> URL: https://issues.apache.org/jira/browse/ATLAS-3755
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0, 2.1.0
>    Reporter: Bolke de Bruin
>    Assignee: Bolke de Bruin
>Priority: Critical
>
> Atlas does not operate in a isolated environment, this is one of the reasons 
> the "homeId" system attribute was introduced. Unfortunately system attributes 
> can only be updated when importing. This means any integration with other 
> services is significantly limited (Kafka, Rest API will not work). (See also 
> ATLAS-3754)
> To resolve this I propose to make it possible to update the system attributes 
> when policy allows it. This introduces new 
> AtlasPrivilege.ENTITY_UPDATE_SYSTEM_ATTRIBUTE and 
> AtlasPrivilege.ENTITY_CREATE_SYSTEM_ATTRIBUTE next to 
> AtlasPrivilege.ENTITY_UPDATE_ATTRIBUTE and 
> AtlasPrivilege.ENTITY_CREATE_ATTRIBUTE rather than just checking on the 
> entity level. In certain places we will then drop the requirement for an 
> import to be active as this can now happen through other channels as well.
> This allows operators to specify policies that allow granular controls over 
> attributes and system attributes.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (ATLAS-3754) System/Business Attributes (like homeId) not being saved

2020-04-27 Thread Bolke de Bruin (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17093419#comment-17093419
 ] 

Bolke de Bruin commented on ATLAS-3754:
---

[~madhan] See the linked issue please. I am working on a patch to refactors 
preCreateOrUpdate and createOrUpdate to allow for this per policy. Changes are 
being detected and only set if the policy allows it. Of course if a client 
still specifies an (system)attribute but is not allowed to set it, it would 
fail now instead of silently dropping such items. Imho, that should be fine as 
the client would then be misbehaving.

> System/Business Attributes (like homeId) not being saved
> 
>
> Key: ATLAS-3754
> URL: https://issues.apache.org/jira/browse/ATLAS-3754
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: trunk
>        Reporter: Bolke de Bruin
>Priority: Blocker
>
> System attributes and business attributes are not being saved (homeId at 
> least isn't)
> {code}
> skipping unchanged entity: AtlasEntity{AtlasStruct{typeName='hdfs_path', 
> attributes=[name:electionresults, 
> qualifiedName:electionresults.electionresults@Sandbox, 
> path:/user/dmaster/electionresults/ls2014.tsv, 
> clusterName:Sandbox]}guid='8e6ede0c-22c5-4b41-bee4-757568b4b99a', 
> homeId='my_home_id_x', isProxy='false', isIncomplete=false, provenanceType=0, 
> status=null, createdBy='null', updatedBy='null', createTime=null, 
> updateTime=null, version=0, relationshipAttributes=[], classifications=[], 
> meanings=[], customAttributes=[], businessAttributes=[], labels=[]} 
> (AtlasEntityStoreV2:1204)
> {code}
> {code}
> skipping unchanged entity: AtlasEntity{AtlasStruct{typeName='hdfs_path', 
> attributes=[name:electionresults, 
> qualifiedName:electionresults.electionresults@Sandbox, 
> path:/user/dmaster/electionresults/ls2014.tsv, 
> clusterName:Sandbox]}guid='8e6ede0c-22c5-4b41-bee4-757568b4b99a', 
> homeId='my_home_id_2', isProxy='false', isIncomplete=false, provenanceType=0, 
> status=null, createdBy='null', updatedBy='null', createTime=null, 
> updateTime=null, version=0, relationshipAttributes=[], classifications=[], 
> meanings=[], customAttributes=[], businessAttributes=[], labels=[]} 
> (AtlasEntityStoreV2:1204)
> {code}
> Note the change in homeId and the entity not being updated



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3755) Allow system attributes to be updated when policy allows

2020-04-27 Thread Bolke de Bruin (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin updated ATLAS-3755:
--
Description: 
Atlas does not operate in a isolated environment, this is one of the reasons 
the "homeId" system attribute was introduced. Unfortunately system attributes 
can only be updated when importing. This means any integration with other 
services is significantly limited (Kafka, Rest API will not work). (See also 
ATLAS-3754)

To resolve this I propose to make it possible to update the system attributes 
when policy allows it. This introduces new 
AtlasPrivilege.ENTITY_UPDATE_SYSTEM_ATTRIBUTE and 
AtlasPrivilege.ENTITY_CREATE_SYSTEM_ATTRIBUTE next to 
AtlasPrivilege.ENTITY_UPDATE_ATTRIBUTE and 
AtlasPrivilege.ENTITY_CREATE_ATTRIBUTE rather than just checking on the entity 
level. In certain places we will then drop the requirement for an import to be 
active as this can now happen through other channels as well.

This allows operators to specify policies that allow granular controls over 
attributes and system attributes.




  was:
Atlas does not operate in a isolated environment, this is one of the reasons 
the "homeId" system attribute was introduced. Unfortunately system attributes 
can only be updated when importing. This means any integration with other 
services is significantly limited (Kafka, Rest API will not work). (See also 
ATLAS-3754)

To resolve this I propose to make it possible to update the system attributes 
when `homeId` is present in the entity update. An access check should then 
verify whether such an update can happen if a `homeId` is present.






> Allow system attributes to be updated when policy allows
> 
>
> Key: ATLAS-3755
> URL: https://issues.apache.org/jira/browse/ATLAS-3755
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0, 2.1.0
>    Reporter: Bolke de Bruin
>Assignee: Bolke de Bruin
>Priority: Critical
>
> Atlas does not operate in a isolated environment, this is one of the reasons 
> the "homeId" system attribute was introduced. Unfortunately system attributes 
> can only be updated when importing. This means any integration with other 
> services is significantly limited (Kafka, Rest API will not work). (See also 
> ATLAS-3754)
> To resolve this I propose to make it possible to update the system attributes 
> when policy allows it. This introduces new 
> AtlasPrivilege.ENTITY_UPDATE_SYSTEM_ATTRIBUTE and 
> AtlasPrivilege.ENTITY_CREATE_SYSTEM_ATTRIBUTE next to 
> AtlasPrivilege.ENTITY_UPDATE_ATTRIBUTE and 
> AtlasPrivilege.ENTITY_CREATE_ATTRIBUTE rather than just checking on the 
> entity level. In certain places we will then drop the requirement for an 
> import to be active as this can now happen through other channels as well.
> This allows operators to specify policies that allow granular controls over 
> attributes and system attributes.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3755) Allow system attributes to be updated when policy allows

2020-04-27 Thread Bolke de Bruin (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin updated ATLAS-3755:
--
Summary: Allow system attributes to be updated when policy allows  (was: 
Allow system attributes to be updated when homeId is specified)

> Allow system attributes to be updated when policy allows
> 
>
> Key: ATLAS-3755
> URL: https://issues.apache.org/jira/browse/ATLAS-3755
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0, 2.1.0
>    Reporter: Bolke de Bruin
>    Assignee: Bolke de Bruin
>Priority: Critical
>
> Atlas does not operate in a isolated environment, this is one of the reasons 
> the "homeId" system attribute was introduced. Unfortunately system attributes 
> can only be updated when importing. This means any integration with other 
> services is significantly limited (Kafka, Rest API will not work). (See also 
> ATLAS-3754)
> To resolve this I propose to make it possible to update the system attributes 
> when `homeId` is present in the entity update. An access check should then 
> verify whether such an update can happen if a `homeId` is present.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (ATLAS-3755) Allow system attributes to be updated when homeId is specified

2020-04-25 Thread Bolke de Bruin (Jira)
Bolke de Bruin created ATLAS-3755:
-

 Summary: Allow system attributes to be updated when homeId is 
specified
 Key: ATLAS-3755
 URL: https://issues.apache.org/jira/browse/ATLAS-3755
 Project: Atlas
  Issue Type: Improvement
  Components:  atlas-core
Affects Versions: 2.0.0, 2.1.0
Reporter: Bolke de Bruin
Assignee: Bolke de Bruin


Atlas does not operate in a isolated environment, this is one of the reasons 
the "homeId" system attribute was introduced. Unfortunately system attributes 
can only be updated when importing. This means any integration with other 
services is significantly limited (Kafka, Rest API will not work). (See also 
ATLAS-3754)

To resolve this I propose to make it possible to update the system attributes 
when `homeId` is present in the entity update. An access check should then 
verify whether such an update can happen if a `homeId` is present.







--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3754) System/Business Attributes (like homeId) not being saved

2020-04-24 Thread Bolke de Bruin (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin updated ATLAS-3754:
--
Description: 
System attributes and business attributes are not being saved (homeId at least 
isn't)


{code}
skipping unchanged entity: AtlasEntity{AtlasStruct{typeName='hdfs_path', 
attributes=[name:electionresults, 
qualifiedName:electionresults.electionresults@Sandbox, 
path:/user/dmaster/electionresults/ls2014.tsv, 
clusterName:Sandbox]}guid='8e6ede0c-22c5-4b41-bee4-757568b4b99a', 
homeId='my_home_id_x', isProxy='false', isIncomplete=false, provenanceType=0, 
status=null, createdBy='null', updatedBy='null', createTime=null, 
updateTime=null, version=0, relationshipAttributes=[], classifications=[], 
meanings=[], customAttributes=[], businessAttributes=[], labels=[]} 
(AtlasEntityStoreV2:1204)
{code}

{code}
skipping unchanged entity: AtlasEntity{AtlasStruct{typeName='hdfs_path', 
attributes=[name:electionresults, 
qualifiedName:electionresults.electionresults@Sandbox, 
path:/user/dmaster/electionresults/ls2014.tsv, 
clusterName:Sandbox]}guid='8e6ede0c-22c5-4b41-bee4-757568b4b99a', 
homeId='my_home_id_2', isProxy='false', isIncomplete=false, provenanceType=0, 
status=null, createdBy='null', updatedBy='null', createTime=null, 
updateTime=null, version=0, relationshipAttributes=[], classifications=[], 
meanings=[], customAttributes=[], businessAttributes=[], labels=[]} 
(AtlasEntityStoreV2:1204)
{code}

Note the change in homeId and the entity not being updated

  was:
Since introducing the name spacing attributes that fall outside of the typedef 
i.e. system attributes and business attributes are not being saved anymore due 
to a Noop in AtlasStructType

{code}
public AtlasAttribute getAttribute(String attributeName) {
AtlasAttribute ret = allAttributes.get(attributeName);

if (ret == null) {
ret = getSystemAttribute(attributeName);
}

if (ret == null) {
ret = getBusinesAAttribute(attributeName);
}

return ret;
}

public AtlasAttribute getSystemAttribute(String attributeName) {
return null;
}

public AtlasAttribute getBusinesAAttribute(String attributeName) {
return null;
}
{code}

{code}
skipping unchanged entity: AtlasEntity{AtlasStruct{typeName='hdfs_path', 
attributes=[name:electionresults, 
qualifiedName:electionresults.electionresults@Sandbox, 
path:/user/dmaster/electionresults/ls2014.tsv, 
clusterName:Sandbox]}guid='8e6ede0c-22c5-4b41-bee4-757568b4b99a', 
homeId='my_home_id_x', isProxy='false', isIncomplete=false, provenanceType=0, 
status=null, createdBy='null', updatedBy='null', createTime=null, 
updateTime=null, version=0, relationshipAttributes=[], classifications=[], 
meanings=[], customAttributes=[], businessAttributes=[], labels=[]} 
(AtlasEntityStoreV2:1204)
{code}

{code}
skipping unchanged entity: AtlasEntity{AtlasStruct{typeName='hdfs_path', 
attributes=[name:electionresults, 
qualifiedName:electionresults.electionresults@Sandbox, 
path:/user/dmaster/electionresults/ls2014.tsv, 
clusterName:Sandbox]}guid='8e6ede0c-22c5-4b41-bee4-757568b4b99a', 
homeId='my_home_id_2', isProxy='false', isIncomplete=false, provenanceType=0, 
status=null, createdBy='null', updatedBy='null', createTime=null, 
updateTime=null, version=0, relationshipAttributes=[], classifications=[], 
meanings=[], customAttributes=[], businessAttributes=[], labels=[]} 
(AtlasEntityStoreV2:1204)
{code}

Note the change in homeId and the entity not being updated


> System/Business Attributes (like homeId) not being saved
> 
>
> Key: ATLAS-3754
> URL: https://issues.apache.org/jira/browse/ATLAS-3754
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: trunk
>        Reporter: Bolke de Bruin
>Priority: Blocker
>
> System attributes and business attributes are not being saved (homeId at 
> least isn't)
> {code}
> skipping unchanged entity: AtlasEntity{AtlasStruct{typeName='hdfs_path', 
> attributes=[name:electionresults, 
> qualifiedName:electionresults.electionresults@Sandbox, 
> path:/user/dmaster/electionresults/ls2014.tsv, 
> clusterName:Sandbox]}guid='8e6ede0c-22c5-4b41-bee4-757568b4b99a', 
> homeId='my_home_id_x', isProxy='false', isIncomplete=false, provenanceType=0, 
> status=null, createdBy='null', updatedBy='null', createTime=null, 
> updateTime=null, version=0, relationshipAttributes=[], classifications=[], 
> meanings=[], customAttributes=[], businessAttributes=[], labels=[]} 
> (AtlasEntityStoreV2:1204)
> {code}
> {code}
> skipping unchanged entity: AtlasEntity{AtlasStruct{typeName='hdfs_path', 
> attributes=[name:electionresults, 
> qualifiedName:electionresults.electionresults@Sandbox, 

[jira] [Commented] (ATLAS-3754) System/Business Attributes (like homeId) not being saved

2020-04-24 Thread Bolke de Bruin (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091455#comment-17091455
 ] 

Bolke de Bruin commented on ATLAS-3754:
---

cc [~madhan]

> System/Business Attributes (like homeId) not being saved
> 
>
> Key: ATLAS-3754
> URL: https://issues.apache.org/jira/browse/ATLAS-3754
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: trunk
>        Reporter: Bolke de Bruin
>Priority: Blocker
>
> Since introducing the name spacing attributes that fall outside of the 
> typedef i.e. system attributes and business attributes are not being saved 
> anymore due to a Noop in AtlasStructType
> {code}
> public AtlasAttribute getAttribute(String attributeName) {
> AtlasAttribute ret = allAttributes.get(attributeName);
> if (ret == null) {
> ret = getSystemAttribute(attributeName);
> }
> if (ret == null) {
> ret = getBusinesAAttribute(attributeName);
> }
> return ret;
> }
> public AtlasAttribute getSystemAttribute(String attributeName) {
> return null;
> }
> public AtlasAttribute getBusinesAAttribute(String attributeName) {
> return null;
> }
> {code}
> {code}
> skipping unchanged entity: AtlasEntity{AtlasStruct{typeName='hdfs_path', 
> attributes=[name:electionresults, 
> qualifiedName:electionresults.electionresults@Sandbox, 
> path:/user/dmaster/electionresults/ls2014.tsv, 
> clusterName:Sandbox]}guid='8e6ede0c-22c5-4b41-bee4-757568b4b99a', 
> homeId='my_home_id_x', isProxy='false', isIncomplete=false, provenanceType=0, 
> status=null, createdBy='null', updatedBy='null', createTime=null, 
> updateTime=null, version=0, relationshipAttributes=[], classifications=[], 
> meanings=[], customAttributes=[], businessAttributes=[], labels=[]} 
> (AtlasEntityStoreV2:1204)
> {code}
> {code}
> skipping unchanged entity: AtlasEntity{AtlasStruct{typeName='hdfs_path', 
> attributes=[name:electionresults, 
> qualifiedName:electionresults.electionresults@Sandbox, 
> path:/user/dmaster/electionresults/ls2014.tsv, 
> clusterName:Sandbox]}guid='8e6ede0c-22c5-4b41-bee4-757568b4b99a', 
> homeId='my_home_id_2', isProxy='false', isIncomplete=false, provenanceType=0, 
> status=null, createdBy='null', updatedBy='null', createTime=null, 
> updateTime=null, version=0, relationshipAttributes=[], classifications=[], 
> meanings=[], customAttributes=[], businessAttributes=[], labels=[]} 
> (AtlasEntityStoreV2:1204)
> {code}
> Note the change in homeId and the entity not being updated



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (ATLAS-3754) Attributes (homeId) not being saved

2020-04-24 Thread Bolke de Bruin (Jira)
Bolke de Bruin created ATLAS-3754:
-

 Summary: Attributes (homeId) not being saved
 Key: ATLAS-3754
 URL: https://issues.apache.org/jira/browse/ATLAS-3754
 Project: Atlas
  Issue Type: Bug
Affects Versions: trunk
Reporter: Bolke de Bruin


Since introducing the name spacing attributes that fall outside of the typedef 
i.e. system attributes and business attributes are not being saved anymore due 
to a Noop in AtlasStructType

{code}
public AtlasAttribute getAttribute(String attributeName) {
AtlasAttribute ret = allAttributes.get(attributeName);

if (ret == null) {
ret = getSystemAttribute(attributeName);
}

if (ret == null) {
ret = getBusinesAAttribute(attributeName);
}

return ret;
}

public AtlasAttribute getSystemAttribute(String attributeName) {
return null;
}

public AtlasAttribute getBusinesAAttribute(String attributeName) {
return null;
}
{code}

{code}
skipping unchanged entity: AtlasEntity{AtlasStruct{typeName='hdfs_path', 
attributes=[name:electionresults, 
qualifiedName:electionresults.electionresults@Sandbox, 
path:/user/dmaster/electionresults/ls2014.tsv, 
clusterName:Sandbox]}guid='8e6ede0c-22c5-4b41-bee4-757568b4b99a', 
homeId='my_home_id_x', isProxy='false', isIncomplete=false, provenanceType=0, 
status=null, createdBy='null', updatedBy='null', createTime=null, 
updateTime=null, version=0, relationshipAttributes=[], classifications=[], 
meanings=[], customAttributes=[], businessAttributes=[], labels=[]} 
(AtlasEntityStoreV2:1204)
{code}

{code}
skipping unchanged entity: AtlasEntity{AtlasStruct{typeName='hdfs_path', 
attributes=[name:electionresults, 
qualifiedName:electionresults.electionresults@Sandbox, 
path:/user/dmaster/electionresults/ls2014.tsv, 
clusterName:Sandbox]}guid='8e6ede0c-22c5-4b41-bee4-757568b4b99a', 
homeId='my_home_id_2', isProxy='false', isIncomplete=false, provenanceType=0, 
status=null, createdBy='null', updatedBy='null', createTime=null, 
updateTime=null, version=0, relationshipAttributes=[], classifications=[], 
meanings=[], customAttributes=[], businessAttributes=[], labels=[]} 
(AtlasEntityStoreV2:1204)
{code}

Note the change in homeId and the entity not being updated



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3754) System/Business Attributes (like homeId) not being saved

2020-04-24 Thread Bolke de Bruin (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin updated ATLAS-3754:
--
Summary: System/Business Attributes (like homeId) not being saved  (was: 
Attributes (homeId) not being saved)

> System/Business Attributes (like homeId) not being saved
> 
>
> Key: ATLAS-3754
> URL: https://issues.apache.org/jira/browse/ATLAS-3754
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: trunk
>        Reporter: Bolke de Bruin
>Priority: Blocker
>
> Since introducing the name spacing attributes that fall outside of the 
> typedef i.e. system attributes and business attributes are not being saved 
> anymore due to a Noop in AtlasStructType
> {code}
> public AtlasAttribute getAttribute(String attributeName) {
> AtlasAttribute ret = allAttributes.get(attributeName);
> if (ret == null) {
> ret = getSystemAttribute(attributeName);
> }
> if (ret == null) {
> ret = getBusinesAAttribute(attributeName);
> }
> return ret;
> }
> public AtlasAttribute getSystemAttribute(String attributeName) {
> return null;
> }
> public AtlasAttribute getBusinesAAttribute(String attributeName) {
> return null;
> }
> {code}
> {code}
> skipping unchanged entity: AtlasEntity{AtlasStruct{typeName='hdfs_path', 
> attributes=[name:electionresults, 
> qualifiedName:electionresults.electionresults@Sandbox, 
> path:/user/dmaster/electionresults/ls2014.tsv, 
> clusterName:Sandbox]}guid='8e6ede0c-22c5-4b41-bee4-757568b4b99a', 
> homeId='my_home_id_x', isProxy='false', isIncomplete=false, provenanceType=0, 
> status=null, createdBy='null', updatedBy='null', createTime=null, 
> updateTime=null, version=0, relationshipAttributes=[], classifications=[], 
> meanings=[], customAttributes=[], businessAttributes=[], labels=[]} 
> (AtlasEntityStoreV2:1204)
> {code}
> {code}
> skipping unchanged entity: AtlasEntity{AtlasStruct{typeName='hdfs_path', 
> attributes=[name:electionresults, 
> qualifiedName:electionresults.electionresults@Sandbox, 
> path:/user/dmaster/electionresults/ls2014.tsv, 
> clusterName:Sandbox]}guid='8e6ede0c-22c5-4b41-bee4-757568b4b99a', 
> homeId='my_home_id_2', isProxy='false', isIncomplete=false, provenanceType=0, 
> status=null, createdBy='null', updatedBy='null', createTime=null, 
> updateTime=null, version=0, relationshipAttributes=[], classifications=[], 
> meanings=[], customAttributes=[], businessAttributes=[], labels=[]} 
> (AtlasEntityStoreV2:1204)
> {code}
> Note the change in homeId and the entity not being updated



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re:

2020-04-03 Thread Bolke de Bruin
Thanks Madhan. I'll check with the author. But I also see possible confusion. 
Basically you dont want to fork to the background when logging to the console. 

It would have been nice to have this feedback on Jira or the review board now 
it was unclear why it was stuck after some marked it "ship it".

Cheers
Bolke

Sent from my iPhone
> Bolke,
> 
> With the changes in this patch, server startup script (atlas_start.py) 
> doesn't exit until Atlas server process exits - with 
> 'ENABLE_LOGGING_TO_CONSOLE=true' in atlas-env.sh. This script is supposed to 
> launch the  process, write the process-id in atlas.pid, and exit. Please 
> review and update.
> 
> Thanks,
> Madhan
> 
> On 4/3/20, 4:54 AM, "Bolke de Bruin"  wrote:
> 
>Hi Team,
> 
>Can ATLAS-3610 (https://reviews.apache.org/r/72104/) be merged please? It
>has been open for more than 1.5 months, all comments have been addressed.
>What is holding this back?
> 
>Cheers
>Bolke
> 
> 
> 


Please merge ATLAS-3610

2020-04-03 Thread Bolke de Bruin
Hi Team,

Can ATLAS-3610 (https://reviews.apache.org/r/72104/) be merged please? It
has been open for more than 1.5 months, all comments have been addressed.
What is holding this back?

Cheers
Bolke


Re: Apache Atlas 2.1 release

2020-03-27 Thread Bolke de Bruin
That would be great! A faster release cycle would be appreciated as well.

When is the vote due?

Thanks
Bolke

Sent from my iPhone

> On 19 Feb 2020, at 02:10, Madhan Neethiraj  wrote:
> 
> Atlas community,
> 
> 
> 
> Over past months the dev community has been busy in enhancing Apache Atlas 
> with new features, improvements and fixes. Here are few features/enhancements 
> since last major release, Apache Atlas 2.0:
> 
> - added quick-search feature, to provide a simpler search experience with 
> type-ahead suggestions
> 
> - introduced Namespaces feature, which allows grouping of attributes to 
> be applied to multiple entity-types
> 
> - introduced labels on entity instances, and search for entities using 
> the label
> 
> - enhancement to support entity instance specific custom attributes
> 
> - enhanced search to find entities by more than one classification
> 
> - introduced shell/incomplete entities to handle notifications 
> referencing entities that don’t (yet) exist in Atlas
> 
> - added REST APIs to purge deleted entities
> 
> - performance improvements in lineage retrieval and tag-propagation
> 
> - updated Atlas server to process notifications from multiple Kafka topics
> 
> - updated Hive hook to track process executions, via 
> hive_process_execution entities
> 
> - updated Hive hook to capture DDL operations, via hive_db_ddl and 
> hive_table_ddl entities
> 
> - added models for Spark; introduced new models for AWS S3
> 
> - updated versions of dependent libraries/components: JanusGraph, Jackson 
> parser, Spring Framework,
> 
> - updated authorization model to cover new features/APIs, like add/remove 
> labels, purge entities, update namespace attributes
> 
> 
> 
> With significant improvements in place, it is time for the next maintenance 
> release of Apache Atlas!
> 
> 
> 
> I propose to release Apache Atlas 2.1 by early next month. Please review and 
> send your comments.
> 
> 
> 
> Regards,
> 
> Madhan
> 


[DISCUSS] How to speed up contributions?

2020-03-27 Thread Bolke de Bruin
Dear Atlas Dev Community,

Myself and my team have been contributors to Apache Atlas. We have contributed 
in the area of authentication and kubernetes/docker improvements. We like to 
contribute in the context of performance and usability. 

Unfortunately, we also notice that it is hard to get attention to the 
contributions. Saying it out loud: the dev list is hardly alive with close to 
none discussion (e.g. see the Apache Atlas 2.1 release ‘thread’ of February), 
Jira items feel lost and abandoned, items on the review board take months to 
get integrated although they are “ship it” marked. There are some 
‘fast-tracked’ items, but they seem to come from a selective group of people 
and feels not inclusive. This results in loss of interest/energy of potential 
contributors.

What can be done to improve this situation? I think there are a couple of 
options that could help with improving this. I don’t want to stipulate any of 
them but want to put them up for consideration, like:

1. Add new committers. I have not seen the latest report to the ASF, but I am 
going to assume that assume that it has been a while that new committers have 
added to the project.
2. Allow and accept PRs on Github. The ASF has moved to GitHub and it does give 
the same if not better and more native workflow for contributors.
3. Consider moving to Github issues. GH Issues and JIRA are both not perfect, 
but Jira does tend to make discussions less visible.
4. Integrate with a CI like Travis or Azure pipelines. Visibility whether a 
particular PR passes tests increases confidence in the PR and speeds up 
contributions. ASF has a contract with Travis although Azure pipelines seem to 
gain more popularity.

What do you think?

Kind regards,
Bolke

Verstuurd vanaf mijn iPad

Fwd: Review Request 72104: ATLAS-3610 Enable logging to STDOUT in Atlas startup scripts

2020-03-26 Thread Bolke de Bruin
Can this be merged please?

Sent from my iPhone

Begin forwarded message:

> From: Bolke de Bruin 
> Date: 26 March 2020 at 14:38:51 CET
> To: Madhan Neethiraj , Sarath Subramanian 
> , Ashutosh Mestry , Nixon Rodrigues 
> , Bolke de Bruin 
> Cc: atlas , Mariusz Górski 
> 
> Subject: Re:  Review Request 72104: ATLAS-3610 Enable logging to STDOUT in 
> Atlas startup scripts
> Reply-To: Bolke de Bruin 
> 
> 
> This is an automatically generated e-mail. To reply, visit: 
> https://reviews.apache.org/r/72104/
> 
> Ship it!
> 
> Ship It!
> 
> - Bolke de Bruin
> 
> 
> On March 26th, 2020, 7:41 a.m. UTC, Mariusz Górski wrote:
> 
> Review request for atlas, Ashutosh Mestry, Bolke de Bruin, Madhan Neethiraj, 
> Nixon Rodrigues, and Sarath Subramanian.
> By Mariusz Górski.
> Updated March 26, 2020, 7:41 a.m.
> 
> Repository: atlas
> Description
> 
> This patch enables logging to STDOUT from processes spawned through 
> atlas_start.py or atlas_config.py scripts. For now, even if configured in 
> log4j, logging to STDOUT was supressed (by redirecting to log files only).
> Testing
> 
> The tests were concluded by:
> 
> Building & running Atlas locally with default log4j configuration and 
> -Dlog.console mvn build option set to true.
> Building & running Atlas on Openshift with log4j configuration pushing 
> everything to STDOUT and -Dlog.console mvn build option set to true.
> Running distro tests on localhost
> 
> In both scenarios logs were available both in the .out/.err files in the 
> Atlas logdir (backwards compatibility), but at the same time I was able to 
> see them in STDOUT of both running process (local installation) and pod 
> (openshift deployment).
> Diffs
> 
> distro/pom.xml (7159b16cf)
> distro/src/bin/atlas_config.py (f09026ff9)
> distro/src/bin/atlas_start.py (963faf402)
> distro/src/bin/cputil.py (98a9dc36d)
> distro/src/conf/atlas-env.sh (c4241e665)
> distro/src/test/python/scripts/TestMetadata.py (f1235f747)
> View Diff


Re: Review Request 72104: ATLAS-3610 Enable logging to STDOUT in Atlas startup scripts

2020-03-26 Thread Bolke de Bruin

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72104/#review220088
---


Ship it!




Ship It!

- Bolke de Bruin


On March 26, 2020, 7:41 a.m., Mariusz Górski wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72104/
> ---
> 
> (Updated March 26, 2020, 7:41 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Bolke de Bruin, Madhan Neethiraj, 
> Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This patch enables logging to STDOUT from processes spawned through 
> atlas_start.py or atlas_config.py scripts. For now, even if configured in 
> log4j, logging to STDOUT was supressed (by redirecting to log files only).
> 
> 
> Diffs
> -
> 
>   distro/pom.xml 7159b16cf 
>   distro/src/bin/atlas_config.py f09026ff9 
>   distro/src/bin/atlas_start.py 963faf402 
>   distro/src/bin/cputil.py 98a9dc36d 
>   distro/src/conf/atlas-env.sh c4241e665 
>   distro/src/test/python/scripts/TestMetadata.py f1235f747 
> 
> 
> Diff: https://reviews.apache.org/r/72104/diff/4/
> 
> 
> Testing
> ---
> 
> The tests were concluded by:
> 
> 1. Building & running Atlas locally with default log4j configuration and 
> -Dlog.console mvn build option set to true.
> 2. Building & running Atlas on Openshift with log4j configuration pushing 
> everything to STDOUT and -Dlog.console mvn build option set to true.
> 3. Running distro tests on localhost
> 
> In both scenarios logs were available both in the .out/.err files in the 
> Atlas logdir (backwards compatibility), but at the same time I was able to 
> see them in STDOUT of both running process (local installation) and pod 
> (openshift deployment).
> 
> 
> Thanks,
> 
> Mariusz Górski
> 
>



Fwd: Review Request 72104: Enable logging to STDOUT in Atlas startup scripts

2020-02-12 Thread Bolke de Bruin


Sent from my iPhone

Begin forwarded message:

> From: Bolke de Bruin 
> Date: 12 February 2020 at 09:38:43 CET
> To: Madhan Neethiraj , dev@atlas.apache.org
> Subject: Fwd:  Review Request 72104: Enable logging to STDOUT in Atlas 
> startup scripts
> 
> Can this be reviewed please?
> 
> Sent from my iPhone
> 
> Begin forwarded message:
> 
>> From: Mariusz Górski 
>> Date: 10 February 2020 at 12:22:51 CET
>> To: Bolke de Bruin 
>> Cc: atlas , Mariusz Górski 
>> 
>> Subject: Review Request 72104: Enable logging to STDOUT in Atlas startup 
>> scripts
>> Reply-To: Mariusz Górski @reviews.apache.org
>> 
>> 
>> This is an automatically generated e-mail. To reply, visit: 
>> https://reviews.apache.org/r/72104/
>> 
>> Review request for atlas and Bolke de Bruin.
>> By Mariusz Górski.
>> Repository: atlas
>> Description
>> 
>> This patch enables logging to STDOUT from processes spawned through 
>> atlas_start.py or atlas_config.py scripts. For now, even if configured in 
>> log4j, logging to STDOUT was supressed (by redirecting to log files only).
>> Testing
>> 
>> The tests were concluded by:
>> 
>> Building & running Atlas locally with default log4j configuration and 
>> ENABLE_LOGGING_TO_CONSOLE variable set to true.
>> Building & running Atlas on Openshift with log4j configuration pushing 
>> everything to STDOUT and ENABLE_LOGGING_TO_CONSOLE variable set to true.
>> 
>> In both scenarios logs were available both in the .out/.err files in the 
>> Atlas logdir (backwards compatibility), but at the same time I was able to 
>> see them in STDOUT of both running process (local installation) and pod 
>> (openshift deployment).
>> Diffs
>> 
>> distro/src/bin/atlas_config.py (f09026ff9)
>> distro/src/bin/atlas_start.py (963faf402)
>> View Diff


Fwd: Review Request 72104: Enable logging to STDOUT in Atlas startup scripts

2020-02-12 Thread Bolke de Bruin
Can this be reviewed please?

Sent from my iPhone

Begin forwarded message:

> From: Mariusz Górski 
> Date: 10 February 2020 at 12:22:51 CET
> To: Bolke de Bruin 
> Cc: atlas , Mariusz Górski 
> 
> Subject: Review Request 72104: Enable logging to STDOUT in Atlas startup 
> scripts
> Reply-To: Mariusz Górski @reviews.apache.org
> 
> 
> This is an automatically generated e-mail. To reply, visit: 
> https://reviews.apache.org/r/72104/
> 
> Review request for atlas and Bolke de Bruin.
> By Mariusz Górski.
> Repository: atlas
> Description
> 
> This patch enables logging to STDOUT from processes spawned through 
> atlas_start.py or atlas_config.py scripts. For now, even if configured in 
> log4j, logging to STDOUT was supressed (by redirecting to log files only).
> Testing
> 
> The tests were concluded by:
> 
> Building & running Atlas locally with default log4j configuration and 
> ENABLE_LOGGING_TO_CONSOLE variable set to true.
> Building & running Atlas on Openshift with log4j configuration pushing 
> everything to STDOUT and ENABLE_LOGGING_TO_CONSOLE variable set to true.
> 
> In both scenarios logs were available both in the .out/.err files in the 
> Atlas logdir (backwards compatibility), but at the same time I was able to 
> see them in STDOUT of both running process (local installation) and pod 
> (openshift deployment).
> Diffs
> 
> distro/src/bin/atlas_config.py (f09026ff9)
> distro/src/bin/atlas_start.py (963faf402)
> View Diff


Re: Review Request 72104: Enable logging to STDOUT in Atlas startup scripts

2020-02-12 Thread Bolke de Bruin

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72104/#review219554
---


Ship it!




Ship It!

- Bolke de Bruin


On Feb. 10, 2020, 11:22 a.m., Mariusz Górski wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72104/
> ---
> 
> (Updated Feb. 10, 2020, 11:22 a.m.)
> 
> 
> Review request for atlas and Bolke de Bruin.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This patch enables logging to STDOUT from processes spawned through 
> atlas_start.py or atlas_config.py scripts. For now, even if configured in 
> log4j, logging to STDOUT was supressed (by redirecting to log files only).
> 
> 
> Diffs
> -
> 
>   distro/src/bin/atlas_config.py f09026ff9 
>   distro/src/bin/atlas_start.py 963faf402 
> 
> 
> Diff: https://reviews.apache.org/r/72104/diff/1/
> 
> 
> Testing
> ---
> 
> The tests were concluded by:
> 
> 1. Building & running Atlas locally with default log4j configuration and 
> ENABLE_LOGGING_TO_CONSOLE variable set to true.
> 2. Building & running Atlas on Openshift with log4j configuration pushing 
> everything to STDOUT and ENABLE_LOGGING_TO_CONSOLE variable set to true.
> 
> In both scenarios logs were available both in the .out/.err files in the 
> Atlas logdir (backwards compatibility), but at the same time I was able to 
> see them in STDOUT of both running process (local installation) and pod 
> (openshift deployment).
> 
> 
> Thanks,
> 
> Mariusz Górski
> 
>



[jira] [Commented] (ATLAS-3546) isOptional for composition relationship category?

2019-12-17 Thread Bolke de Bruin (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16998540#comment-16998540
 ] 

Bolke de Bruin commented on ATLAS-3546:
---

+1 on that last comment see also ATLAS-3254

> isOptional for composition relationship category?
> -
>
> Key: ATLAS-3546
> URL: https://issues.apache.org/jira/browse/ATLAS-3546
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: charles shen
>Priority: Major
> Attachments: json.jpg, xml.jpg, xml.jpg
>
>
> I noticed since 
> [ATLAS-3051|https://issues.apache.org/jira/browse/ATLAS-3051], the 
> relationship attribute must be specified in the end def which is not 
> container and relationship category is composition. 
> I understand it's to prohibit orphan children but is it too strong? Reason 
> below:
>  # I have to provide all the entities along the relationship path in the 
> payload when creating a child, eg, for RDBMS, I have to provide 
> rdbms_instance, rdbms_db, rdbms_table, rdbms_column where I just want to 
> create a single rdbms_column, it brings performance overhead to check 
> existence of rdbms_instance, rdbms_db, etc..., 
>  # I have defined a composition relationship type where each end is the same 
> entity type, it couldn't be created successfully anymore since it always 
> requires the mandatory attribute where it's the same type and then falls into 
> infinite loop.
> Three possible fixes:
>  # Remove the isOptional constraint, since ownedRef/inverseRef doesn't have 
> such constraint.
>  # Add isOptional to relationship type end def.
>  # Add option in Rest to ignore the isOptional constraint for relationship 
> type.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Comment Edited] (ATLAS-3254) Atlas entity with large array of refs causes performance issues for lineage

2019-12-17 Thread Bolke de Bruin (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16998538#comment-16998538
 ] 

Bolke de Bruin edited comment on ATLAS-3254 at 12/17/19 8:08 PM:
-

What [~mayank_nj] do you consider to load “properly”? What is the time taken to 
show the properties? What is the size of the json sent over the network (ours 
is > 27mb)? What is the load time? What is the render time?

Are you saying the loading of 200K objects in a pseudodir is taking over 1h? 
That is not “proper” I think?


was (Author: bolke):
What [~mayank_nj] do you consider to load “properly”? What is the time taken to 
show the properties? What is the size of the json sent over the network (ours 
is > 27mb)? What is the load time? What is the render time?

> Atlas entity with large array of refs causes performance issues for lineage
> ---
>
> Key: ATLAS-3254
> URL: https://issues.apache.org/jira/browse/ATLAS-3254
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core, atlas-webui
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Adam Rempter
>Assignee: Mayank Jain
>Priority: Major
>  Labels: performance
> Attachments: Screenshot 2019-11-28 at 21.18.44.png, 
> entity_auto_create.sh, example_create_entities.json, 
> rest_entity_get_pseudodir.json
>
>
> We use “aws_s3_pseudo_dir” type from 3020-aws_s3_typedefs.json model.
> It has following property: 
> "name":    "s3Objects",
> "typeName":    "array"
>  
> Now in AWS buckets you can have thousands of objects. This causes that 
> s3Objects array grows quite quickly, causing aws_s3_pseudo_dir entity Json to 
> rich easly few MBs.
>  
> Then we start seeing problems like:
>  * UI is dying on displaying entity properties or lineage
>  * Error in logs: audit record too long: entityType=aws_s3_pseudo_dir, 
> guid=24398271-6ba0-4db5-adfa-38e432dc55ce, size=1053931; maxSize=1048576. 
> entity attribute values not stored in audit (EntityAuditListenerV2:234)
>  * Some errors with write to HBase (java.lang.IllegalArgumentException: 
> KeyValue size too large, as workaround we set hbase.client.keyvalue.maxsize 
> param to 0)
>  * kafka consumer errors (we can of course set some parameters on consumer, 
> but I think it is just workaround)
> …
> Exception in NotificationHookConsumer (NotificationHookConsumer:332)
> org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be 
> completed since the group has already rebalanced and assigned the partitions 
> to another member. This means that the time between subsequen
> t calls to poll() was longer than the configured max.poll.interval.ms, which 
> typically implies that the poll loop is spending too much time message 
> processing. You can address this either by increasing the sessio
> n timeout or by reducing the maximum size of batches returned in poll() with 
> max.poll.records.
> …
> Specifying pseudo_dir is required for s3objects:
> name": "pseudoDirectory",
> "typeName": "aws_s3_pseudo_dir",
> "cardinality": "SINGLE",
> "isIndexable": false,
> *"isOptional": false,*
> "isUnique": false,
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (ATLAS-3254) Atlas entity with large array of refs causes performance issues for lineage

2019-12-17 Thread Bolke de Bruin (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16998538#comment-16998538
 ] 

Bolke de Bruin commented on ATLAS-3254:
---

What [~mayank_nj] do you consider to load “properly”? What is the time taken to 
show the properties? What is the size of the json sent over the network (ours 
is > 27mb)? What is the load time? What is the render time?

> Atlas entity with large array of refs causes performance issues for lineage
> ---
>
> Key: ATLAS-3254
> URL: https://issues.apache.org/jira/browse/ATLAS-3254
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core, atlas-webui
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Adam Rempter
>Assignee: Mayank Jain
>Priority: Major
>  Labels: performance
> Attachments: Screenshot 2019-11-28 at 21.18.44.png, 
> entity_auto_create.sh, example_create_entities.json, 
> rest_entity_get_pseudodir.json
>
>
> We use “aws_s3_pseudo_dir” type from 3020-aws_s3_typedefs.json model.
> It has following property: 
> "name":    "s3Objects",
> "typeName":    "array"
>  
> Now in AWS buckets you can have thousands of objects. This causes that 
> s3Objects array grows quite quickly, causing aws_s3_pseudo_dir entity Json to 
> rich easly few MBs.
>  
> Then we start seeing problems like:
>  * UI is dying on displaying entity properties or lineage
>  * Error in logs: audit record too long: entityType=aws_s3_pseudo_dir, 
> guid=24398271-6ba0-4db5-adfa-38e432dc55ce, size=1053931; maxSize=1048576. 
> entity attribute values not stored in audit (EntityAuditListenerV2:234)
>  * Some errors with write to HBase (java.lang.IllegalArgumentException: 
> KeyValue size too large, as workaround we set hbase.client.keyvalue.maxsize 
> param to 0)
>  * kafka consumer errors (we can of course set some parameters on consumer, 
> but I think it is just workaround)
> …
> Exception in NotificationHookConsumer (NotificationHookConsumer:332)
> org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be 
> completed since the group has already rebalanced and assigned the partitions 
> to another member. This means that the time between subsequen
> t calls to poll() was longer than the configured max.poll.interval.ms, which 
> typically implies that the poll loop is spending too much time message 
> processing. You can address this either by increasing the sessio
> n timeout or by reducing the maximum size of batches returned in poll() with 
> max.poll.records.
> …
> Specifying pseudo_dir is required for s3objects:
> name": "pseudoDirectory",
> "typeName": "aws_s3_pseudo_dir",
> "cardinality": "SINGLE",
> "isIndexable": false,
> *"isOptional": false,*
> "isUnique": false,
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3553) Integrity issue: Recreated deleted (propagated) classification has same entities

2019-12-08 Thread Bolke de Bruin (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin updated ATLAS-3553:
--
Affects Version/s: trunk
   2.0.0

> Integrity issue: Recreated deleted (propagated) classification has same 
> entities
> 
>
> Key: ATLAS-3553
> URL: https://issues.apache.org/jira/browse/ATLAS-3553
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 2.0.0, trunk
>    Reporter: Bolke de Bruin
>Priority: Blocker
>
> ATLAS-3545 solved the issue to the null pointer exception. The classification 
> still lists entities in the classification part of the UI. Now when trying to 
> remove the classification this works, but when recreating the classification 
> with the same name the same entities shows up again (same list).
> Another observation is that looking up the properties of the entity does not 
> show the classification so this is contrary what the classification UI is 
> saying. This reduces confidence in the security (i.e. can we rely on the 
> classifications) of Atlas.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (ATLAS-3553) Integrity issue: Recreated deleted (propagated) classification has same entities

2019-12-08 Thread Bolke de Bruin (Jira)
Bolke de Bruin created ATLAS-3553:
-

 Summary: Integrity issue: Recreated deleted (propagated) 
classification has same entities
 Key: ATLAS-3553
 URL: https://issues.apache.org/jira/browse/ATLAS-3553
 Project: Atlas
  Issue Type: Bug
Reporter: Bolke de Bruin


ATLAS-3545 solved the issue to the null pointer exception. The classification 
still lists entities in the classification part of the UI. Now when trying to 
remove the classification this works, but when recreating the classification 
with the same name the same entities shows up again (same list).

Another observation is that looking up the properties of the entity does not 
show the classification so this is contrary what the classification UI is 
saying. This reduces confidence in the security (i.e. can we rely on the 
classifications) of Atlas.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3552) Race condition while adding a classification

2019-12-08 Thread Bolke de Bruin (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin updated ATLAS-3552:
--
Description: 
Following up on ATLAS-3545

we set a classification on a aws_s3_bucket with several thousands (200K+) of 
objects. As the UI wasn’t clear if the classification was applied so we tried 
it a couple of times. This resulted in the UI displaying it 3  times

  was:
Following up on ATLAS-3454

we set a classification on a aws_s3_bucket with several thousands (200K+) of 
objects. As the UI wasn’t clear if the classification was applied so we tried 
it a couple of times. This resulted in the UI displaying it 3  times


> Race condition while adding a classification
> 
>
> Key: ATLAS-3552
> URL: https://issues.apache.org/jira/browse/ATLAS-3552
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 2.0.0, trunk
>        Reporter: Bolke de Bruin
>Priority: Critical
>
> Following up on ATLAS-3545
> we set a classification on a aws_s3_bucket with several thousands (200K+) of 
> objects. As the UI wasn’t clear if the classification was applied so we tried 
> it a couple of times. This resulted in the UI displaying it 3  times



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (ATLAS-3552) Race condition while adding a classification

2019-12-08 Thread Bolke de Bruin (Jira)
Bolke de Bruin created ATLAS-3552:
-

 Summary: Race condition while adding a classification
 Key: ATLAS-3552
 URL: https://issues.apache.org/jira/browse/ATLAS-3552
 Project: Atlas
  Issue Type: Bug
Affects Versions: 2.0.0, trunk
Reporter: Bolke de Bruin


Following up on ATLAS-3454

we set a classification on a aws_s3_bucket with several thousands (200K+) of 
objects. As the UI wasn’t clear if the classification was applied so we tried 
it a couple of times. This resulted in the UI displaying it 3  times



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (ATLAS-3254) Atlas entity with large array of refs causes performance issues for lineage

2019-12-05 Thread Bolke de Bruin (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16989021#comment-16989021
 ] 

Bolke de Bruin commented on ATLAS-3254:
---

1. Create aws_s3_bucket
2. Add 200K+ objects in the same pseudodir to this bucket
3. Try to load the properties from the pseudodirs


> Atlas entity with large array of refs causes performance issues for lineage
> ---
>
> Key: ATLAS-3254
> URL: https://issues.apache.org/jira/browse/ATLAS-3254
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core, atlas-webui
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Adam Rempter
>Assignee: Mayank Jain
>Priority: Major
>  Labels: performance
> Attachments: Screenshot 2019-11-28 at 21.18.44.png, 
> example_create_entities.json, rest_entity_get_pseudodir.json
>
>
> We use “aws_s3_pseudo_dir” type from 3020-aws_s3_typedefs.json model.
> It has following property: 
> "name":    "s3Objects",
> "typeName":    "array"
>  
> Now in AWS buckets you can have thousands of objects. This causes that 
> s3Objects array grows quite quickly, causing aws_s3_pseudo_dir entity Json to 
> rich easly few MBs.
>  
> Then we start seeing problems like:
>  * UI is dying on displaying entity properties or lineage
>  * Error in logs: audit record too long: entityType=aws_s3_pseudo_dir, 
> guid=24398271-6ba0-4db5-adfa-38e432dc55ce, size=1053931; maxSize=1048576. 
> entity attribute values not stored in audit (EntityAuditListenerV2:234)
>  * Some errors with write to HBase (java.lang.IllegalArgumentException: 
> KeyValue size too large, as workaround we set hbase.client.keyvalue.maxsize 
> param to 0)
>  * kafka consumer errors (we can of course set some parameters on consumer, 
> but I think it is just workaround)
> …
> Exception in NotificationHookConsumer (NotificationHookConsumer:332)
> org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be 
> completed since the group has already rebalanced and assigned the partitions 
> to another member. This means that the time between subsequen
> t calls to poll() was longer than the configured max.poll.interval.ms, which 
> typically implies that the poll loop is spending too much time message 
> processing. You can address this either by increasing the sessio
> n timeout or by reducing the maximum size of batches returned in poll() with 
> max.poll.records.
> …
> Specifying pseudo_dir is required for s3objects:
> name": "pseudoDirectory",
> "typeName": "aws_s3_pseudo_dir",
> "cardinality": "SINGLE",
> "isIndexable": false,
> *"isOptional": false,*
> "isUnique": false,
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (ATLAS-3545) NullPointerException while trying to delete classification

2019-12-03 Thread Bolke de Bruin (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987628#comment-16987628
 ] 

Bolke de Bruin commented on ATLAS-3545:
---

Thanks [~sarath]. We will apply the patch and observe the behavior. Please note 
that this only removes the symptom from search/retrieval. On entity update an 
invalid propagating classification is still being evaluated. That problem is 
worse as it creates performance issues. The question thus also becomes, apart 
from the race condition, how to delete the invalid vertex?

> NullPointerException while trying to delete classification
> --
>
> Key: ATLAS-3545
> URL: https://issues.apache.org/jira/browse/ATLAS-3545
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: trunk
>        Reporter: Bolke de Bruin
>Assignee: Sarath Subramanian
>Priority: Critical
> Attachments: ATLAS-3545.001.patch, Screenshot 2019-11-30 at 
> 22.16.44.png, Screenshot 2019-11-30 at 22.28.00.png, Screenshot 2019-12-02 at 
> 21.43.29.png
>
>
> We see an issue where there is a NullPointerException while trying to delete 
> a classification that is propagating.
> It seems (stack trace is as of yet unavailable) that it is caused due to a 
> typeName being NULL in AtlasTypeRegistry.getType. The UI looks fishy too see 
> screenshots.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (ATLAS-3547) Don’t evaluate propagating classifications for non relevant update

2019-12-03 Thread Bolke de Bruin (Jira)
Bolke de Bruin created ATLAS-3547:
-

 Summary: Don’t evaluate propagating classifications for non 
relevant update
 Key: ATLAS-3547
 URL: https://issues.apache.org/jira/browse/ATLAS-3547
 Project: Atlas
  Issue Type: Bug
Affects Versions: trunk
Reporter: Bolke de Bruin


In case of an aws_s3_bucket with a propagating classification and with several 
thousands (200K+) objects we are observing that every addition of an object to 
this aws_s3_bucket triggers a re-evaluation of all entities with the 
propagating tag. This exponentially increases the time that it takes a message 
to be consumed if the bucket is growing in size.

We send a create or update message for an aws_s3_object by Kafka which has all 
relevant information inside the message (ie. Aws_s3_bucket, aws_s3_pseudodir, 
aws_s3_object) as the producer system is unaware if the relevant metadata for 
the bucket and pseudodir are already available. This triggers an evaluation of 
the classifications and all propagating classifications. This seems over eager.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (ATLAS-3545) NullPointerException while trying to delete classification

2019-12-03 Thread Bolke de Bruin (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987118#comment-16987118
 ] 

Bolke de Bruin commented on ATLAS-3545:
---

[~ayubpathan] they were included above, but just to get it clear:

1. Add a propagating classification to an entity with a large container (e.g. 
aws_s3_bucket). 
2. Add the same propagating classification again before the UI shows the 
classification is applied
3. Observe 2 classifications have been applied with the same name
4. Remove one of the propagating classifications
5. A Try to remove the other one(s), this should fail
B Observe that the count in the UI basic search view is 1 or higher (e.g. 
it reads something like MANAGED (2) )
C Try to search for this classification with basic search, this should fail 
with an error message
D Try to read classification properties in the classification view, this 
should fail
E Observer that the propagating classification still is applied to derived 
entities (e.g. aws_s3_objects)

Obviously we are seeing multiple issues here (race condition, null pointer). 
Our goal is first to remove the “orphan” propagating classification

4. Observer

> NullPointerException while trying to delete classification
> --
>
> Key: ATLAS-3545
> URL: https://issues.apache.org/jira/browse/ATLAS-3545
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: trunk
>        Reporter: Bolke de Bruin
>Priority: Critical
> Attachments: Screenshot 2019-11-30 at 22.16.44.png, Screenshot 
> 2019-11-30 at 22.28.00.png, Screenshot 2019-12-02 at 21.43.29.png
>
>
> We see an issue where there is a NullPointerException while trying to delete 
> a classification that is propagating.
> It seems (stack trace is as of yet unavailable) that it is caused due to a 
> typeName being NULL in AtlasTypeRegistry.getType. The UI looks fishy too see 
> screenshots.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (ATLAS-3532) Test case failure in GremlinQueryComposerTest

2019-12-02 Thread Bolke de Bruin (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16986402#comment-16986402
 ] 

Bolke de Bruin commented on ATLAS-3532:
---

Sorry got confused I guess. Master looks okay.

> Test case failure in GremlinQueryComposerTest
> -
>
> Key: ATLAS-3532
> URL: https://issues.apache.org/jira/browse/ATLAS-3532
> Project: Atlas
>  Issue Type: Bug
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
> Attachments: 
> ATLAS-3532-DSL-Composer-update-to-address-_-test-cas.patch, 
> ATLAS-3532-Test-case-failure-in-GremlinQueryComposer.patch
>
>
> {code:java}
> Tests run: 39, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 15.464 sec 
> <<< FAILURE! - in 
> org.apache.atlas.query.GremlinQueryComposerTestwhereClauseTextContains(org.apache.atlas.query.GremlinQueryComposerTest)
>   Time elapsed: 0.697 sec  <<< FAILURE!java.lang.AssertionError: Table where 
> owner like "Tab*" expected [g.V().has('__typeName', 
> 'Table').has('Table.owner', 
> org.janusgraph.core.attribute.Text.textContainsRegex("Tab.*")).dedup().limit(25).toList()]
>  but found [g.V().has('__typeName', 'Table').has('Table.owner', 
> org.janusgraph.core.attribute.Text.textRegex("Tab.*")).dedup().limit(25).toList()]
>  at org.testng.Assert.fail(Assert.java:94)
>  at org.testng.Assert.failNotEquals(Assert.java:496)
>  at org.testng.Assert.assertEquals(Assert.java:125)
>  at org.testng.Assert.assertEquals(Assert.java:178)
>  at 
> org.apache.atlas.query.GremlinQueryComposerTest.verify(GremlinQueryComposerTest.java:362)
>  at 
> org.apache.atlas.query.GremlinQueryComposerTest.verify(GremlinQueryComposerTest.java:371)
>  at 
> org.apache.atlas.query.GremlinQueryComposerTest.whereClauseTextContains(GremlinQueryComposerTest.java:170)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (ATLAS-3545) NullPointerException while trying to delete classification

2019-12-02 Thread Bolke de Bruin (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16986354#comment-16986354
 ] 

Bolke de Bruin commented on ATLAS-3545:
---

[~madhan] latest screenshot gives a stack trace 

> NullPointerException while trying to delete classification
> --
>
> Key: ATLAS-3545
> URL: https://issues.apache.org/jira/browse/ATLAS-3545
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: trunk
>        Reporter: Bolke de Bruin
>Priority: Critical
> Attachments: Screenshot 2019-11-30 at 22.16.44.png, Screenshot 
> 2019-11-30 at 22.28.00.png, Screenshot 2019-12-02 at 21.43.29.png
>
>
> We see an issue where there is a NullPointerException while trying to delete 
> a classification that is propagating.
> It seems (stack trace is as of yet unavailable) that it is caused due to a 
> typeName being NULL in AtlasTypeRegistry.getType. The UI looks fishy too see 
> screenshots.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3545) NullPointerException while trying to delete classification

2019-12-02 Thread Bolke de Bruin (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin updated ATLAS-3545:
--
Attachment: Screenshot 2019-12-02 at 21.43.29.png

> NullPointerException while trying to delete classification
> --
>
> Key: ATLAS-3545
> URL: https://issues.apache.org/jira/browse/ATLAS-3545
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: trunk
>        Reporter: Bolke de Bruin
>Priority: Critical
> Attachments: Screenshot 2019-11-30 at 22.16.44.png, Screenshot 
> 2019-11-30 at 22.28.00.png, Screenshot 2019-12-02 at 21.43.29.png
>
>
> We see an issue where there is a NullPointerException while trying to delete 
> a classification that is propagating.
> It seems (stack trace is as of yet unavailable) that it is caused due to a 
> typeName being NULL in AtlasTypeRegistry.getType. The UI looks fishy too see 
> screenshots.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Comment Edited] (ATLAS-3305) Unable to scale atlas kafka consumers

2019-12-02 Thread Bolke de Bruin (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16986051#comment-16986051
 ] 

Bolke de Bruin edited comment on ATLAS-3305 at 12/2/19 1:34 PM:


I suggest using a different topic for this (ATLAS_PARTIONED) or having a 
different config item (token.partioned) to ensure administrators know what to 
do with this (making sure that the messages are atomic and/or partioned by key).




was (Author: bolke):
I suggest using a different topic for this (ATLAS_PARTIONED) to ensure 
administrators know what to do with this (making sure that the messages are 
atomic and/or partioned by key).



> Unable to scale atlas kafka consumers
> -
>
> Key: ATLAS-3305
> URL: https://issues.apache.org/jira/browse/ATLAS-3305
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core, atlas-intg
>Affects Versions: 1.1.0, 2.0.0
>Reporter: Adam Rempter
>Priority: Major
>  Labels: performance
> Attachments: ATLAS-3305_multiple_kafka_consumers.patch, 
> multiple_consumers_perf.png
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> We wanted to scale kafka consumers for atlas, as we are getting many lineage 
> messages and processing them just with one consumer is not enough. 
>  
> There is parameter atlas.notification.hook.numthreads to scale consumers in  
> NotificationHookConsumer.
> But the method:
>  
> notificationInterface.createConsumers(NotificationType.HOOK, numThreads)
>  
> is always returning one element list, which effectively always starts one 
> consumer
> List> consumers = 
> Collections.singletonList(kafkaConsumer);
>  
> Log incorrectly says that nuber of consumers has been created:
> LOG.info("<== KafkaNotification.createConsumers(notificationType={}, 
> numConsumers={}, autoCommitEnabled={})", notificationType, numConsumers, 
> autoCommitEnabled)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (ATLAS-3305) Unable to scale atlas kafka consumers

2019-12-02 Thread Bolke de Bruin (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16986051#comment-16986051
 ] 

Bolke de Bruin commented on ATLAS-3305:
---

I suggest using a different topic for this (ATLAS_PARTIONED) to ensure 
administrators know what to do with this (making sure that the messages are 
atomic and/or partioned by key).



> Unable to scale atlas kafka consumers
> -
>
> Key: ATLAS-3305
> URL: https://issues.apache.org/jira/browse/ATLAS-3305
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core, atlas-intg
>Affects Versions: 1.1.0, 2.0.0
>Reporter: Adam Rempter
>Priority: Major
>  Labels: performance
> Attachments: ATLAS-3305_multiple_kafka_consumers.patch, 
> multiple_consumers_perf.png
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> We wanted to scale kafka consumers for atlas, as we are getting many lineage 
> messages and processing them just with one consumer is not enough. 
>  
> There is parameter atlas.notification.hook.numthreads to scale consumers in  
> NotificationHookConsumer.
> But the method:
>  
> notificationInterface.createConsumers(NotificationType.HOOK, numThreads)
>  
> is always returning one element list, which effectively always starts one 
> consumer
> List> consumers = 
> Collections.singletonList(kafkaConsumer);
>  
> Log incorrectly says that nuber of consumers has been created:
> LOG.info("<== KafkaNotification.createConsumers(notificationType={}, 
> numConsumers={}, autoCommitEnabled={})", notificationType, numConsumers, 
> autoCommitEnabled)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (ATLAS-3545) NullPointerException while trying to delete classification

2019-12-01 Thread Bolke de Bruin (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16985659#comment-16985659
 ] 

Bolke de Bruin commented on ATLAS-3545:
---

Stack trace is unavailable at the moment as Atlas swallows the exception. We 
will patch it to make sure the whole trace is available.

We are observing 3 separate issues that are related. 1) we set a classification 
on a aws_s3_bucket with several thousands (200K+) of objects. As the UI wasn’t 
clear if the classification was applied so we tried it a couple of times. This 
resulted in the UI displaying it 3 (!) times. We then tried to remove one, 
which seemed to work but trying to remove the others fails. 

Moreover, the original classification still seems to be there as we face 
scaleability issues when updating the aws_s3_bucket in question: it then 
eagerly fetches all vertices that have a propagating tag which exponentially 
increases the time spend on this when s3_objects are added. This will be filed 
separately in a jira.

> NullPointerException while trying to delete classification
> --
>
> Key: ATLAS-3545
> URL: https://issues.apache.org/jira/browse/ATLAS-3545
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: trunk
>        Reporter: Bolke de Bruin
>Priority: Critical
> Attachments: Screenshot 2019-11-30 at 22.16.44.png, Screenshot 
> 2019-11-30 at 22.28.00.png
>
>
> We see an issue where there is a NullPointerException while trying to delete 
> a classification that is propagating.
> It seems (stack trace is as of yet unavailable) that it is caused due to a 
> typeName being NULL in AtlasTypeRegistry.getType. The UI looks fishy too see 
> screenshots.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: Review Request 71841: ATLAS-3261: added option to authorize notifications using username given in the message

2019-12-01 Thread Bolke de Bruin

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71841/#review218866
---


Ship it!




Ship It!

- Bolke de Bruin


On Nov. 27, 2019, 9:43 p.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71841/
> ---
> 
> (Updated Nov. 27, 2019, 9:43 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Bolke de Bruin, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3261
> https://issues.apache.org/jira/browse/ATLAS-3261
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Adam Rempter (arempter) provided the patch for this improvement in 
> https://github.com/apache/atlas/pull/58#issuecomment-514541803. This review 
> has further updates - to cache user-authencation for configured amount of 
> time (5 minutes by default), to reduce the performance impact of generating 
> authentication object from username.
> 
> 
> Diffs
> -
> 
>   pom.xml b2506e70e 
>   webapp/pom.xml 57cab62a3 
>   
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  41a6c2eff 
> 
> 
> Diff: https://reviews.apache.org/r/71841/diff/1/
> 
> 
> Testing
> ---
> 
> - manual verification of authorization based on username in notification 
> message
> - successful UT, IT runs
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>



[jira] [Updated] (ATLAS-3545) NullPointerException while trying to delete classification

2019-11-30 Thread Bolke de Bruin (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin updated ATLAS-3545:
--
Attachment: Screenshot 2019-11-30 at 22.16.44.png

> NullPointerException while trying to delete classification
> --
>
> Key: ATLAS-3545
> URL: https://issues.apache.org/jira/browse/ATLAS-3545
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: trunk
>        Reporter: Bolke de Bruin
>Priority: Critical
> Attachments: Screenshot 2019-11-30 at 22.16.44.png, Screenshot 
> 2019-11-30 at 22.28.00.png
>
>
> We see an issue where there is a NullPointerException while trying to delete 
> a classification that is propagating.
> It seems (stack trace is as of yet unavailable) that it is caused due to a 
> typeName being NULL in AtlasTypeRegistry.getType. The UI looks fishy too see 
> screenshots.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3545) NullPointerException while trying to delete classification

2019-11-30 Thread Bolke de Bruin (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin updated ATLAS-3545:
--
Attachment: Screenshot 2019-11-30 at 22.28.00.png

> NullPointerException while trying to delete classification
> --
>
> Key: ATLAS-3545
> URL: https://issues.apache.org/jira/browse/ATLAS-3545
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: trunk
>        Reporter: Bolke de Bruin
>Priority: Critical
> Attachments: Screenshot 2019-11-30 at 22.16.44.png, Screenshot 
> 2019-11-30 at 22.28.00.png
>
>
> We see an issue where there is a NullPointerException while trying to delete 
> a classification that is propagating.
> It seems (stack trace is as of yet unavailable) that it is caused due to a 
> typeName being NULL in AtlasTypeRegistry.getType. The UI looks fishy too see 
> screenshots.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (ATLAS-3545) NullPointerException while trying to delete classification

2019-11-30 Thread Bolke de Bruin (Jira)
Bolke de Bruin created ATLAS-3545:
-

 Summary: NullPointerException while trying to delete classification
 Key: ATLAS-3545
 URL: https://issues.apache.org/jira/browse/ATLAS-3545
 Project: Atlas
  Issue Type: Bug
Affects Versions: trunk
Reporter: Bolke de Bruin


We see an issue where there is a NullPointerException while trying to delete a 
classification that is propagating.

It seems (stack trace is as of yet unavailable) that it is caused due to a 
typeName being NULL in AtlasTypeRegistry.getType. The UI looks fishy too see 
screenshots.
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3544) NullPointerException in AtlasGraphUtilsV2.getStateAsString

2019-11-29 Thread Bolke de Bruin (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin updated ATLAS-3544:
--
Attachment: Screenshot 2019-11-29 at 19.06.47.png

> NullPointerException in AtlasGraphUtilsV2.getStateAsString
> --
>
> Key: ATLAS-3544
> URL: https://issues.apache.org/jira/browse/ATLAS-3544
> Project: Atlas
>  Issue Type: Bug
>        Reporter: Bolke de Bruin
>Assignee: Sidharth Kumar Mishra
>Priority: Major
> Attachments: Screenshot 2019-11-29 at 19.06.47.png
>
>
> ATLAS-3477 creates a NullPointerException in getStateAsString of 
> AtlasGraphUtilsV2



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (ATLAS-3544) NullPointerException in AtlasGraphUtilsV2.getStateAsString

2019-11-29 Thread Bolke de Bruin (Jira)
Bolke de Bruin created ATLAS-3544:
-

 Summary: NullPointerException in AtlasGraphUtilsV2.getStateAsString
 Key: ATLAS-3544
 URL: https://issues.apache.org/jira/browse/ATLAS-3544
 Project: Atlas
  Issue Type: Bug
Reporter: Bolke de Bruin
Assignee: Sidharth Kumar Mishra


ATLAS-3477 creates a NullPointerException in getStateAsString of 
AtlasGraphUtilsV2



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Comment Edited] (ATLAS-3254) Atlas entity with large array of refs causes performance issues for lineage

2019-11-28 Thread Bolke de Bruin (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16984631#comment-16984631
 ] 

Bolke de Bruin edited comment on ATLAS-3254 at 11/28/19 8:24 PM:
-

[~madhan] we see now serious issues with entity deletion. The attached 
screenshot shows 30min processing time. So this not only has an effect on the 
UI.

This is on master btw.


was (Author: bolke):
[~madhan] we see now serious issues with entity deletion. The attached 
screenshot shows 30min processing time. So this not only has an effect on the 
UI.

> Atlas entity with large array of refs causes performance issues for lineage
> ---
>
> Key: ATLAS-3254
> URL: https://issues.apache.org/jira/browse/ATLAS-3254
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core, atlas-webui
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Adam Rempter
>Priority: Major
>  Labels: performance
> Attachments: Screenshot 2019-11-28 at 21.18.44.png, 
> example_create_entities.json, rest_entity_get_pseudodir.json
>
>
> We use “aws_s3_pseudo_dir” type from 3020-aws_s3_typedefs.json model.
> It has following property: 
> "name":    "s3Objects",
> "typeName":    "array"
>  
> Now in AWS buckets you can have thousands of objects. This causes that 
> s3Objects array grows quite quickly, causing aws_s3_pseudo_dir entity Json to 
> rich easly few MBs.
>  
> Then we start seeing problems like:
>  * UI is dying on displaying entity properties or lineage
>  * Error in logs: audit record too long: entityType=aws_s3_pseudo_dir, 
> guid=24398271-6ba0-4db5-adfa-38e432dc55ce, size=1053931; maxSize=1048576. 
> entity attribute values not stored in audit (EntityAuditListenerV2:234)
>  * Some errors with write to HBase (java.lang.IllegalArgumentException: 
> KeyValue size too large, as workaround we set hbase.client.keyvalue.maxsize 
> param to 0)
>  * kafka consumer errors (we can of course set some parameters on consumer, 
> but I think it is just workaround)
> …
> Exception in NotificationHookConsumer (NotificationHookConsumer:332)
> org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be 
> completed since the group has already rebalanced and assigned the partitions 
> to another member. This means that the time between subsequen
> t calls to poll() was longer than the configured max.poll.interval.ms, which 
> typically implies that the poll loop is spending too much time message 
> processing. You can address this either by increasing the sessio
> n timeout or by reducing the maximum size of batches returned in poll() with 
> max.poll.records.
> …
> Specifying pseudo_dir is required for s3objects:
> name": "pseudoDirectory",
> "typeName": "aws_s3_pseudo_dir",
> "cardinality": "SINGLE",
> "isIndexable": false,
> *"isOptional": false,*
> "isUnique": false,
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (ATLAS-3254) Atlas entity with large array of refs causes performance issues for lineage

2019-11-28 Thread Bolke de Bruin (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16984631#comment-16984631
 ] 

Bolke de Bruin commented on ATLAS-3254:
---

[~madhan] we see now serious issues with entity deletion. The attached 
screenshot shows 30min processing time. So this not only has an effect on the 
UI.

> Atlas entity with large array of refs causes performance issues for lineage
> ---
>
> Key: ATLAS-3254
> URL: https://issues.apache.org/jira/browse/ATLAS-3254
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core, atlas-webui
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Adam Rempter
>Priority: Major
>  Labels: performance
> Attachments: Screenshot 2019-11-28 at 21.18.44.png, 
> example_create_entities.json, rest_entity_get_pseudodir.json
>
>
> We use “aws_s3_pseudo_dir” type from 3020-aws_s3_typedefs.json model.
> It has following property: 
> "name":    "s3Objects",
> "typeName":    "array"
>  
> Now in AWS buckets you can have thousands of objects. This causes that 
> s3Objects array grows quite quickly, causing aws_s3_pseudo_dir entity Json to 
> rich easly few MBs.
>  
> Then we start seeing problems like:
>  * UI is dying on displaying entity properties or lineage
>  * Error in logs: audit record too long: entityType=aws_s3_pseudo_dir, 
> guid=24398271-6ba0-4db5-adfa-38e432dc55ce, size=1053931; maxSize=1048576. 
> entity attribute values not stored in audit (EntityAuditListenerV2:234)
>  * Some errors with write to HBase (java.lang.IllegalArgumentException: 
> KeyValue size too large, as workaround we set hbase.client.keyvalue.maxsize 
> param to 0)
>  * kafka consumer errors (we can of course set some parameters on consumer, 
> but I think it is just workaround)
> …
> Exception in NotificationHookConsumer (NotificationHookConsumer:332)
> org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be 
> completed since the group has already rebalanced and assigned the partitions 
> to another member. This means that the time between subsequen
> t calls to poll() was longer than the configured max.poll.interval.ms, which 
> typically implies that the poll loop is spending too much time message 
> processing. You can address this either by increasing the sessio
> n timeout or by reducing the maximum size of batches returned in poll() with 
> max.poll.records.
> …
> Specifying pseudo_dir is required for s3objects:
> name": "pseudoDirectory",
> "typeName": "aws_s3_pseudo_dir",
> "cardinality": "SINGLE",
> "isIndexable": false,
> *"isOptional": false,*
> "isUnique": false,
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: Review Request 71841: ATLAS-3261: added option to authorize notifications using username given in the message

2019-11-28 Thread Bolke de Bruin

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71841/#review218845
---



will authorization fail if authentication is null? if it doesnt should that 
notbe configurable?

- Bolke de Bruin


On nov 27, 2019, 9:43 p.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71841/
> ---
> 
> (Updated nov 27, 2019, 9:43 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Bolke de Bruin, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3261
> https://issues.apache.org/jira/browse/ATLAS-3261
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Adam Rempter (arempter) provided the patch for this improvement in 
> https://github.com/apache/atlas/pull/58#issuecomment-514541803. This review 
> has further updates - to cache user-authencation for configured amount of 
> time (5 minutes by default), to reduce the performance impact of generating 
> authentication object from username.
> 
> 
> Diffs
> -
> 
>   pom.xml b2506e70e 
>   webapp/pom.xml 57cab62a3 
>   
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  41a6c2eff 
> 
> 
> Diff: https://reviews.apache.org/r/71841/diff/1/
> 
> 
> Testing
> ---
> 
> - manual verification of authorization based on username in notification 
> message
> - successful UT, IT runs
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>



[jira] [Reopened] (ATLAS-3532) Test case failure in GremlinQueryComposerTest

2019-11-25 Thread Bolke de Bruin (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin reopened ATLAS-3532:
---

> Test case failure in GremlinQueryComposerTest
> -
>
> Key: ATLAS-3532
> URL: https://issues.apache.org/jira/browse/ATLAS-3532
> Project: Atlas
>  Issue Type: Bug
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
> Attachments: 
> ATLAS-3532-Test-case-failure-in-GremlinQueryComposer.patch
>
>
> {code:java}
> Tests run: 39, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 15.464 sec 
> <<< FAILURE! - in 
> org.apache.atlas.query.GremlinQueryComposerTestwhereClauseTextContains(org.apache.atlas.query.GremlinQueryComposerTest)
>   Time elapsed: 0.697 sec  <<< FAILURE!java.lang.AssertionError: Table where 
> owner like "Tab*" expected [g.V().has('__typeName', 
> 'Table').has('Table.owner', 
> org.janusgraph.core.attribute.Text.textContainsRegex("Tab.*")).dedup().limit(25).toList()]
>  but found [g.V().has('__typeName', 'Table').has('Table.owner', 
> org.janusgraph.core.attribute.Text.textRegex("Tab.*")).dedup().limit(25).toList()]
>  at org.testng.Assert.fail(Assert.java:94)
>  at org.testng.Assert.failNotEquals(Assert.java:496)
>  at org.testng.Assert.assertEquals(Assert.java:125)
>  at org.testng.Assert.assertEquals(Assert.java:178)
>  at 
> org.apache.atlas.query.GremlinQueryComposerTest.verify(GremlinQueryComposerTest.java:362)
>  at 
> org.apache.atlas.query.GremlinQueryComposerTest.verify(GremlinQueryComposerTest.java:371)
>  at 
> org.apache.atlas.query.GremlinQueryComposerTest.whereClauseTextContains(GremlinQueryComposerTest.java:170)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (ATLAS-3532) Test case failure in GremlinQueryComposerTest

2019-11-25 Thread Bolke de Bruin (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16981747#comment-16981747
 ] 

Bolke de Bruin commented on ATLAS-3532:
---

This does not seem to be the right chance as there is no test for 
textContainsRegex anymore now afaik. Does the query contain the special '_' so 
it warrants switching to textRegex which is *slow*?

> Test case failure in GremlinQueryComposerTest
> -
>
> Key: ATLAS-3532
> URL: https://issues.apache.org/jira/browse/ATLAS-3532
> Project: Atlas
>  Issue Type: Bug
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
> Attachments: 
> ATLAS-3532-Test-case-failure-in-GremlinQueryComposer.patch
>
>
> {code:java}
> Tests run: 39, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 15.464 sec 
> <<< FAILURE! - in 
> org.apache.atlas.query.GremlinQueryComposerTestwhereClauseTextContains(org.apache.atlas.query.GremlinQueryComposerTest)
>   Time elapsed: 0.697 sec  <<< FAILURE!java.lang.AssertionError: Table where 
> owner like "Tab*" expected [g.V().has('__typeName', 
> 'Table').has('Table.owner', 
> org.janusgraph.core.attribute.Text.textContainsRegex("Tab.*")).dedup().limit(25).toList()]
>  but found [g.V().has('__typeName', 'Table').has('Table.owner', 
> org.janusgraph.core.attribute.Text.textRegex("Tab.*")).dedup().limit(25).toList()]
>  at org.testng.Assert.fail(Assert.java:94)
>  at org.testng.Assert.failNotEquals(Assert.java:496)
>  at org.testng.Assert.assertEquals(Assert.java:125)
>  at org.testng.Assert.assertEquals(Assert.java:178)
>  at 
> org.apache.atlas.query.GremlinQueryComposerTest.verify(GremlinQueryComposerTest.java:362)
>  at 
> org.apache.atlas.query.GremlinQueryComposerTest.verify(GremlinQueryComposerTest.java:371)
>  at 
> org.apache.atlas.query.GremlinQueryComposerTest.whereClauseTextContains(GremlinQueryComposerTest.java:170)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (ATLAS-3420) New logo for Apache Atlas

2019-09-24 Thread Bolke de Bruin (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936433#comment-16936433
 ] 

Bolke de Bruin commented on ATLAS-3420:
---

[~sarath] if you mean by that that i should upload a vector image of the logo, 
i can ask my team to design one. It will look different of course. I was 
(wrongfully?) assuming that the basis for these images was a vector image.

let me know.

> New logo for Apache Atlas
> -
>
> Key: ATLAS-3420
> URL: https://issues.apache.org/jira/browse/ATLAS-3420
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Keval Bhatt
>Assignee: Keval Bhatt
>Priority: Major
> Attachments: Atlas_Favicon.ico, Atlas_Logo.png, 
> Atlas_Logo_monogram.png
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (ATLAS-3420) New logo for Apache Atlas

2019-09-23 Thread Bolke de Bruin (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935607#comment-16935607
 ] 

Bolke de Bruin commented on ATLAS-3420:
---

Looks cool! Can I suggest a vector image?

> New logo for Apache Atlas
> -
>
> Key: ATLAS-3420
> URL: https://issues.apache.org/jira/browse/ATLAS-3420
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Keval Bhatt
>Assignee: Keval Bhatt
>Priority: Major
> Attachments: Atlas_Favicon.ico, Atlas_Logo.png, 
> Atlas_Logo_monogram.png
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: Review Request 71431: Add indexed order by (sort) to basic search

2019-09-19 Thread Bolke de Bruin

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71431/
---

(Updated Sept. 19, 2019, 6:05 p.m.)


Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
Madhan Neethiraj, and Nixon Rodrigues.


Changes
---

Removed tag attribute sort for now.


Bugs: https://issues.apache.org/jira/browse/ATLAS-3399

https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-3399


Repository: atlas


Description
---

Add indexed order by to basic search.


Diffs (updated)
-

  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
 f3722b827 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusIndexQuery.java
 7c258b7a3 
  graphdb/janus/src/main/java/org/janusgraph/diskstorage/solr/Solr6Index.java 
e457866af 
  intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 
aac6b5aa3 
  
repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java
 479ddfd89 
  
repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java 
f7d8f08c7 
  repository/src/main/java/org/apache/atlas/discovery/SearchContext.java 
9bd0382eb 
  webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java e6b338b84 
  webapp/src/test/java/org/apache/atlas/web/integration/BasicSearchIT.java 
06931b38c 
  webapp/src/test/resources/json/search-parameters/entity-filters.json 
f4b2efe63 
  webapp/src/test/resources/json/search-parameters/tag-filters.json 5e74328d6 


Diff: https://reviews.apache.org/r/71431/diff/7/

Changes: https://reviews.apache.org/r/71431/diff/6-7/


Testing
---

- production and tests added


Thanks,

Bolke de Bruin



Fwd: Review Request 71431: Add indexed order by (sort) to basic search

2019-09-18 Thread Bolke de Bruin
Friendly ping.

Sent from my iPhone

Begin forwarded message:

> From: Bolke de Bruin 
> Date: 16 September 2019 at 21:11:17 CEST
> To: Aadarsh Jajodia , Ashutosh Mestry 
> , Sridhar K , Le Ma 
> , Nixon Rodrigues 
> , Madhan Neethiraj 
> Cc: Sarath Subramanian , atlas 
> , Bolke de Bruin 
> Subject: Re: Review Request 71431: Add indexed order by (sort) to basic search
> Reply-To: Bolke de Bruin 
> 
> 
> This is an automatically generated e-mail. To reply, visit: 
> https://reviews.apache.org/r/71431/
> 
> On September 16th, 2019, 7:05 p.m. UTC, Madhan Neethiraj wrote:
> 
> repository/src/main/java/org/apache/atlas/discovery/SearchContext.java (Diff 
> revision 4)
> 272   
> FilterCriteria filterCriteria = new FilterCriteria();
> Instead of creating FilterCriteria, consider using lines #263 - #265 here:
>   if (StringUtils.isNotEmpty(attributeName) && 
> structType.getAttributeType(attributeName) == null) {
> throw new AtlasBaseException(AtlasErrorCode.UNKNOWN_ATTRIBUTE, 
> attributeName, structType.getTypeName());
> 
> Better yet, update 'else' block at #260 to call this method:
>   } else {
> validateAttribute(structType, filterCriteria.getAttributeName();
>   }
> Nice. I was struggling with that a bit. Fixed.
> 
> - Bolke
> 
> 
> On September 16th, 2019, 7:10 p.m. UTC, Bolke de Bruin wrote:
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
> Madhan Neethiraj, and Nixon Rodrigues.
> By Bolke de Bruin.
> Updated Sept. 16, 2019, 7:10 p.m.
> 
> Bugs: https://issues.apache.org/jira/browse/ATLAS-3399
> Repository: atlas
> Description
> 
> Add indexed order by to basic search.
> Testing
> 
> production and tests added
> Diffs
> 
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
>  (f3722b827)
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusIndexQuery.java
>  (7c258b7a3)
> graphdb/janus/src/main/java/org/janusgraph/diskstorage/solr/Solr6Index.java 
> (e457866af)
> intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 
> (aac6b5aa3)
> repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java
>  (479ddfd89)
> repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java
>  (f7d8f08c7)
> repository/src/main/java/org/apache/atlas/discovery/SearchContext.java 
> (9bd0382eb)
> webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java (e6b338b84)
> webapp/src/test/java/org/apache/atlas/web/integration/BasicSearchIT.java 
> (06931b38c)
> webapp/src/test/resources/json/search-parameters/entity-filters.json 
> (f4b2efe63)
> webapp/src/test/resources/json/search-parameters/tag-filters.json (5e74328d6)
> View Diff


Re: Review Request 71431: Add indexed order by (sort) to basic search

2019-09-16 Thread Bolke de Bruin

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71431/
---

(Updated Sept. 16, 2019, 7:10 p.m.)


Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
Madhan Neethiraj, and Nixon Rodrigues.


Changes
---

Addressed comment.


Bugs: https://issues.apache.org/jira/browse/ATLAS-3399

https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-3399


Repository: atlas


Description
---

Add indexed order by to basic search.


Diffs (updated)
-

  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
 f3722b827 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusIndexQuery.java
 7c258b7a3 
  graphdb/janus/src/main/java/org/janusgraph/diskstorage/solr/Solr6Index.java 
e457866af 
  intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 
aac6b5aa3 
  
repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java
 479ddfd89 
  
repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java 
f7d8f08c7 
  repository/src/main/java/org/apache/atlas/discovery/SearchContext.java 
9bd0382eb 
  webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java e6b338b84 
  webapp/src/test/java/org/apache/atlas/web/integration/BasicSearchIT.java 
06931b38c 
  webapp/src/test/resources/json/search-parameters/entity-filters.json 
f4b2efe63 
  webapp/src/test/resources/json/search-parameters/tag-filters.json 5e74328d6 


Diff: https://reviews.apache.org/r/71431/diff/6/

Changes: https://reviews.apache.org/r/71431/diff/5-6/


Testing
---

- production and tests added


Thanks,

Bolke de Bruin



Re: Review Request 71431: Add indexed order by (sort) to basic search

2019-09-16 Thread Bolke de Bruin

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71431/
---

(Updated Sept. 16, 2019, 7:03 p.m.)


Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
Madhan Neethiraj, and Nixon Rodrigues.


Changes
---

Addressed comments. Only verification of attributename against entity


Bugs: https://issues.apache.org/jira/browse/ATLAS-3399

https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-3399


Repository: atlas


Description
---

Add indexed order by to basic search.


Diffs (updated)
-

  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
 f3722b827 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusIndexQuery.java
 7c258b7a3 
  graphdb/janus/src/main/java/org/janusgraph/diskstorage/solr/Solr6Index.java 
e457866af 
  intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 
aac6b5aa3 
  
repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java
 479ddfd89 
  
repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java 
f7d8f08c7 
  repository/src/main/java/org/apache/atlas/discovery/SearchContext.java 
9bd0382eb 
  webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java e6b338b84 
  webapp/src/test/java/org/apache/atlas/web/integration/BasicSearchIT.java 
06931b38c 
  webapp/src/test/resources/json/search-parameters/entity-filters.json 
f4b2efe63 
  webapp/src/test/resources/json/search-parameters/tag-filters.json 5e74328d6 


Diff: https://reviews.apache.org/r/71431/diff/5/

Changes: https://reviews.apache.org/r/71431/diff/4-5/


Testing
---

- production and tests added


Thanks,

Bolke de Bruin



Re: Review Request 71431: Add indexed order by (sort) to basic search

2019-09-16 Thread Bolke de Bruin


> On Sept. 14, 2019, 7:16 p.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java
> > Lines 205 (patched)
> > <https://reviews.apache.org/r/71431/diff/2/?file=2165082#file2165082line205>
> >
> > graphQuery requires 'qualifiedName' of the attribute (i.e. Asset.name, 
> > Asset.owner) -  refer to SearchProcessor.toGraphFilterQuery(). Is 'sortBy' 
> > here qualifiedName of the attribute?
> 
> Bolke de Bruin wrote:
> Good point. It seems (but please correct me if I am wrong) given the 
> tests that it is not required. However, I assume I will need to update 
> SearchParameters to validate whether the attribute exists?
> 
> Madhan Neethiraj wrote:
> Validation of attribute names is handled in 
> SearchContext.validateAttributes(); consider updating this method to validate 
> the new field SearchParameters.sortBy.
> 
> Bolke de Bruin wrote:
> Can you search on classificationType and entityType at the same time? I'm 
> assuming that I need to verify the sortBy parameter against entityType and 
> classifictionType. It seems a bit odd though

Ah used the ui to find out a bit more. Now I am going to assume I just need to 
validate against entityType.


- Bolke


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71431/#review217738
-------


On Sept. 16, 2019, 6:45 p.m., Bolke de Bruin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71431/
> ---
> 
> (Updated Sept. 16, 2019, 6:45 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
> Madhan Neethiraj, and Nixon Rodrigues.
> 
> 
> Bugs: https://issues.apache.org/jira/browse/ATLAS-3399
> 
> https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-3399
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Add indexed order by to basic search.
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
>  f3722b827 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusIndexQuery.java
>  7c258b7a3 
>   graphdb/janus/src/main/java/org/janusgraph/diskstorage/solr/Solr6Index.java 
> e457866af 
>   intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 
> aac6b5aa3 
>   
> repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java
>  479ddfd89 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java
>  f7d8f08c7 
>   repository/src/main/java/org/apache/atlas/discovery/SearchContext.java 
> 9bd0382eb 
>   webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java e6b338b84 
>   webapp/src/test/java/org/apache/atlas/web/integration/BasicSearchIT.java 
> 06931b38c 
>   webapp/src/test/resources/json/search-parameters/entity-filters.json 
> f4b2efe63 
>   webapp/src/test/resources/json/search-parameters/tag-filters.json 5e74328d6 
> 
> 
> Diff: https://reviews.apache.org/r/71431/diff/4/
> 
> 
> Testing
> ---
> 
> - production and tests added
> 
> 
> Thanks,
> 
> Bolke de Bruin
> 
>



Re: Review Request 71431: Add indexed order by (sort) to basic search

2019-09-16 Thread Bolke de Bruin

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71431/
---

(Updated Sept. 16, 2019, 6:45 p.m.)


Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
Madhan Neethiraj, and Nixon Rodrigues.


Changes
---

Addressed other comments. 

- Question: how to test the classification orderby properly. What 'attribute 
name' is possible?


Bugs: https://issues.apache.org/jira/browse/ATLAS-3399

https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-3399


Repository: atlas


Description
---

Add indexed order by to basic search.


Diffs (updated)
-

  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
 f3722b827 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusIndexQuery.java
 7c258b7a3 
  graphdb/janus/src/main/java/org/janusgraph/diskstorage/solr/Solr6Index.java 
e457866af 
  intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 
aac6b5aa3 
  
repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java
 479ddfd89 
  
repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java 
f7d8f08c7 
  repository/src/main/java/org/apache/atlas/discovery/SearchContext.java 
9bd0382eb 
  webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java e6b338b84 
  webapp/src/test/java/org/apache/atlas/web/integration/BasicSearchIT.java 
06931b38c 
  webapp/src/test/resources/json/search-parameters/entity-filters.json 
f4b2efe63 
  webapp/src/test/resources/json/search-parameters/tag-filters.json 5e74328d6 


Diff: https://reviews.apache.org/r/71431/diff/4/

Changes: https://reviews.apache.org/r/71431/diff/3-4/


Testing
---

- production and tests added


Thanks,

Bolke de Bruin



Re: Review Request 71431: Add indexed order by (sort) to basic search

2019-09-16 Thread Bolke de Bruin

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71431/
---

(Updated Sept. 16, 2019, 6:38 p.m.)


Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
Madhan Neethiraj, and Nixon Rodrigues.


Changes
---

addressing comments


Bugs: https://issues.apache.org/jira/browse/ATLAS-3399

https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-3399


Repository: atlas


Description
---

Add indexed order by to basic search.


Diffs (updated)
-

  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
 f3722b827 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusIndexQuery.java
 7c258b7a3 
  graphdb/janus/src/main/java/org/janusgraph/diskstorage/solr/Solr6Index.java 
e457866af 
  intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 
aac6b5aa3 
  
repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java
 479ddfd89 
  
repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java 
f7d8f08c7 
  repository/src/main/java/org/apache/atlas/discovery/SearchContext.java 
9bd0382eb 
  webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java e6b338b84 
  webapp/src/test/java/org/apache/atlas/web/integration/BasicSearchIT.java 
06931b38c 
  webapp/src/test/resources/json/search-parameters/entity-filters.json 
f4b2efe63 
  webapp/src/test/resources/json/search-parameters/tag-filters.json 5e74328d6 


Diff: https://reviews.apache.org/r/71431/diff/3/

Changes: https://reviews.apache.org/r/71431/diff/2-3/


Testing
---

- production and tests added


Thanks,

Bolke de Bruin



Re: Review Request 71431: Add indexed order by (sort) to basic search

2019-09-16 Thread Bolke de Bruin


> On Sept. 14, 2019, 7:16 p.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java
> > Lines 205 (patched)
> > <https://reviews.apache.org/r/71431/diff/2/?file=2165082#file2165082line205>
> >
> > graphQuery requires 'qualifiedName' of the attribute (i.e. Asset.name, 
> > Asset.owner) -  refer to SearchProcessor.toGraphFilterQuery(). Is 'sortBy' 
> > here qualifiedName of the attribute?
> 
> Bolke de Bruin wrote:
> Good point. It seems (but please correct me if I am wrong) given the 
> tests that it is not required. However, I assume I will need to update 
> SearchParameters to validate whether the attribute exists?
> 
> Madhan Neethiraj wrote:
> Validation of attribute names is handled in 
> SearchContext.validateAttributes(); consider updating this method to validate 
> the new field SearchParameters.sortBy.

Can you search on classificationType and entityType at the same time? I'm 
assuming that I need to verify the sortBy parameter against entityType and 
classifictionType. It seems a bit odd though


- Bolke


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71431/#review217738
-------


On Sept. 14, 2019, 12:45 p.m., Bolke de Bruin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71431/
> ---
> 
> (Updated Sept. 14, 2019, 12:45 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
> Madhan Neethiraj, and Nixon Rodrigues.
> 
> 
> Bugs: https://issues.apache.org/jira/browse/ATLAS-3399
> 
> https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-3399
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Add indexed order by to basic search.
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
>  f3722b827 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusIndexQuery.java
>  7c258b7a3 
>   graphdb/janus/src/main/java/org/janusgraph/diskstorage/solr/Solr6Index.java 
> e457866af 
>   intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 
> aac6b5aa3 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java
>  f7d8f08c7 
>   webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java e6b338b84 
>   webapp/src/test/java/org/apache/atlas/web/integration/BasicSearchIT.java 
> 06931b38c 
>   webapp/src/test/resources/json/search-parameters/entity-filters.json 
> f4b2efe63 
> 
> 
> Diff: https://reviews.apache.org/r/71431/diff/2/
> 
> 
> Testing
> ---
> 
> - production and tests added
> 
> 
> Thanks,
> 
> Bolke de Bruin
> 
>



Re: Review Request 71431: Add indexed order by (sort) to basic search

2019-09-15 Thread Bolke de Bruin


> On Sept. 14, 2019, 7:16 p.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java
> > Line 50 (original), 57 (patched)
> > <https://reviews.apache.org/r/71431/diff/2/?file=2165082#file2165082line57>
> >
> > ClassificationSearchProcessor needs to be updated as well to support 
> > sortBy. Please review and update.

Good catch. Im planning to add the same logic as in EntitySearchProcessor and 
add it to "tagGraphQueryWithAttributes" and "entityGraphQueryTraitNames". Does 
that make sense?


> On Sept. 14, 2019, 7:16 p.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java
> > Lines 205 (patched)
> > <https://reviews.apache.org/r/71431/diff/2/?file=2165082#file2165082line205>
> >
> > graphQuery requires 'qualifiedName' of the attribute (i.e. Asset.name, 
> > Asset.owner) -  refer to SearchProcessor.toGraphFilterQuery(). Is 'sortBy' 
> > here qualifiedName of the attribute?

Good point. It seems (but please correct me if I am wrong) given the tests that 
it is not required. However, I assume I will need to update SearchParameters to 
validate whether the attribute exists?


- Bolke


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71431/#review217738
---


On Sept. 14, 2019, 12:45 p.m., Bolke de Bruin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71431/
> ---
> 
> (Updated Sept. 14, 2019, 12:45 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
> Madhan Neethiraj, and Nixon Rodrigues.
> 
> 
> Bugs: https://issues.apache.org/jira/browse/ATLAS-3399
> 
> https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-3399
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Add indexed order by to basic search.
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
>  f3722b827 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusIndexQuery.java
>  7c258b7a3 
>   graphdb/janus/src/main/java/org/janusgraph/diskstorage/solr/Solr6Index.java 
> e457866af 
>   intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 
> aac6b5aa3 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java
>  f7d8f08c7 
>   webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java e6b338b84 
>   webapp/src/test/java/org/apache/atlas/web/integration/BasicSearchIT.java 
> 06931b38c 
>   webapp/src/test/resources/json/search-parameters/entity-filters.json 
> f4b2efe63 
> 
> 
> Diff: https://reviews.apache.org/r/71431/diff/2/
> 
> 
> Testing
> ---
> 
> - production and tests added
> 
> 
> Thanks,
> 
> Bolke de Bruin
> 
>



Re: Review Request 71282: Expose total count of index searches

2019-09-15 Thread Bolke de Bruin

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71282/
---

(Updated Sept. 15, 2019, 8:10 a.m.)


Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
Madhan Neethiraj, and Nixon Rodrigues.


Changes
---

Addressed comments.


Bugs: https://issues.apache.org/jira/browse/ATLAS-3367

https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-3367


Repository: atlas


Description
---

Both ElasticSearch and SOLR can expose the total count
of results without returning all results. This is useful
to give the user or client an idea how many results there are.

This patch ensures the total is returned if available. This total
is an approximate as scrubbing of the results still needs to happen.
Therefore, one should not only rely on this information to provide
,for example, pagination.


Diffs (updated)
-

  intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResult.java 
d274cc07e 
  
repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java
 479ddfd89 
  
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 
f667aa399 
  
repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java 
f7d8f08c7 
  
repository/src/main/java/org/apache/atlas/discovery/FreeTextSearchProcessor.java
 1dd1afaa3 
  
repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java
 0ffd61c07 
  repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 
f7847edff 
  repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java 
c253d544b 


Diff: https://reviews.apache.org/r/71282/diff/5/

Changes: https://reviews.apache.org/r/71282/diff/4-5/


Testing
---


Thanks,

Bolke de Bruin



Re: Review Request 71282: Expose total count of index searches

2019-09-14 Thread Bolke de Bruin

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71282/
---

(Updated Sept. 14, 2019, 6:03 p.m.)


Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
Madhan Neethiraj, and Nixon Rodrigues.


Changes
---

Use -1 to initialize


Bugs: https://issues.apache.org/jira/browse/ATLAS-3367

https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-3367


Repository: atlas


Description
---

Both ElasticSearch and SOLR can expose the total count
of results without returning all results. This is useful
to give the user or client an idea how many results there are.

This patch ensures the total is returned if available. This total
is an approximate as scrubbing of the results still needs to happen.
Therefore, one should not only rely on this information to provide
,for example, pagination.


Diffs (updated)
-

  intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResult.java 
d274cc07e 
  
repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java
 479ddfd89 
  
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 
f667aa399 
  
repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java 
f7d8f08c7 
  
repository/src/main/java/org/apache/atlas/discovery/FreeTextSearchProcessor.java
 1dd1afaa3 
  
repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java
 0ffd61c07 
  repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 
f7847edff 
  repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java 
c253d544b 


Diff: https://reviews.apache.org/r/71282/diff/4/

Changes: https://reviews.apache.org/r/71282/diff/3-4/


Testing
---


Thanks,

Bolke de Bruin



Re: Review Request 71282: Expose total count of index searches

2019-09-14 Thread Bolke de Bruin


> On Sept. 14, 2019, 4:53 p.m., Madhan Neethiraj wrote:
> > intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResult.java
> > Lines 71 (patched)
> > <https://reviews.apache.org/r/71282/diff/3/?file=2161896#file2161896line71>
> >
> > I suggest to initialize approximateCount to "-1", to enable 
> > distinguishing "empty results" from "unavailability of approximate count".

Will do.


> On Sept. 14, 2019, 4:53 p.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java
> > Lines 131 (patched)
> > <https://reviews.apache.org/r/71282/diff/3/?file=2161902#file2161902line131>
> >
> > - given "getResultCount()" can only return approximate count, that too 
> > not in all cases, consider renaming this to "getApproximateResultCount()"
> > - and the implementations should return "-1" when approximate count is 
> > not available

That's not true in this case although maybe a bit semantic. This call does get 
the total result count from the search backend scrubbing happens only after the 
API call. I suggest leaving it as it might be used in different circumstances 
in the future (hypothetical: if it would become possible to do scrubbing in the 
search engine for example).


- Bolke


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71282/#review217737
---


On Sept. 14, 2019, 12:46 p.m., Bolke de Bruin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71282/
> ---
> 
> (Updated Sept. 14, 2019, 12:46 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
> Madhan Neethiraj, and Nixon Rodrigues.
> 
> 
> Bugs: https://issues.apache.org/jira/browse/ATLAS-3367
> 
> https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-3367
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Both ElasticSearch and SOLR can expose the total count
> of results without returning all results. This is useful
> to give the user or client an idea how many results there are.
> 
> This patch ensures the total is returned if available. This total
> is an approximate as scrubbing of the results still needs to happen.
> Therefore, one should not only rely on this information to provide
> ,for example, pagination.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResult.java 
> d274cc07e 
>   
> repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java
>  479ddfd89 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  c67e34772 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java
>  f7d8f08c7 
>   
> repository/src/main/java/org/apache/atlas/discovery/FreeTextSearchProcessor.java
>  1dd1afaa3 
>   
> repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java
>  0ffd61c07 
>   repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 
> a7ccaebc8 
>   
> repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java 
> c253d544b 
> 
> 
> Diff: https://reviews.apache.org/r/71282/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Bolke de Bruin
> 
>



Re: Review Request 71278: Set log level to warn for relationships without relationship definition

2019-09-14 Thread Bolke de Bruin

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71278/
---

(Updated Sept. 14, 2019, 1:09 p.m.)


Review request for atlas, Nixon Rodrigues and Saqeeb Shaikh.


Changes
---

Addressed comments


Bugs: https://issues.apache.org/jira/browse/ATLAS-3368

https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-3368


Repository: atlas


Description
---

If an entity is created that has a definition with a reference to a
non built in type, it will trigger a legacy code path in EntityGraphMapper.
This path can take excessive amounts of time (>5s per commit), but operators
can be caught unaware as the log message for this is at debug
level and explain the portential issue.


Diffs (updated)
-

  docs/src/site/twiki/Configuration.twiki 6190a7c5f 
  intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 345b105ff 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
 c02f80905 


Diff: https://reviews.apache.org/r/71278/diff/2/

Changes: https://reviews.apache.org/r/71278/diff/1-2/


Testing
---

- reviewed patch logging to warn


Thanks,

Bolke de Bruin



Re: Review Request 71431: Add indexed order by (sort) to basic search

2019-09-14 Thread Bolke de Bruin

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71431/
---

(Updated Sept. 14, 2019, 12:45 p.m.)


Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
Madhan Neethiraj, and Nixon Rodrigues.


Changes
---

rebased against master


Bugs: https://issues.apache.org/jira/browse/ATLAS-3399

https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-3399


Repository: atlas


Description (updated)
---

Add indexed order by to basic search.


Diffs (updated)
-

  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
 f3722b827 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusIndexQuery.java
 7c258b7a3 
  graphdb/janus/src/main/java/org/janusgraph/diskstorage/solr/Solr6Index.java 
e457866af 
  intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 
aac6b5aa3 
  
repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java 
f7d8f08c7 
  webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java e6b338b84 
  webapp/src/test/java/org/apache/atlas/web/integration/BasicSearchIT.java 
06931b38c 
  webapp/src/test/resources/json/search-parameters/entity-filters.json 
f4b2efe63 


Diff: https://reviews.apache.org/r/71431/diff/2/

Changes: https://reviews.apache.org/r/71431/diff/1-2/


Testing
---

- production and tests added


Thanks,

Bolke de Bruin



Fwd: Outstanding review requests (patches!)

2019-09-12 Thread Bolke de Bruin
Ping!

I wonder what to do to grab some attention, because I see there is even
some duplication of effort going on now - ATLAS-3399 is partially
duplicated ATLAS-3378 which did receive reviews quickly? Do I need to add
reviewers? I do think the added functionality is worthwhile and will
benefit others too, but please let me know if those patches don’t make
sense.

Thanks for responding.

Bolke

On 4 September 2019 at 14:47:41, Bolke de Bruin (bdbr...@gmail.com) wrote:

Hello Team!

I have some outstanding review requests and I would appreciate feedback or
(duh) merging. Can someone take a look:

* Add order by (sort) to basic search : https://reviews.apache.org/r/71431/
* Use fulltext indices for dsl search :
https://reviews.apache.org/r/71344/ (Received
one “ship it!”, no further action)
* Expose total count of index searches : https://reviews.apache.org/r/71282/
* Set log level to warn for relationships without relationship definition :
https://reviews.apache.org/r/71278/. (This one is under review)

Thanks!
Bolke


[jira] [Commented] (ATLAS-3409) Remove the duplicate code in Solr6Index.

2019-09-12 Thread Bolke de Bruin (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928633#comment-16928633
 ] 

Bolke de Bruin commented on ATLAS-3409:
---

This is a duplicate of atlas-3397

> Remove the duplicate code in Solr6Index.
> 
>
> Key: ATLAS-3409
> URL: https://issues.apache.org/jira/browse/ATLAS-3409
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Sridhar
>Assignee: Sridhar
>Priority: Major
>
> Historically, we made a copy of SolrIndex from Janus graph and referred to it 
> as Solr6Index. Over time, SolrIndex in janus graph has mode enough progress 
> w.r.t Kerberos support etc. So, we should remove the duplicate code 
> Solr6Index.
> This ticket should be done after ATLAS-3378 is completed.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Commented] (ATLAS-3378) Update JanusGraph to 0.4.0

2019-09-10 Thread Bolke de Bruin (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926837#comment-16926837
 ] 

Bolke de Bruin commented on ATLAS-3378:
---

see also: ATLAS-3399

> Update JanusGraph to 0.4.0
> --
>
> Key: ATLAS-3378
> URL: https://issues.apache.org/jira/browse/ATLAS-3378
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0
>    Reporter: Bolke de Bruin
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 2.1.0
>
>
> This release supports hbase 2 out of the box



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


Re: Review Request 71278: Set log level to warn for relationships without relationship definition

2019-09-05 Thread Bolke de Bruin


> On sep 5, 2019, 8:15 p.m., Ashutosh Mestry wrote:
> > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
> > Line 907 (original), 907 (patched)
> > <https://reviews.apache.org/r/71278/diff/1/?file=2160649#file2160649line907>
> >
> > Are you OK with this warning being under debug log level?
> 
> Bolke de Bruin wrote:
> thanks for reviewing. I dont follow you comment? We noticed performance 
> (10s per entity for ingestion) issues but we had to turn on debugging to find 
> that we were triggering an unoptimized code path. Having this at warn would 
> have saved us a lot of time and maybe others. INFO might work to. happy to 
> change it to that if that is what you require.

arghhh got what you mean. will fix that


- Bolke


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71278/#review217597
-----------


On aug 21, 2019, 7:42 p.m., Bolke de Bruin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71278/
> ---
> 
> (Updated aug 21, 2019, 7:42 p.m.)
> 
> 
> Review request for atlas, Nixon Rodrigues and Saqeeb Shaikh.
> 
> 
> Bugs: https://issues.apache.org/jira/browse/ATLAS-3368
> 
> https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-3368
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> If an entity is created that has a definition with a reference to a
> non built in type, it will trigger a legacy code path in 
> EntityGraphMapper.
> This path can take excessive amounts of time (>5s per commit), but 
> operators
> can be caught unaware as the log message for this is at debug
> level and explain the portential issue.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
>  f9010fefb 
> 
> 
> Diff: https://reviews.apache.org/r/71278/diff/1/
> 
> 
> Testing
> ---
> 
> - reviewed patch logging to warn
> 
> 
> Thanks,
> 
> Bolke de Bruin
> 
>



Re: Review Request 71278: Set log level to warn for relationships without relationship definition

2019-09-05 Thread Bolke de Bruin


> On sep 5, 2019, 8:15 p.m., Ashutosh Mestry wrote:
> > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
> > Line 907 (original), 907 (patched)
> > <https://reviews.apache.org/r/71278/diff/1/?file=2160649#file2160649line907>
> >
> > Are you OK with this warning being under debug log level?

thanks for reviewing. I dont follow you comment? We noticed performance (10s 
per entity for ingestion) issues but we had to turn on debugging to find that 
we were triggering an unoptimized code path. Having this at warn would have 
saved us a lot of time and maybe others. INFO might work to. happy to change it 
to that if that is what you require.


- Bolke


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71278/#review217597
-------


On aug 21, 2019, 7:42 p.m., Bolke de Bruin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71278/
> ---
> 
> (Updated aug 21, 2019, 7:42 p.m.)
> 
> 
> Review request for atlas, Nixon Rodrigues and Saqeeb Shaikh.
> 
> 
> Bugs: https://issues.apache.org/jira/browse/ATLAS-3368
> 
> https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-3368
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> If an entity is created that has a definition with a reference to a
> non built in type, it will trigger a legacy code path in 
> EntityGraphMapper.
> This path can take excessive amounts of time (>5s per commit), but 
> operators
> can be caught unaware as the log message for this is at debug
> level and explain the portential issue.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
>  f9010fefb 
> 
> 
> Diff: https://reviews.apache.org/r/71278/diff/1/
> 
> 
> Testing
> ---
> 
> - reviewed patch logging to warn
> 
> 
> Thanks,
> 
> Bolke de Bruin
> 
>



Outstanding review requests

2019-09-04 Thread Bolke de Bruin
Hello Team!

I have some outstanding review requests and I would appreciate feedback or
(duh) merging. Can someone take a look:

* Add order by (sort) to basic search : https://reviews.apache.org/r/71431/
* Use fulltext indices for dsl search : https://reviews.apache.org/r/71344/
* Expose total count of index searches : https://reviews.apache.org/r/71282/
* Set log level to warn for relationships without relationship definition :
https://reviews.apache.org/r/71278/

Thanks!
Bolke


[jira] [Updated] (ATLAS-3399) Add order by (sort) to basic search

2019-09-04 Thread Bolke de Bruin (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin updated ATLAS-3399:
--
External issue URL: https://reviews.apache.org/r/71431/

> Add order by (sort) to basic search
> ---
>
> Key: ATLAS-3399
> URL: https://issues.apache.org/jira/browse/ATLAS-3399
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 2.0.0
>    Reporter: Bolke de Bruin
>    Assignee: Bolke de Bruin
>Priority: Major
> Fix For: trunk
>
> Attachments: 0001-ATLAS-3399-Add-order-by-to-basic-search.patch
>
>
> JanusGraph 0.4.0 has added support for indexed "Order By" using the search 
> backend. This would allow sorting on names, popularity scores etc en improve 
> the capabilities for data discovery



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


Review Request 71431: Add indexed order by (sort) to basic search

2019-09-04 Thread Bolke de Bruin

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71431/
---

Review request for atlas and Madhan Neethiraj.


Bugs: https://issues.apache.org/jira/browse/ATLAS-3399

https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-3399


Repository: atlas


Description
---

Add indexed order by to basic search. This upgrade janusgraph to 0.4.0


Diffs
-

  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
 f3722b827 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusIndexQuery.java
 7c258b7a3 
  graphdb/janus/src/main/java/org/janusgraph/diskstorage/solr/Solr6Index.java 
e457866af 
  intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 
aac6b5aa3 
  
repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java 
f7d8f08c7 
  webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java e6b338b84 
  webapp/src/test/java/org/apache/atlas/web/integration/BasicSearchIT.java 
06931b38c 
  webapp/src/test/resources/json/search-parameters/entity-filters.json 
f4b2efe63 


Diff: https://reviews.apache.org/r/71431/diff/1/


Testing
---

- production and tests added


Thanks,

Bolke de Bruin



[jira] [Assigned] (ATLAS-3399) Add order by (sort) to basic search

2019-09-04 Thread Bolke de Bruin (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin reassigned ATLAS-3399:
-

Assignee: Bolke de Bruin

> Add order by (sort) to basic search
> ---
>
> Key: ATLAS-3399
> URL: https://issues.apache.org/jira/browse/ATLAS-3399
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 2.0.0
>    Reporter: Bolke de Bruin
>    Assignee: Bolke de Bruin
>Priority: Major
> Fix For: trunk
>
>
> JanusGraph 0.4.0 has added support for indexed "Order By" using the search 
> backend. This would allow sorting on names, popularity scores etc en improve 
> the capabilities for data discovery



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Created] (ATLAS-3399) Add order by (sort) to basic search

2019-09-04 Thread Bolke de Bruin (Jira)
Bolke de Bruin created ATLAS-3399:
-

 Summary: Add order by (sort) to basic search
 Key: ATLAS-3399
 URL: https://issues.apache.org/jira/browse/ATLAS-3399
 Project: Atlas
  Issue Type: New Feature
  Components:  atlas-core
Affects Versions: 2.0.0
Reporter: Bolke de Bruin
 Fix For: trunk


JanusGraph 0.4.0 has added support for indexed "Order By" using the search 
backend. This would allow sorting on names, popularity scores etc en improve 
the capabilities for data discovery



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


  1   2   >