[jira] [Created] (ATLAS-3654) Support solr in standalone (http) mode
Damian Warszawski created ATLAS-3654: Summary: Support solr in standalone (http) mode Key: ATLAS-3654 URL: https://issues.apache.org/jira/browse/ATLAS-3654 Project: Atlas Issue Type: Improvement Components: atlas-core Affects Versions: 3.0.0 Reporter: Damian Warszawski *Problem description* Atlas does not support running Solr in standalone(http) mode. *Goals* It is especially useful for testing purposes to make setup as simple as possible without Zookeeper. It also enables full integration with JanusGraph as it support both mode of running Solr `cloud` and `http` [https://docs.janusgraph.org/index-backend/solr/]. Additional benefit is to decouple hbase and solr while running embedded mode so that solr can be run in embbeded mode with external hbase. *Proposed solution* * call solr V1 API while creating/updating request handlers in standalone solr * update atlas start script to enable standalone embedded solr -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3654) Support solr in standalone (http) mode
[ https://issues.apache.org/jira/browse/ATLAS-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Warszawski updated ATLAS-3654: - Attachment: ATLAS-3654.patch > Support solr in standalone (http) mode > -- > > Key: ATLAS-3654 > URL: https://issues.apache.org/jira/browse/ATLAS-3654 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Damian Warszawski >Priority: Minor > Attachments: ATLAS-3654.patch > > > *Problem description* > Atlas does not support running Solr in standalone(http) mode. > *Goals* > It is especially useful for testing purposes to make setup as simple as > possible without Zookeeper. It also enables full integration with JanusGraph > as it support both mode of running Solr `cloud` and `http` > [https://docs.janusgraph.org/index-backend/solr/]. Additional benefit is to > decouple hbase and solr while running embedded mode so that solr can be run > in embbeded mode with external hbase. > *Proposed solution* > * call solr V1 API while creating/updating request handlers in standalone > solr > * update atlas start script to enable standalone embedded solr > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3654) Support solr in standalone (http) mode
[ https://issues.apache.org/jira/browse/ATLAS-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Warszawski updated ATLAS-3654: - External issue URL: https://github.com/apache/atlas/pull/90 > Support solr in standalone (http) mode > -- > > Key: ATLAS-3654 > URL: https://issues.apache.org/jira/browse/ATLAS-3654 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Damian Warszawski >Priority: Minor > Attachments: ATLAS-3654.patch > > Time Spent: 10m > Remaining Estimate: 0h > > *Problem description* > Atlas does not support running Solr in standalone(http) mode. > *Goals* > It is especially useful for testing purposes to make setup as simple as > possible without Zookeeper. It also enables full integration with JanusGraph > as it support both mode of running Solr `cloud` and `http` > [https://docs.janusgraph.org/index-backend/solr/]. Additional benefit is to > decouple hbase and solr while running embedded mode so that solr can be run > in embbeded mode with external hbase. > *Proposed solution* > * call solr V1 API while creating/updating request handlers in standalone > solr > * update atlas start script to enable standalone embedded solr > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3654) Support solr in standalone (http) mode
[ https://issues.apache.org/jira/browse/ATLAS-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Warszawski updated ATLAS-3654: - Attachment: (was: ATLAS-3654.patch) > Support solr in standalone (http) mode > -- > > Key: ATLAS-3654 > URL: https://issues.apache.org/jira/browse/ATLAS-3654 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Damian Warszawski >Priority: Minor > Attachments: ATLAS-3654.patch > > > *Problem description* > Atlas does not support running Solr in standalone(http) mode. > *Goals* > It is especially useful for testing purposes to make setup as simple as > possible without Zookeeper. It also enables full integration with JanusGraph > as it support both mode of running Solr `cloud` and `http` > [https://docs.janusgraph.org/index-backend/solr/]. Additional benefit is to > decouple hbase and solr while running embedded mode so that solr can be run > in embbeded mode with external hbase. > *Proposed solution* > * call solr V1 API while creating/updating request handlers in standalone > solr > * update atlas start script to enable standalone embedded solr > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3654) Support solr in standalone (http) mode
[ https://issues.apache.org/jira/browse/ATLAS-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Warszawski updated ATLAS-3654: - Attachment: ATLAS-3654.patch > Support solr in standalone (http) mode > -- > > Key: ATLAS-3654 > URL: https://issues.apache.org/jira/browse/ATLAS-3654 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Damian Warszawski >Priority: Minor > Attachments: ATLAS-3654.patch > > > *Problem description* > Atlas does not support running Solr in standalone(http) mode. > *Goals* > It is especially useful for testing purposes to make setup as simple as > possible without Zookeeper. It also enables full integration with JanusGraph > as it support both mode of running Solr `cloud` and `http` > [https://docs.janusgraph.org/index-backend/solr/]. Additional benefit is to > decouple hbase and solr while running embedded mode so that solr can be run > in embbeded mode with external hbase. > *Proposed solution* > * call solr V1 API while creating/updating request handlers in standalone > solr > * update atlas start script to enable standalone embedded solr > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3760) Optimize FreeTextSearchProcessor to apply exclude deleted entity filter on solr side.
Damian Warszawski created ATLAS-3760: Summary: Optimize FreeTextSearchProcessor to apply exclude deleted entity filter on solr side. Key: ATLAS-3760 URL: https://issues.apache.org/jira/browse/ATLAS-3760 Project: Atlas Issue Type: Improvement Components: atlas-core Reporter: Damian Warszawski Fix For: 3.0.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3654) Support solr in standalone (http) mode
[ https://issues.apache.org/jira/browse/ATLAS-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Warszawski updated ATLAS-3654: - Attachment: ATLAS-3654.patch > Support solr in standalone (http) mode > -- > > Key: ATLAS-3654 > URL: https://issues.apache.org/jira/browse/ATLAS-3654 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Damian Warszawski >Priority: Minor > Attachments: ATLAS-3654.patch > > Time Spent: 10m > Remaining Estimate: 0h > > *Problem description* > Atlas does not support running Solr in standalone(http) mode. > *Goals* > It is especially useful for testing purposes to make setup as simple as > possible without Zookeeper. It also enables full integration with JanusGraph > as it support both mode of running Solr `cloud` and `http` > [https://docs.janusgraph.org/index-backend/solr/]. Additional benefit is to > decouple hbase and solr while running embedded mode so that solr can be run > in embbeded mode with external hbase. > *Proposed solution* > * call solr V1 API while creating/updating request handlers in standalone > solr > * update atlas start script to enable standalone embedded solr > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3758) Support sort params for FreeTextSearchProcessor
[ https://issues.apache.org/jira/browse/ATLAS-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Warszawski updated ATLAS-3758: - Description: *Problem description* No way to sort results by specified attribute while freetext search is enabled. *Goals* As a team we are working to use Atlas as a metadata storage for [https://github.com/lyft/amundsen]. It is required to sort results by any particular attribute e.g. custom attribute which represents popularity score to provide basic search relevancy for end users. *Proposed solution* * add required parameters in the indexed query if specified was: *Problem description* No way to sort results by specified attribute while freetext search is enabled. *Goals* As a team we are working to use Atlas as a metadata storage for [https://github.com/lyft/amundsen]. It is required to sort results by particular attribute e.g. popularityScore to provide basic search relevancy for end users. *Proposed solution* * add required parameters in the indexed query if specified > Support sort params for FreeTextSearchProcessor > --- > > Key: ATLAS-3758 > URL: https://issues.apache.org/jira/browse/ATLAS-3758 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Damian Warszawski >Priority: Minor > Attachments: ATLAS-3758.patch > > > *Problem description* > No way to sort results by specified attribute while freetext search is > enabled. > *Goals* > As a team we are working to use Atlas as a metadata storage for > [https://github.com/lyft/amundsen]. It is required to sort results by any > particular attribute e.g. custom attribute which represents popularity score > to provide basic search relevancy for end users. > *Proposed solution* > * add required parameters in the indexed query if specified -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3654) Support solr in standalone (http) mode
[ https://issues.apache.org/jira/browse/ATLAS-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Warszawski updated ATLAS-3654: - Attachment: (was: ATLAS-3654.patch) > Support solr in standalone (http) mode > -- > > Key: ATLAS-3654 > URL: https://issues.apache.org/jira/browse/ATLAS-3654 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Damian Warszawski >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > *Problem description* > Atlas does not support running Solr in standalone(http) mode. > *Goals* > It is especially useful for testing purposes to make setup as simple as > possible without Zookeeper. It also enables full integration with JanusGraph > as it support both mode of running Solr `cloud` and `http` > [https://docs.janusgraph.org/index-backend/solr/]. Additional benefit is to > decouple hbase and solr while running embedded mode so that solr can be run > in embbeded mode with external hbase. > *Proposed solution* > * call solr V1 API while creating/updating request handlers in standalone > solr > * update atlas start script to enable standalone embedded solr > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3758) Support sort params for FreeTextSearchProcessor
Damian Warszawski created ATLAS-3758: Summary: Support sort params for FreeTextSearchProcessor Key: ATLAS-3758 URL: https://issues.apache.org/jira/browse/ATLAS-3758 Project: Atlas Issue Type: Improvement Components: atlas-core Affects Versions: 3.0.0 Reporter: Damian Warszawski *Problem description* No way to sort results by specified attribute while freetext search is enabled. *Goals* As a team we are working to use Atlas as a metadata storage for [https://github.com/lyft/amundsen]. It is required to sort results by particular attribute e.g. popularityScore to provide basic search relevancy for end users. *Proposed solution* * add required parameters in the indexed query if specified -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3758) Support sort params for FreeTextSearchProcessor
[ https://issues.apache.org/jira/browse/ATLAS-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Warszawski updated ATLAS-3758: - Attachment: ATLAS-3758.patch > Support sort params for FreeTextSearchProcessor > --- > > Key: ATLAS-3758 > URL: https://issues.apache.org/jira/browse/ATLAS-3758 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Damian Warszawski >Priority: Minor > Attachments: ATLAS-3758.patch > > > *Problem description* > No way to sort results by specified attribute while freetext search is > enabled. > *Goals* > As a team we are working to use Atlas as a metadata storage for > [https://github.com/lyft/amundsen]. It is required to sort results by > particular attribute e.g. popularityScore to provide basic search relevancy > for end users. > *Proposed solution* > * add required parameters in the indexed query if specified -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3760) Optimize FreeTextSearchProcessor to apply exclude deleted entity filter on solr side.
[ https://issues.apache.org/jira/browse/ATLAS-3760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Warszawski updated ATLAS-3760: - Description: *Problem description* Current implementation of FreeTextSearchProcessor applies filtering in memory to exclude deleted entities. This introduces significant performance overhead by generating redundant calls to solr index. *Goals* Improve performance of FreeTextSearchProcessor by applying filter in solr query. *Proposed solution* * replace in-memory filtering with filter in solr query. > Optimize FreeTextSearchProcessor to apply exclude deleted entity filter on > solr side. > -- > > Key: ATLAS-3760 > URL: https://issues.apache.org/jira/browse/ATLAS-3760 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Reporter: Damian Warszawski >Priority: Minor > Fix For: 3.0.0 > > > *Problem description* > Current implementation of FreeTextSearchProcessor applies filtering in memory > to exclude deleted entities. > This introduces significant performance overhead by generating redundant > calls to solr index. > *Goals* > Improve performance of FreeTextSearchProcessor by applying filter in solr > query. > *Proposed solution* > * replace in-memory filtering with filter in solr query. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (ATLAS-3654) Support solr in standalone (http) mode
[ https://issues.apache.org/jira/browse/ATLAS-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17095892#comment-17095892 ] Damian Warszawski edited comment on ATLAS-3654 at 4/29/20, 9:09 PM: [~nixon], it is controlled with following application property `atlas.graph.index.search.solr.mode` which is also used by JanusGraph. Package is build with the profile `embedded-hbase-solr` as it used to be for `cloud` mode for compatibility reasons. Perhaps, it would useful to create another profile for `embedded-solr` only. was (Author: dwarszawski): it is controlled with following application property `atlas.graph.index.search.solr.mode` which is also used by JanusGraph. Package is build with the profile `embedded-hbase-solr` as it used to be for `cloud` mode for compatibility reasons. Perhaps, it would useful to create another profile for `embedded-solr` only. > Support solr in standalone (http) mode > -- > > Key: ATLAS-3654 > URL: https://issues.apache.org/jira/browse/ATLAS-3654 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Damian Warszawski >Priority: Minor > Attachments: ATLAS-3654.patch > > Time Spent: 10m > Remaining Estimate: 0h > > *Problem description* > Atlas does not support running Solr in standalone(http) mode. > *Goals* > It is especially useful for testing purposes to make setup as simple as > possible without Zookeeper. It also enables full integration with JanusGraph > as it support both mode of running Solr `cloud` and `http` > [https://docs.janusgraph.org/index-backend/solr/]. Additional benefit is to > decouple hbase and solr while running embedded mode so that solr can be run > in embbeded mode with external hbase. > *Proposed solution* > * call solr V1 API while creating/updating request handlers in standalone > solr > * update atlas start script to enable standalone embedded solr > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3654) Support solr in standalone (http) mode
[ https://issues.apache.org/jira/browse/ATLAS-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17095892#comment-17095892 ] Damian Warszawski commented on ATLAS-3654: -- it is controlled with following application property `atlas.graph.index.search.solr.mode` which is also used by JanusGraph. Package is build with the profile `embedded-hbase-solr` as it used to be for `cloud` mode for compatibility reasons. Perhaps, it would useful to create another profile for `embedded-solr` only. > Support solr in standalone (http) mode > -- > > Key: ATLAS-3654 > URL: https://issues.apache.org/jira/browse/ATLAS-3654 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Damian Warszawski >Priority: Minor > Attachments: ATLAS-3654.patch > > Time Spent: 10m > Remaining Estimate: 0h > > *Problem description* > Atlas does not support running Solr in standalone(http) mode. > *Goals* > It is especially useful for testing purposes to make setup as simple as > possible without Zookeeper. It also enables full integration with JanusGraph > as it support both mode of running Solr `cloud` and `http` > [https://docs.janusgraph.org/index-backend/solr/]. Additional benefit is to > decouple hbase and solr while running embedded mode so that solr can be run > in embbeded mode with external hbase. > *Proposed solution* > * call solr V1 API while creating/updating request handlers in standalone > solr > * update atlas start script to enable standalone embedded solr > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (ATLAS-3654) Support solr in standalone (http) mode
[ https://issues.apache.org/jira/browse/ATLAS-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17095892#comment-17095892 ] Damian Warszawski edited comment on ATLAS-3654 at 4/29/20, 9:10 PM: [~nixon], it is controlled with following application property `_atlas.graph.index.search.solr.mode_` which is also used by JanusGraph. Package is build with the profile `_embedded-hbase-solr_` as it used to be for `cloud` mode for compatibility reasons. Perhaps, it would useful to create another profile for `_embedded-solr_` only. was (Author: dwarszawski): [~nixon], it is controlled with following application property `atlas.graph.index.search.solr.mode` which is also used by JanusGraph. Package is build with the profile `embedded-hbase-solr` as it used to be for `cloud` mode for compatibility reasons. Perhaps, it would useful to create another profile for `embedded-solr` only. > Support solr in standalone (http) mode > -- > > Key: ATLAS-3654 > URL: https://issues.apache.org/jira/browse/ATLAS-3654 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Damian Warszawski >Priority: Minor > Attachments: ATLAS-3654.patch > > Time Spent: 10m > Remaining Estimate: 0h > > *Problem description* > Atlas does not support running Solr in standalone(http) mode. > *Goals* > It is especially useful for testing purposes to make setup as simple as > possible without Zookeeper. It also enables full integration with JanusGraph > as it support both mode of running Solr `cloud` and `http` > [https://docs.janusgraph.org/index-backend/solr/]. Additional benefit is to > decouple hbase and solr while running embedded mode so that solr can be run > in embbeded mode with external hbase. > *Proposed solution* > * call solr V1 API while creating/updating request handlers in standalone > solr > * update atlas start script to enable standalone embedded solr > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3760) Optimize FreeTextSearchProcessor to apply exclude deleted entity filter on solr side.
[ https://issues.apache.org/jira/browse/ATLAS-3760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17095884#comment-17095884 ] Damian Warszawski commented on ATLAS-3760: -- [~madhan] thanks for getting this done so quickly. > Optimize FreeTextSearchProcessor to apply exclude deleted entity filter on > solr side. > -- > > Key: ATLAS-3760 > URL: https://issues.apache.org/jira/browse/ATLAS-3760 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Reporter: Damian Warszawski >Priority: Minor > Fix For: 2.1.0, 3.0.0 > > > *Problem description* > Current implementation of FreeTextSearchProcessor applies filtering in memory > to exclude deleted entities. > This introduces significant performance overhead by generating redundant > calls to solr index. > *Goals* > Improve performance of FreeTextSearchProcessor by applying filter in solr > query. > *Proposed solution* > * replace in-memory filtering with filter in solr query. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3776) graph query fails when orderBy attribute is specified
Damian Warszawski created ATLAS-3776:
Summary: graph query fails when orderBy attribute is specified
Key: ATLAS-3776
URL: https://issues.apache.org/jira/browse/ATLAS-3776
Project: Atlas
Issue Type: Bug
Components: atlas-core
Affects Versions: 3.0.0
Reporter: Damian Warszawski
EntitySearchProcessor fails when doing search by classification and specify
orderBy attribute. The issue is that for graph query you cannot refer to
attribute by name but need to provide absolute path to entity attribute e.g.
```
{
"attributes": [
"description",
"comment",
"popularityScore"
],
"classification": "customer_NON_PII",
"excludeDeletedEntities": "False",
"limit": "",
"offset": 100,
"sortBy": "Table.popularityScore",
"sortOrder": "DESCENDING",
"typeName": "hive_table"
}
```
this query fails with following exception:
```
{"exception":{"message":"Provided key does not exist:
hive_table.popularityScore","class":"java.lang.IllegalArgumentException","stacktrace":"java.lang.IllegalArgumentException:
Provided key does not exist: hive_table.popularityScore\n\tat
com.google.common.base.Preconditions.checkArgument(Preconditions.java:163)\n\tat
org.janusgraph.graphdb.query.graph.GraphCentricQueryBuilder.
orderBy(GraphCentricQueryBuilder.java:160)
```
When specify full reference to attribute e.g.
```
{
"attributes": [
"description",
"comment",
"popularityScore"
],
"classification": "customer_NON_PII",
"excludeDeletedEntities": "False",
"limit": "",
"offset": 100,
"sortBy": "Table.popularityScore",
"sortOrder": "DESCENDING",
"typeName": "hive_table"
}
```
it fails on validation stage
```
{"exception":{"message":"Attribute Table.popularityScore not found for type
Table","class":"org.apache.atlas.exception.AtlasBaseException","stacktrace":"org.apache.atlas.exception.AtlasBaseException:
Attribute Table.popularityScore not found for type Table\n\tat
org.apache.atlas.discovery.SearchContext.validateAttributes(SearchContext.java:288)
```
workaround is provided as a patch.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-3776) graph query fails when orderBy attribute is specified
[
https://issues.apache.org/jira/browse/ATLAS-3776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Damian Warszawski updated ATLAS-3776:
-
Description:
EntitySearchProcessor fails when doing search by classification and specify
orderBy attribute. The issue is that for graph query you cannot refer to
attribute by name but need to provide absolute path to entity attribute e.g.
```
{ "attributes": [ "description", "comment", "popularityScore" ],
"classification": "customer_NON_PII", "excludeDeletedEntities": "False",
"limit": "", "offset": 100, "sortBy": "Table.popularityScore", "sortOrder":
"DESCENDING", "typeName": "hive_table" }
```
this query fails with following exception:
```
{"exception":{"message":"Provided key does not exist:
Table.popularityScore","class":"java.lang.IllegalArgumentException","stacktrace":"java.lang.IllegalArgumentException:
Provided key does not exist: hive_table.popularityScore\n\tat
com.google.common.base.Preconditions.checkArgument(Preconditions.java:163)\n\tat
org.janusgraph.graphdb.query.graph.GraphCentricQueryBuilder.
orderBy(GraphCentricQueryBuilder.java:160)
```
When specify full reference to attribute e.g.
```
{ "attributes": [ "description", "comment", "popularityScore" ],
"classification": "customer_NON_PII", "excludeDeletedEntities": "False",
"limit": "", "offset": 100, "sortBy": "Table.popularityScore", "sortOrder":
"DESCENDING", "typeName": "hive_table" }
```
it fails on validation stage
```
{"exception":{"message":"Attribute Table.popularityScore not found for type
Table","class":"org.apache.atlas.exception.AtlasBaseException","stacktrace":"org.apache.atlas.exception.AtlasBaseException:
Attribute Table.popularityScore not found for type Table\n\tat
org.apache.atlas.discovery.SearchContext.validateAttributes(SearchContext.java:288)
```
workaround is provided as a patch.
was:
EntitySearchProcessor fails when doing search by classification and specify
orderBy attribute. The issue is that for graph query you cannot refer to
attribute by name but need to provide absolute path to entity attribute e.g.
```
{
"attributes": [
"description",
"comment",
"popularityScore"
],
"classification": "customer_NON_PII",
"excludeDeletedEntities": "False",
"limit": "",
"offset": 100,
"sortBy": "Table.popularityScore",
"sortOrder": "DESCENDING",
"typeName": "hive_table"
}
```
this query fails with following exception:
```
{"exception":{"message":"Provided key does not exist:
hive_table.popularityScore","class":"java.lang.IllegalArgumentException","stacktrace":"java.lang.IllegalArgumentException:
Provided key does not exist: hive_table.popularityScore\n\tat
com.google.common.base.Preconditions.checkArgument(Preconditions.java:163)\n\tat
org.janusgraph.graphdb.query.graph.GraphCentricQueryBuilder.
orderBy(GraphCentricQueryBuilder.java:160)
```
When specify full reference to attribute e.g.
```
{
"attributes": [
"description",
"comment",
"popularityScore"
],
"classification": "customer_NON_PII",
"excludeDeletedEntities": "False",
"limit": "",
"offset": 100,
"sortBy": "Table.popularityScore",
"sortOrder": "DESCENDING",
"typeName": "hive_table"
}
```
it fails on validation stage
```
{"exception":{"message":"Attribute Table.popularityScore not found for type
Table","class":"org.apache.atlas.exception.AtlasBaseException","stacktrace":"org.apache.atlas.exception.AtlasBaseException:
Attribute Table.popularityScore not found for type Table\n\tat
org.apache.atlas.discovery.SearchContext.validateAttributes(SearchContext.java:288)
```
workaround is provided as a patch.
> graph query fails when orderBy attribute is specified
> -
>
> Key: ATLAS-3776
> URL: https://issues.apache.org/jira/browse/ATLAS-3776
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Affects Versions: 3.0.0
>Reporter: Damian Warszawski
>Priority: Minor
>
> EntitySearchProcessor fails when doing search by classification and specify
> orderBy attribute. The issue is that for graph query you cannot refer to
> attribute by name but need to provide absolute path to entity attribute e.g.
>
> ```
> { "attributes": [ "description", "comment", "popularityScore" ],
> "classification": "customer_NON_PII", "excludeDeletedEntities": "False",
> "limit": "", "offset": 100, "sortBy": "Table.popularityScore", "sortOrder":
> "DESCENDING", "typeName": "hive_table" }
> ```
> this query fails with following exception:
>
> ```
> {"exception":{"message":"Provided key does not exist:
> Table.popularityScore","class":"java.lang.IllegalArgumentException","stacktrace":"java.lang.IllegalArgumentException:
> Provided key does not exist: hive_table.popularityScore\n\tat
>
[jira] [Closed] (ATLAS-3758) Support sort params for FreeTextSearchProcessor
[ https://issues.apache.org/jira/browse/ATLAS-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Warszawski closed ATLAS-3758. Fixed and merged. > Support sort params for FreeTextSearchProcessor > --- > > Key: ATLAS-3758 > URL: https://issues.apache.org/jira/browse/ATLAS-3758 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Damian Warszawski >Priority: Minor > Fix For: 2.1.0, 3.0.0 > > Attachments: ATLAS-3758.patch > > > *Problem description* > No way to sort results by specified attribute while freetext search is > enabled. > *Goals* > As a team we are working to use Atlas as a metadata storage for > [https://github.com/lyft/amundsen]. It is required to sort results by any > particular attribute e.g. custom attribute which represents popularity score > to provide basic search relevancy for end users. > *Proposed solution* > * add required parameters in the indexed query if specified -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (ATLAS-3760) Optimize FreeTextSearchProcessor to apply exclude deleted entity filter on solr side.
[ https://issues.apache.org/jira/browse/ATLAS-3760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Warszawski closed ATLAS-3760. Fixed and merged > Optimize FreeTextSearchProcessor to apply exclude deleted entity filter on > solr side. > -- > > Key: ATLAS-3760 > URL: https://issues.apache.org/jira/browse/ATLAS-3760 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Reporter: Damian Warszawski >Priority: Minor > Fix For: 2.1.0, 3.0.0 > > > *Problem description* > Current implementation of FreeTextSearchProcessor applies filtering in memory > to exclude deleted entities. > This introduces significant performance overhead by generating redundant > calls to solr index. > *Goals* > Improve performance of FreeTextSearchProcessor by applying filter in solr > query. > *Proposed solution* > * replace in-memory filtering with filter in solr query. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (ATLAS-3776) graph query fails when orderBy attribute is specified
[
https://issues.apache.org/jira/browse/ATLAS-3776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Damian Warszawski closed ATLAS-3776.
fixed and merged.
> graph query fails when orderBy attribute is specified
> -
>
> Key: ATLAS-3776
> URL: https://issues.apache.org/jira/browse/ATLAS-3776
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Affects Versions: 3.0.0
>Reporter: Damian Warszawski
>Priority: Minor
> Fix For: 2.1.0, 3.0.0
>
>
> EntitySearchProcessor fails when doing search by classification and specify
> orderBy attribute. The issue is that for graph query you cannot refer to
> attribute by name but need to provide absolute path to entity attribute e.g.
>
> ```
> { "attributes": [ "description", "comment", "popularityScore" ],
> "classification": "customer_NON_PII", "excludeDeletedEntities": "False",
> "limit": "", "offset": 100, "sortBy": "Table.popularityScore", "sortOrder":
> "DESCENDING", "typeName": "hive_table" }
> ```
> this query fails with following exception:
>
> ```
> {"exception":{"message":"Provided key does not exist:
> Table.popularityScore","class":"java.lang.IllegalArgumentException","stacktrace":"java.lang.IllegalArgumentException:
> Provided key does not exist: hive_table.popularityScore\n\tat
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:163)\n\tat
> org.janusgraph.graphdb.query.graph.GraphCentricQueryBuilder.
> orderBy(GraphCentricQueryBuilder.java:160)
> ```
>
> When specify full reference to attribute e.g.
>
> ```
> { "attributes": [ "description", "comment", "popularityScore" ],
> "classification": "customer_NON_PII", "excludeDeletedEntities": "False",
> "limit": "", "offset": 100, "sortBy": "Table.popularityScore", "sortOrder":
> "DESCENDING", "typeName": "hive_table" }
> ```
> it fails on validation stage
>
> ```
> {"exception":{"message":"Attribute Table.popularityScore not found for type
> Table","class":"org.apache.atlas.exception.AtlasBaseException","stacktrace":"org.apache.atlas.exception.AtlasBaseException:
> Attribute Table.popularityScore not found for type Table\n\tat
> org.apache.atlas.discovery.SearchContext.validateAttributes(SearchContext.java:288)
> ```
> workaround is provided as a patch.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (ATLAS-3654) Support solr in standalone (http) mode
[ https://issues.apache.org/jira/browse/ATLAS-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Warszawski closed ATLAS-3654. fixed and merged > Support solr in standalone (http) mode > -- > > Key: ATLAS-3654 > URL: https://issues.apache.org/jira/browse/ATLAS-3654 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Damian Warszawski >Priority: Minor > Fix For: 2.1.0, 3.0.0 > > Attachments: ATLAS-3654.patch > > Time Spent: 10m > Remaining Estimate: 0h > > *Problem description* > Atlas does not support running Solr in standalone(http) mode. > *Goals* > It is especially useful for testing purposes to make setup as simple as > possible without Zookeeper. It also enables full integration with JanusGraph > as it support both mode of running Solr `cloud` and `http` > [https://docs.janusgraph.org/index-backend/solr/]. Additional benefit is to > decouple hbase and solr while running embedded mode so that solr can be run > in embbeded mode with external hbase. > *Proposed solution* > * call solr V1 API while creating/updating request handlers in standalone > solr > * update atlas start script to enable standalone embedded solr > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3758) Support sort params for FreeTextSearchProcessor
[ https://issues.apache.org/jira/browse/ATLAS-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17121023#comment-17121023 ] Damian Warszawski commented on ATLAS-3758: -- thx for update > Support sort params for FreeTextSearchProcessor > --- > > Key: ATLAS-3758 > URL: https://issues.apache.org/jira/browse/ATLAS-3758 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Damian Warszawski >Priority: Minor > Fix For: 2.1.0, 3.0.0 > > Attachments: ATLAS-3758-2-branch-2.0.patch, ATLAS-3758.patch > > > *Problem description* > No way to sort results by specified attribute while freetext search is > enabled. > *Goals* > As a team we are working to use Atlas as a metadata storage for > [https://github.com/lyft/amundsen]. It is required to sort results by any > particular attribute e.g. custom attribute which represents popularity score > to provide basic search relevancy for end users. > *Proposed solution* > * add required parameters in the indexed query if specified -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3398) Duplicates for unique attributes
[ https://issues.apache.org/jira/browse/ATLAS-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17169160#comment-17169160 ] Damian Warszawski commented on ATLAS-3398: -- Thanks [~amestry]. Of course, I don't mind. > Duplicates for unique attributes > - > > Key: ATLAS-3398 > URL: https://issues.apache.org/jira/browse/ATLAS-3398 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.0.0, trunk >Reporter: Bolke de Bruin >Assignee: Ashutosh Mestry >Priority: Blocker > Labels: integrity > Attachments: zrzut_ekranu_2019-09-03_o_10.28.50.png > > > We are seeing issues with entities being added to Atlas with duplicate > "qualifiedName". The guids differ and other attributes do also differ. Below > a graph that shows the distribution over time for duplicates. We have > difficulty determining which one is the right one (as they are different) in > order to clean them up. > We are also not the only ones encountering this as you can in the linked > issue. > We have noticed that Atlas does not use the > [locking|https://docs.janusgraph.org/master/advanced-topics/eventual-consistency/] > mechanism of Janus to prevent this: > > !zrzut_ekranu_2019-09-03_o_10.28.50.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3398) Duplicates for unique attributes
[ https://issues.apache.org/jira/browse/ATLAS-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17161569#comment-17161569 ] Damian Warszawski commented on ATLAS-3398: -- Optional configuration to support locks on JanusGraph to ensure data consitency -> [https://reviews.apache.org/r/72695/] > Duplicates for unique attributes > - > > Key: ATLAS-3398 > URL: https://issues.apache.org/jira/browse/ATLAS-3398 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.0.0, trunk >Reporter: Bolke de Bruin >Priority: Blocker > Labels: integrity > Attachments: zrzut_ekranu_2019-09-03_o_10.28.50.png > > > We are seeing issues with entities being added to Atlas with duplicate > "qualifiedName". The guids differ and other attributes do also differ. Below > a graph that shows the distribution over time for duplicates. We have > difficulty determining which one is the right one (as they are different) in > order to clean them up. > We are also not the only ones encountering this as you can in the linked > issue. > We have noticed that Atlas does not use the > [locking|https://docs.janusgraph.org/master/advanced-topics/eventual-consistency/] > mechanism of Janus to prevent this: > > !zrzut_ekranu_2019-09-03_o_10.28.50.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3398) Duplicates for unique attributes
[ https://issues.apache.org/jira/browse/ATLAS-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17161884#comment-17161884 ] Damian Warszawski commented on ATLAS-3398: -- According to our observations that can be a race condition between hive-hook (kafka event) and profiler (org.apache.atlas:atlas-client-v2:2.0.0). It can be also the case with atlas-client which make implicit retries while calling Atlas API. > Duplicates for unique attributes > - > > Key: ATLAS-3398 > URL: https://issues.apache.org/jira/browse/ATLAS-3398 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.0.0, trunk >Reporter: Bolke de Bruin >Priority: Blocker > Labels: integrity > Attachments: zrzut_ekranu_2019-09-03_o_10.28.50.png > > > We are seeing issues with entities being added to Atlas with duplicate > "qualifiedName". The guids differ and other attributes do also differ. Below > a graph that shows the distribution over time for duplicates. We have > difficulty determining which one is the right one (as they are different) in > order to clean them up. > We are also not the only ones encountering this as you can in the linked > issue. > We have noticed that Atlas does not use the > [locking|https://docs.janusgraph.org/master/advanced-topics/eventual-consistency/] > mechanism of Janus to prevent this: > > !zrzut_ekranu_2019-09-03_o_10.28.50.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3398) Duplicates for unique attributes
[ https://issues.apache.org/jira/browse/ATLAS-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17168268#comment-17168268 ] Damian Warszawski commented on ATLAS-3398: -- [~mad...@apache.org], [~amestry] uploaded another patch to fix the unit tests. Please have a look on it. Thanks. > Duplicates for unique attributes > - > > Key: ATLAS-3398 > URL: https://issues.apache.org/jira/browse/ATLAS-3398 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.0.0, trunk >Reporter: Bolke de Bruin >Priority: Blocker > Labels: integrity > Attachments: zrzut_ekranu_2019-09-03_o_10.28.50.png > > > We are seeing issues with entities being added to Atlas with duplicate > "qualifiedName". The guids differ and other attributes do also differ. Below > a graph that shows the distribution over time for duplicates. We have > difficulty determining which one is the right one (as they are different) in > order to clean them up. > We are also not the only ones encountering this as you can in the linked > issue. > We have noticed that Atlas does not use the > [locking|https://docs.janusgraph.org/master/advanced-topics/eventual-consistency/] > mechanism of Janus to prevent this: > > !zrzut_ekranu_2019-09-03_o_10.28.50.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
