Re: Review Request 61274: ATLAS-1944 - Fix for ConcurrentModificationException Exception in HookConsumer (KafkaConsumer) while stopping Atlas server
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61274/#review182027 --- In _commit_ needs: _(offset + 1)_ See this: http://kafka.apache.org/0110/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html - Ashutosh Mestry On Aug. 2, 2017, 8:48 a.m., Nixon Rodrigues wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/61274/ > --- > > (Updated Aug. 2, 2017, 8:48 a.m.) > > > Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and > Sarath Subramanian. > > > Bugs: ATLAS-1944 > https://issues.apache.org/jira/browse/ATLAS-1944 > > > Repository: atlas > > > Description > --- > > Background:- NotificationHookConsumer was throwing > ConcurrentModificationException exception while shutting down the consumer > thread. The stop method of thread was calling the kafkacosumer.stop method > while in run method of HookConsumer which is busy in cosuming the kafka > message by polling kafka server. This simaltanous action of stoping and > polling causes ConcurrentModificationException. > > Fix:- > > * Added try finally block in run method which wraps the while loop and when > loop ends the finally blocks calls the consumer close method is called which > ensures that close is called after consuming is ended. > > * Replaced the ShutdownThread interface which was delaying the shutdown > process of consumer with Runnable interface used earlier. > > > Reference : - > https://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html > > > Diffs > - > > notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java > d431176 > > notification/src/main/java/org/apache/atlas/notification/NotificationConsumer.java > 0bd75e1 > > notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java > bcee00c > > webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java > 51276d3 > > > Diff: https://reviews.apache.org/r/61274/diff/2/ > > > Testing > --- > > Tested the Atlas kafkaconsumer from hive hook and shutdown of atlas server to > check if there is issue while shutdown process. > Unit test and integration test are passing. > > > Thanks, > > Nixon Rodrigues > >
Re: Review Request 61263: ATLAS-2010: Hook consumer should start consumption after type registry init is complete
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61263/#review182038 --- Ship it! Ship It! - Ashutosh Mestry On July 31, 2017, 7:07 p.m., Apoorv Naik wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/61263/ > --- > > (Updated July 31, 2017, 7:07 p.m.) > > > Review request for atlas. > > > Bugs: ATLAS-2010 > https://issues.apache.org/jira/browse/ATLAS-2010 > > > Repository: atlas > > > Description > --- > > Sometimes the Hook consumer bean gets created before the type registry has > been fully initialized and when it's trying to create the entity, the type > name is missing from the registry. > Add depends on the HookConsumer class to make sure that the creation only > happens after type registry is completely bootstrapped. > > > Diffs > - > > intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java e35fb33f > > webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java > b8255b36 > > > Diff: https://reviews.apache.org/r/61263/diff/1/ > > > Testing > --- > > mvn clean package executes successfully > > Atlas server startup was successful and basic operations were good, no > anomaly. > > > Thanks, > > Apoorv Naik > >
Review Request 61665: ATLAS-2047: Exception Thrown by Kafka Consumer Ends up Filling Logs Due to Incorrect Handling
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61665/
---
Review request for atlas, Madhan Neethiraj and Nixon Rodrigues.
Bugs: ATLAS-2047
https://issues.apache.org/jira/browse/ATLAS-2047
Repository: atlas
Description
---
Please refer to [ATLAS-2047](https://issues.apache.org/jira/browse/ATLAS-2047)
for background and analysis.
**Implementation**
Please take a look at this scala code. This is _ShutdownableThread_. The thread
does the job of handling all exceptions. Upon exception, it manages the
_shutdownLatch_ (from yesterday’s bug fix) and gets out of the _isRunning_ loop.
```scala
override def run(): Unit = {
info("Starting ")
try{
while(isRunning.get()){
doWork()
}
} catch{
case e: Throwable =>
if(isRunning.get())
error("Error due to ", e)
}
shutdownLatch.countDown()
info("Stopped ")
}
```
Moved _try...catch_ block to leave exception handling to _ShutdownableThread_.
Diffs
-
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
ef64c3b
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java
a6f58e8
Diff: https://reviews.apache.org/r/61665/diff/1/
Testing
---
**Unit tests**
Updated unit tests to reproduce the scenarios and verify the fix.
**Functional tests**
Verified regular notification scenarios.
Thanks,
Ashutosh Mestry
Re: Review Request 61518: ATLAS-2033: Changed Pretty JSON Generation to Ugly for v1 Entity Serializers
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61518/ --- (Updated Aug. 9, 2017, 6:25 p.m.) Review request for atlas and Nixon Rodrigues. Changes --- Updates include: - _Ugly_ & _Pretty_ files. Bugs: ATLAS-2033 https://issues.apache.org/jira/browse/ATLAS-2033 Repository: atlas Description --- **Implementation** Changed _writePretty_ to _write_ for GSON serializer for v1 entities. Diffs - typesystem/src/main/scala/org/apache/atlas/typesystem/json/InstanceSerialization.scala 3354d7c Diff: https://reviews.apache.org/r/61518/diff/1/ Testing --- **Unit tests** Existing suite. File Attachments (updated) Ugly https://reviews.apache.org/media/uploaded/files/2017/08/09/440a989c-81ea-4653-b896-9d0d51465de4__ugly.json Pretty https://reviews.apache.org/media/uploaded/files/2017/08/09/1cf92e66-2dc6-4ded-a546-f1eaa7dcba36__pretty.json Thanks, Ashutosh Mestry
Re: Review Request 61541: ATLAS-2037: Unit Test Failure Resolution: testConsumersAreStoppedWhenInstanceBecomesPassive
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61541/ --- (Updated Aug. 10, 2017, 4:18 p.m.) Review request for atlas, Apoorv Naik and Nixon Rodrigues. Changes --- Updates include: - Addressed review comment. - Additional unit test to handle case where consumer is stopped before starting. Bugs: ATLAS-2037 https://issues.apache.org/jira/browse/ATLAS-2037 Repository: atlas Description --- **Implementation** Used the _doAnswer_ to be as close to regular flow as possible. This mocks submit method and starts the _HookConsumer_ thread that was added in the previous step. Verified _wakeup_ was invoked. Diffs (updated) - webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 5a982bb webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java f4ec56a Diff: https://reviews.apache.org/r/61541/diff/2/ Changes: https://reviews.apache.org/r/61541/diff/1-2/ Testing --- **Unit tests** Executed all unit tests for _NotificationHookConsumerTest_. Thanks, Ashutosh Mestry
Review Request 60722: Fix for: Export of hive_table with fetchType "connected" fails
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/60722/ --- Review request for atlas and Madhan Neethiraj. Bugs: ATLAS-1919 https://issues.apache.org/jira/browse/ATLAS-1919 Repository: atlas Description --- **Analysis** The connected _fetchType_ in Export API had a potential for attempting to save same entity twice. **Fix** The entities saved to the _ZipSink_ are now maintained in a separate set. Before attempting to save to _ZipSink_ existence check is performed. If passed, save is prevented, else entity is saved to _ZipSink_. Diffs - repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 3538cfd7 Diff: https://reviews.apache.org/r/60722/diff/1/ Testing --- **Unit tests** - Executed existing unit tests. **Functional tests** - Executed standard set of tests. Thanks, Ashutosh Mestry
Re: Review Request 60722: Fix for: Export of hive_table with fetchType "connected" fails
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/60722/ --- (Updated July 10, 2017, 3:59 p.m.) Review request for atlas and Madhan Neethiraj. Changes --- Updates include: - Refactoring based on review comments. - Updated Unit tests. Bugs: ATLAS-1919 https://issues.apache.org/jira/browse/ATLAS-1919 Repository: atlas Description --- **Analysis** The connected _fetchType_ in Export API had a potential for attempting to save same entity twice. **Fix** The entities saved to the _ZipSink_ are now maintained in a separate set. Before attempting to save to _ZipSink_ existence check is performed. If passed, save is prevented, else entity is saved to _ZipSink_. Diffs (updated) - repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 3538cfd7 repository/src/main/java/org/apache/atlas/repository/impexp/ZipSink.java 4bb04da9 repository/src/test/java/org/apache/atlas/repository/impexp/ZipSinkTest.java 635caf7f Diff: https://reviews.apache.org/r/60722/diff/2/ Changes: https://reviews.apache.org/r/60722/diff/1-2/ Testing --- **Unit tests** - Executed existing unit tests. **Functional tests** - Executed standard set of tests. Thanks, Ashutosh Mestry
Re: Review Request 60597: ATLAS-1908 : - Updating old Kafka consumer api properties to reflect change in new KafkaConsumer configs for Atlas
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/60597/#review180395 --- distro/src/conf/atlas-application.properties Lines 82 (patched) <https://reviews.apache.org/r/60597/#comment255549> Should this be 1000 ms? My thinking is 100 ms will be too short. - Ashutosh Mestry On July 3, 2017, 1:17 p.m., Nixon Rodrigues wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/60597/ > --- > > (Updated July 3, 2017, 1:17 p.m.) > > > Review request for atlas, Apoorv Naik, Madhan Neethiraj, and Sarath > Subramanian. > > > Bugs: ATLAS-1908 > https://issues.apache.org/jira/browse/ATLAS-1908 > > > Repository: atlas > > > Description > --- > > This patch handles old kafka consumer configs values to be added as new > configs if new configs are not present , so that Kafka consumer starts > gracefully even if new configs are not added explicitly. > > eg. atlas.kafka.auto.commit.enable=false to > atlas.kafka.enable.auto.commit=false > atlas.kafka.auto.offset.reset=smallest to > atlas.kafka.auto.offset.reset=earliest > atlas.kafka.session.timeout.ms=3 > > Also added poll.timeout.ms in configs so kafka consumer can be > configurable.This involve change in AtlasKafkaConsumer receive interface. > > > Diffs > - > > distro/src/conf/atlas-application.properties 474f253 > notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java > 9c15243 > notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java > 366c8a7 > > notification/src/main/java/org/apache/atlas/notification/NotificationConsumer.java > 22e40f9 > notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java > 70059cb > > notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java > c791d43 > > notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java > 8324b57 > > webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java > 9e5b864 > > webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java > 650ca0a > webapp/src/test/java/org/apache/atlas/web/integration/BaseResourceIT.java > c036cfa > > > Diff: https://reviews.apache.org/r/60597/diff/1/ > > > Testing > --- > > Executed UT/IT's using mvn clean install. > Tested KafkaConsumer with Hive Hook by creating table in hive. > Atlas starts gracefully and Kafka consumer is up without error when new > properties are not added. > > > Thanks, > > Nixon Rodrigues > >
Re: Review Request 60857: ATLAS-1950: Import API: Improvement: Specify Supertypes in Import Transforms
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60857/
---
(Updated July 14, 2017, 6 p.m.)
Review request for atlas, Ayub Pathan and Madhan Neethiraj.
Changes
---
Updates include:
- Updated request with issue id.
Summary (updated)
-
ATLAS-1950: Import API: Improvement: Specify Supertypes in Import Transforms
Bugs: ATLAS-1950
https://issues.apache.org/jira/browse/ATLAS-1950
Repository: atlas
Description
---
**Background**
Existing import transforms expects that the user specify each type and
attribute name that needs to be transformed. This may cause large number of
transform enumerating individual types to be specified. Specifying individual
transforms may be tedious and perhaps error prone.
**Solution**
This feature improves on that case by allowing user to specify super types.
That way the sub types of that type will automatically get the transform(s)
applied.
**Implementation**
User can now specify transforms like:
```javascript
{
"options": {
"transforms": "{ \"Asset\": { \"qualifiedName\": [
\"replace:@cl1:@cl2\" ] } }"
}
}
```
At runtime, the subtypes of this type are enumerated. The attribute and
tranform map of the super type are now associated with all the sub-types.
Diffs
-
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java
92217178
repository/src/main/java/org/apache/atlas/repository/impexp/ImportTransforms.java
63e53c50
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java
de8e7ef3
Diff: https://reviews.apache.org/r/60857/diff/1/
Testing
---
**Unit tests**
Unit test verifying this behavior were added.
**Functional tests**
Regular imports were run.
Thanks,
Ashutosh Mestry
Re: Review Request 60868: ATLAS-1940 :- Fix to remove duplicate type which causes Atlas server to fails on startup.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/60868/#review180535 --- Ship it! Ship It! - Ashutosh Mestry On July 14, 2017, 3:13 p.m., Nixon Rodrigues wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/60868/ > --- > > (Updated July 14, 2017, 3:13 p.m.) > > > Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and > Sarath Subramanian. > > > Bugs: ATLAS-1940 > https://issues.apache.org/jira/browse/ATLAS-1940 > > > Repository: atlas > > > Description > --- > > Problem:- Atlas server fails to initialize types on startup from graph store > due to duplicate type registered. > > Fix :- Added removeDuplicateTypeIfAny method which is called init of typeDef > Store to remove duplicate types if any. > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStore.java > 9d9c59d > > > Diff: https://reviews.apache.org/r/60868/diff/1/ > > > Testing > --- > > Exiting UT and IT are running fine. > Quick Start script running fine. > Created new tags & types. > Also tried creating duplicate type from rest client and UI which failed to > create duplicates. > > > Thanks, > > Nixon Rodrigues > >
Re: Review Request 61002: ATLAS-1968: Import File Signature Change
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61002/
---
(Updated July 20, 2017, 5:22 p.m.)
Review request for atlas and Madhan Neethiraj.
Changes
---
Updates include:
- Updated TWIKI documentation.
Bugs: ATLAS-1968
https://issues.apache.org/jira/browse/ATLAS-1968
Repository: atlas
Description
---
**Background**
The Import API that takes file as input was changed to make it consistent with
the other API that takes ZIP stream as input.
This change did not correctly reflect within _ImportService_.
**Solution**
_AtlasImportRequest_ has been updated to add the _fileName_ option.
**CURL**
Note that the file path should be a path on the server.
_importOptions.json_ contents:
```javascript
{
"options": {
"fileName": "/root/smalldb.zip"
}
}
```
CURL
```bash
curl -g -X POST -u admin:admin -H "Content-Type: application/json" -H
"Cache-Control: no-cache" -d @./importOptions.json
"http://localhost:21000/api/atlas/admin/importFile;
```
Diffs (updated)
-
docs/src/site/twiki/Import-API-Options.twiki 85887844
docs/src/site/twiki/Import-API.twiki b5de113f
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java
4d2ac627
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java
92217178
Diff: https://reviews.apache.org/r/61002/diff/2/
Changes: https://reviews.apache.org/r/61002/diff/1-2/
Testing
---
**Functional test**
Imported ZIP file using the new API.
**Unit tests**
None.
Thanks,
Ashutosh Mestry
Re: Review Request 61046: ATLAS-1960: Export, Import & ImportFile APIs EndPoints Handled in HA PASSIVE State
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61046/ --- (Updated July 21, 2017, 7:42 p.m.) Review request for atlas, Apoorv Naik, Madhan Neethiraj, and Nixon Rodrigues. Changes --- Updates include: - added _atlas_ group. Bugs: ATLAS-1960 https://issues.apache.org/jira/browse/ATLAS-1960 Repository: atlas Description --- **Background** Export & Import APIs did not generate correct error in HA-mode on a passive node. **Root Cause** The _ActiveServerFilter_ handles redirection based when HA-mode is active. There is a special treatment given to _admin_ URIs, where _admin_ URIs are allowed to execute even on PASSIVE node. **Solution** Since Export, Import APIs are on _AdminResource_, the special treatmean to _admin_ URIs has been extended to exclude execution of these API end points. Diffs - webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java 36f2688e webapp/src/test/java/org/apache/atlas/web/filters/ActiveServerFilterTest.java a8d11103 Diff: https://reviews.apache.org/r/61046/diff/1/ Testing --- **Unit tests** - Updated unit tests. - Executed existing tests. **Function tests** On local VM by changing application property and using debugger to step thru. Thanks, Ashutosh Mestry
Re: Review Request 61046: ATLAS-1960: Export, Import & ImportFile APIs EndPoints Handled in HA PASSIVE State
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61046/ --- (Updated July 21, 2017, 9:45 p.m.) Review request for atlas, Apoorv Naik, Madhan Neethiraj, and Nixon Rodrigues. Changes --- Updates include: - _Additional Comment_ section added. Bugs: ATLAS-1960 https://issues.apache.org/jira/browse/ATLAS-1960 Repository: atlas Description (updated) --- **Background** Export & Import APIs did not generate correct error in HA-mode on a passive node. **Root Cause** The _ActiveServerFilter_ handles redirection based when HA-mode is active. There is a special treatment given to _admin_ URIs, where _admin_ URIs are allowed to execute even on PASSIVE node. **Solution** Since Export, Import APIs are on _AdminResource_, the special treatmean to _admin_ URIs has been extended to exclude execution of these API end points. **Additional Comment** We may have to add few more method endpoints to this list, as those may encounter similar issue. Potentially, _metrics_ & _activeSearches_ may have this same problem. Diffs - webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java 36f2688e webapp/src/test/java/org/apache/atlas/web/filters/ActiveServerFilterTest.java a8d11103 Diff: https://reviews.apache.org/r/61046/diff/1/ Testing --- **Unit tests** - Updated unit tests. - Executed existing tests. **Function tests** On local VM by changing application property and using debugger to step thru. Thanks, Ashutosh Mestry
Re: Review Request 61103: ATLAS-1985 :- Fix URLEncoding while redirecting to Active instance in HA.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61103/#review181353 --- Ship it! Ship It! - Ashutosh Mestry On July 25, 2017, 11:30 a.m., Nixon Rodrigues wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/61103/ > --- > > (Updated July 25, 2017, 11:30 a.m.) > > > Review request for atlas, Apoorv Naik, Ashutosh Kelkar, Madhan Neethiraj, and > Sarath Subramanian. > > > Repository: atlas > > > Description > --- > > This patch provide fix for url-encoding in Activeserver Filter where url was > encoded wrongly with HtmlQuoting.quoteHtmlChars method which is primarily > meant for htmltext encoding. > > Instead of using HtmlQuoting.quoteHtmlChars method used URLEncode.encode > method to encode the URL. > > > Diffs > - > > webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java > 29d829f > > webapp/src/test/java/org/apache/atlas/web/filters/ActiveServerFilterTest.java > 3396072 > > > Diff: https://reviews.apache.org/r/61103/diff/1/ > > > Testing > --- > > Tested api redirected from passive to active > eg /api/atlas/discovery/search/fulltext?limit=100=hive_table API is > redirection to active with > /api/atlas/discovery/search/fulltext?limit%3D100%26query%3Dhive_table > where querystring is encoded. > > Existing UT and IT's passing. > Added new UT in ActiveServerFilterTest > > > Thanks, > > Nixon Rodrigues > >
Re: Review Request 60857: ATLAS-1950: Import API: Improvement: Specify Supertypes in Import Transforms
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60857/
---
(Updated July 26, 2017, 6:21 p.m.)
Review request for atlas, Ayub Pathan and Madhan Neethiraj.
Changes
---
Updates include:
- rebased.
Bugs: ATLAS-1950
https://issues.apache.org/jira/browse/ATLAS-1950
Repository: atlas
Description
---
**Background**
Existing import transforms expects that the user specify each type and
attribute name that needs to be transformed. This may cause large number of
transform enumerating individual types to be specified. Specifying individual
transforms may be tedious and perhaps error prone.
**Solution**
This feature improves on that case by allowing user to specify super types.
That way the sub types of that type will automatically get the transform(s)
applied.
**Implementation**
User can now specify transforms like:
```javascript
{
"options": {
"transforms": "{ \"Asset\": { \"qualifiedName\": [
\"replace:@cl1:@cl2\" ] } }"
}
}
```
At runtime, the subtypes of this type are enumerated. The attribute and
tranform map of the super type are now associated with all the sub-types.
Diffs (updated)
-
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java
bdeed7d
repository/src/main/java/org/apache/atlas/repository/impexp/ImportTransforms.java
63e53c5
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java
7210799
repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformsTest.java
ccedeb1
Diff: https://reviews.apache.org/r/60857/diff/3/
Changes: https://reviews.apache.org/r/60857/diff/2-3/
Testing
---
**Unit tests**
Unit test verifying this behavior were added.
**Functional tests**
Regular imports were run.
Thanks,
Ashutosh Mestry
Re: Review Request 60857: ATLAS-1950: Import API: Improvement: Specify Supertypes in Import Transforms
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60857/
---
(Updated July 19, 2017, 6:34 p.m.)
Review request for atlas, Ayub Pathan and Madhan Neethiraj.
Changes
---
Updates include:
- Addressed review comments.
- Updated unit tests.
Bugs: ATLAS-1950
https://issues.apache.org/jira/browse/ATLAS-1950
Repository: atlas
Description
---
**Background**
Existing import transforms expects that the user specify each type and
attribute name that needs to be transformed. This may cause large number of
transform enumerating individual types to be specified. Specifying individual
transforms may be tedious and perhaps error prone.
**Solution**
This feature improves on that case by allowing user to specify super types.
That way the sub types of that type will automatically get the transform(s)
applied.
**Implementation**
User can now specify transforms like:
```javascript
{
"options": {
"transforms": "{ \"Asset\": { \"qualifiedName\": [
\"replace:@cl1:@cl2\" ] } }"
}
}
```
At runtime, the subtypes of this type are enumerated. The attribute and
tranform map of the super type are now associated with all the sub-types.
Diffs (updated)
-
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java
92217178
repository/src/main/java/org/apache/atlas/repository/impexp/ImportTransforms.java
63e53c50
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java
404225c2
repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformsTest.java
ccedeb14
Diff: https://reviews.apache.org/r/60857/diff/2/
Changes: https://reviews.apache.org/r/60857/diff/1-2/
Testing
---
**Unit tests**
Unit test verifying this behavior were added.
**Functional tests**
Regular imports were run.
Thanks,
Ashutosh Mestry
Review Request 60816: Addresses Use of KafkaConsumer in Thread by Implementing Kafka's ShutdownableThread
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/60816/ --- Review request for atlas, Ayub Pathan, Madhan Neethiraj, and Nixon Rodrigues. Bugs: ATLAS-1944 https://issues.apache.org/jira/browse/ATLAS-1944 Repository: atlas Description --- **Background** Based on discussion with Kafka team member(s), turns out that _KafkaConsumer_ is not thread-safe. This causes probelem during service shutdown as noted in the ATLAS-1944. **Solution** Kafa provides _ShutdownableThread_ which implements thread-safe way for doing shutdown to _KafaConsumer_ as used in _AtlasKafkaConsumer_. Diffs - webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 9e5b864e Diff: https://reviews.apache.org/r/60816/diff/1/ Testing --- **Unit tests** Executed exisiting tests. **Functional tests** - Created entities via Hive. Thanks, Ashutosh Mestry
Review Request 61518: ATLAS-2033: Changed Pretty JSON Generation to Ugly for v1 Entity Serializers
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61518/ --- Review request for atlas and Nixon Rodrigues. Bugs: ATLAS-2033 https://issues.apache.org/jira/browse/ATLAS-2033 Repository: atlas Description --- **Implementation** Changed _writePretty_ to _write_ for GSON serializer for v1 entities. Diffs - typesystem/src/main/scala/org/apache/atlas/typesystem/json/InstanceSerialization.scala 3354d7c Diff: https://reviews.apache.org/r/61518/diff/1/ Testing --- **Unit tests** Existing suite. Thanks, Ashutosh Mestry
Review Request 61195: ATLAS-1995: Performance Improvement Using IndexQuery for Attribute-based Entity Lookup
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61195/ --- Review request for atlas, Madhan Neethiraj and Sarath Subramanian. Bugs: ATLAS-1995 https://issues.apache.org/jira/browse/ATLAS-1995 Repository: atlas Description --- See [ATLAS-1995](https://issues.apache.org/jira/browse/ATLAS-1995) for details. **Implementation** The _AtlasGraphUtilsV1.findByUniqueAttributes_ now determines if the attribute specified in the request is indexed. If yes, then index query will be used to fetch the entity. If not, the older graph query mechanism will be utilized. Diffs - repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasGraphUtilsV1.java 948d9dd webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 51276d3 Diff: https://reviews.apache.org/r/61195/diff/1/ Testing --- **Unit tests** None. **Functional tests** Verified the performance improvement using the _AtlasPerfTracer_. Thanks, Ashutosh Mestry
Review Request 61200: ATLAS-1996: Log4J Default Configuration Updated
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61200/ --- Review request for atlas and Madhan Neethiraj. Bugs: ATLAS-1996 https://issues.apache.org/jira/browse/ATLAS-1996 Repository: atlas Description --- **Problem** Existing log4J configuration did not include max size parameter. This caused logs to grow making it hard for human log readers. Another contributing factor to log size growth is TitanDB's _IndexSerializer_. When this is used, the query conversion from graph query to indexed query is written as info log entry. With the recent changes, many more index queries are used, thus contributing to growth of log files. **Solution** Added maxFileSize parameter. Changed _com.thinkaurelius.titan_ log level to _warn_. **Additional Info** Defaults need to be set in _Ambari_ so that they reflect correctly as part of deployment. Diffs - addons/hive-bridge/src/main/resources/atlas-hive-import-log4j.xml c1184e5 distro/src/conf/atlas-log4j.xml a0ef6a9 typesystem/src/main/resources/atlas-log4j.xml c30403e Diff: https://reviews.apache.org/r/61200/diff/1/ Testing --- **Functional tests** Observed configuration changes in the logs generated. Thanks, Ashutosh Mestry
Re: Review Request 61665: ATLAS-2047: Exception Thrown by Kafka Consumer Ends up Filling Logs Due to Incorrect Handling
> On Aug. 16, 2017, 9:59 a.m., Nixon Rodrigues wrote:
> > Ship It!
Can you please review? I have updated the patch with retry logic.
- Ashutosh
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61665/#review183030
---
On Aug. 15, 2017, 4:58 p.m., Ashutosh Mestry wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61665/
> ---
>
> (Updated Aug. 15, 2017, 4:58 p.m.)
>
>
> Review request for atlas, Madhan Neethiraj and Nixon Rodrigues.
>
>
> Bugs: ATLAS-2047
> https://issues.apache.org/jira/browse/ATLAS-2047
>
>
> Repository: atlas
>
>
> Description
> ---
>
> Please refer to
> [ATLAS-2047](https://issues.apache.org/jira/browse/ATLAS-2047) for background
> and analysis.
>
> **Implementation**
>
> Please take a look at this scala code. This is _ShutdownableThread_. The
> thread does the job of handling all exceptions. Upon exception, it manages
> the _shutdownLatch_ (from yesterday’s bug fix) and gets out of the
> _isRunning_ loop.
> ```scala
> override def run(): Unit = {
> info("Starting ")
> try{
> while(isRunning.get()){
> doWork()
> }
> } catch{
> case e: Throwable =>
> if(isRunning.get())
> error("Error due to ", e)
> }
> shutdownLatch.countDown()
> info("Stopped ")
> }
> ```
>
> Moved _try...catch_ block to leave exception handling to _ShutdownableThread_.
>
>
> Diffs
> -
>
>
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
> ef64c3b
>
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java
> a6f58e8
>
>
> Diff: https://reviews.apache.org/r/61665/diff/1/
>
>
> Testing
> ---
>
> **Unit tests**
> Updated unit tests to reproduce the scenarios and verify the fix.
>
> **Functional tests**
> Verified regular notification scenarios.
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 58434: Import API: Improved the API to Update Existing Types Attributes
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58434/
---
(Updated June 27, 2017, 5:27 a.m.)
Review request for atlas and Madhan Neethiraj.
Changes
---
Updates include:
- Addressed review comments.
- Fixed unit test by adding execution order.
- Unit test refactoring to reduce clutter.
Bugs: ATLAS-1734
https://issues.apache.org/jira/browse/ATLAS-1734
Repository: atlas
Description
---
**Background**
Existing version of Import API allows for importing types that are not already
present in the system being imported in. This causes import to fail in the
cases where the data being imported happens to have the additional attribute.
**Approach**
New class _TypeAttributeDifference_ does a attribute diff betweeing 2 types.
_ImportService.processTypeDefs_ method now includes additional step for
attribute updation.
Metrics have been updated to reflect the updates.
**CURL**
The ZIP files mentioned below can be found at: _repository/src/test/resources/_
Create a file _importOptions.json_ with contents below:
```javascript
{
"options": {
"updateTypeDefinition": true
}
}
```
Use the following curl to perform an import with default attributes.
```javascript
curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H
"Cache-Control: no-cache" -F data=@salesNewTypeAttrs.zip
"http://localhost:21000/api/atlas/admin/import;
```
Note the attributes of the tables.
Then perform this import with the new option.
```javascript
curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H
"Cache-Control: no-cache" -F request=@importOptions.json -F
data=@salesNewTypeAttrs-next.zip "http://localhost:21000/api/atlas/admin/import;
```
**Impact to API Consumer**
None.
Diffs (updated)
-
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java
b19f7097
repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java
7744c781
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java
4ffbb88c
repository/src/main/java/org/apache/atlas/repository/impexp/ImportTypeDefProcessor.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/TypeAttributeDifference.java
PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/AtlasImportRequestTest.java
0844bf0d
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java
a6fed628
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTestUtils.java
PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/TypeAttributeDifferenceTest.java
PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/UniqueListTest.java
PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java
612c445f
repository/src/test/resources/salesNewTypeAttrs-next.zip PRE-CREATION
repository/src/test/resources/salesNewTypeAttrs.zip PRE-CREATION
Diff: https://reviews.apache.org/r/58434/diff/5/
Changes: https://reviews.apache.org/r/58434/diff/4-5/
Testing
---
**Unit tests**
- _ImportServiceTest_ Updated to handle the new case. New light-weight _zip_
files added test resources to verify the functionality.
- _TypeAttributeDifferenceTest_ Unit test for _TypeAttributeDifference_ class.
- Additional unit tests: _UniqueList_.
- Import-Export APIs verified with test file set.
Thanks,
Ashutosh Mestry
Re: Review Request 61940: ATLAS-2090 : UI - Cache busting for static content (css, js)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61940/#review185305 --- Ship it! Ship It! - Ashutosh Mestry On Sept. 13, 2017, 2:11 p.m., keval bhatt wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/61940/ > --- > > (Updated Sept. 13, 2017, 2:11 p.m.) > > > Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, > Nixon Rodrigues, and Sarath Subramanian. > > > Bugs: ATLAS-2090 > https://issues.apache.org/jira/browse/ATLAS-2090 > > > Repository: atlas > > > Description > --- > > Grunt task is updated to handle Cache busting issue, so now after every > successful build grunt task will add new time as the query string for static > content i.e .CSS and .JS > > > Diffs > - > > dashboardv2/gruntfile.js 900ee40 > dashboardv2/package.json 6068c79 > dashboardv2/public/index.html f572049 > dashboardv2/public/index.html.tpl PRE-CREATION > dashboardv2/public/js/main.js 63e9abc > > > Diff: https://reviews.apache.org/r/61940/diff/2/ > > > Testing > --- > > Build the atlas code and checked the query string for static content i.e.CSS > and .JS, its working as expected. > > > Thanks, > > keval bhatt > >
Re: Review Request 62039: ATLAS-2100: UserProfile & SavedSearch Implementation
uot;admin"
"fullName" : "Admin Admin"
"savedSearchParameters": [
]
}
```
_AtlasSavedSearch_
```javascript
{
"name": "my_search1",
"ownerUserName": "admin",
"searchParameters":
}
```
Diffs (updated)
-
addons/models/0010-base_model.json 9ab4c44f
common/src/main/java/org/apache/atlas/repository/Constants.java 62d74f8f
intg/src/main/java/org/apache/atlas/AtlasErrorCode.java b20f5865
intg/src/main/java/org/apache/atlas/model/AtlasBaseModelObject.java
PRE-CREATION
intg/src/main/java/org/apache/atlas/model/SearchFilter.java 7dccf5ef
intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java
c79b5b90
intg/src/main/java/org/apache/atlas/model/profile/AtlasUserProfile.java
PRE-CREATION
intg/src/main/java/org/apache/atlas/model/profile/AtlasUserSavedSearch.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java
8196a674
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
1e688350
repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java
ca7fad06
repository/src/main/java/org/apache/atlas/repository/orm/AbstractDataTransferObject.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/orm/AtlasSavedSearchDTO.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/orm/AtlasUserProfileDTO.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/orm/DTORegistry.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/orm/DataAccess.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/orm/DataTransferObject.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasAbstractDefStoreV1.java
2bf53a1d
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityChangeNotifier.java
6423aeab
repository/src/main/java/org/apache/atlas/repository/userprofile/UserProfileService.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/util/FilterUtil.java
54d6b405
repository/src/test/java/org/apache/atlas/TestModules.java af3189b9
repository/src/test/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStoreTest.java
728d4180
repository/src/test/java/org/apache/atlas/repository/userprofile/UserProfileServiceTest.java
PRE-CREATION
webapp/src/main/java/org/apache/atlas/notification/NotificationEntityChangeListener.java
6eadc967
webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java 52258e3c
webapp/src/main/java/org/apache/atlas/web/rest/TypesREST.java c32f36ea
webapp/src/main/java/org/apache/atlas/web/util/Servlets.java 4a92763a
Diff: https://reviews.apache.org/r/62039/diff/5/
Changes: https://reviews.apache.org/r/62039/diff/4-5/
Testing
---
**Deployment**
- Copy _0010-base_model.json_ & _0011-user-profile_model.json_ to server
location: _/usr/hdp/current/atlas-server/models_
- Deploy _atlas.war_
**Unit tests**
Added unit tests for:
- _UserProfileService_.
**Functional testing**
- Verification using CURL calls.
- Verification via UI to ensure that internal entities do not show up.
Thanks,
Ashutosh Mestry
Re: Review Request 62382: ATLAS-2144 Add Knox x-forwarded path to Atlas base URL when Atlas is access via knox proxy
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62382/#review185556 --- webapp/src/main/java/org/apache/atlas/web/filters/AtlasKnoxSSOAuthenticationFilter.java Lines 336 (patched) <https://reviews.apache.org/r/62382/#comment261876> Optional: If this is broken in 2 functions, you could easily add tests. webapp/src/main/java/org/apache/atlas/web/filters/AtlasKnoxSSOAuthenticationFilter.java Lines 350 (patched) <https://reviews.apache.org/r/62382/#comment261875> Since this is external input, there may be some benefit in adding validation to the items that are being fetched from _httpRequest_. (See Fortify SCA issues.) webapp/src/main/java/org/apache/atlas/web/filters/AtlasKnoxSSOAuthenticationFilter.java Lines 362 (patched) <https://reviews.apache.org/r/62382/#comment261874> Consider using URI builder instead of hand constructing URLs. webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java Line 165 (original), 165 (patched) <https://reviews.apache.org/r/62382/#comment261877> For my education: What is the implication of this change? - Ashutosh Mestry On Sept. 18, 2017, 1:06 p.m., Nixon Rodrigues wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62382/ > --- > > (Updated Sept. 18, 2017, 1:06 p.m.) > > > Review request for atlas, Apoorv Naik, Ashutosh Mestry, keval bhatt, Madhan > Neethiraj, and Sarath Subramanian. > > > Bugs: ATLAS-2144 > https://issues.apache.org/jira/browse/ATLAS-2144 > > > Repository: atlas > > > Description > --- > > This patch includes fix to add Knox x-forwarded path to Atlas base URL when > Atlas is access via knox proxy. > > Also the changed the httpSecurity filter precendence between SSOFilter and > basicFilter. > When request dispatched from knox proxy is landed in atlas with basic header > and hadoopJwt cookie header, the basicFilter is invoked first before > ssoFilter causing issue in SSO login. > > > Diffs > - > > > webapp/src/main/java/org/apache/atlas/web/filters/AtlasKnoxSSOAuthenticationFilter.java > 665fa34 > webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java > 3bec838 > > > Diff: https://reviews.apache.org/r/62382/diff/1/ > > > Testing > --- > > Tested Atlas with knox authentication > Tested Atlas knox proxy with Form login and Knox SSO. > Tested Atlas form based Login > Tested Atlas api with basic and knox cookie header. > > > Thanks, > > Nixon Rodrigues > >
Re: Review Request 62039: ATLAS-2100: UserProfile & SavedSearch Implementation
javascript
{
"username": "admin"
"fullName" : "Admin Admin"
"savedSearchParameters": [
]
}
```
_AtlasSavedSearch_
```javascript
{
"name": "my_search1",
"ownerUserName": "admin",
"searchParameters":
}
```
Diffs (updated)
-
addons/models/0010-base_model.json 9ab4c44f
common/src/main/java/org/apache/atlas/repository/Constants.java 62d74f8f
intg/src/main/java/org/apache/atlas/AtlasErrorCode.java b20f5865
intg/src/main/java/org/apache/atlas/model/AtlasBaseModelObject.java
PRE-CREATION
intg/src/main/java/org/apache/atlas/model/SearchFilter.java 7dccf5ef
intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java
c79b5b90
intg/src/main/java/org/apache/atlas/model/profile/AtlasUserProfile.java
PRE-CREATION
intg/src/main/java/org/apache/atlas/model/profile/AtlasUserSavedSearch.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java
8196a674
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
1e688350
repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java
ca7fad06
repository/src/main/java/org/apache/atlas/repository/ogm/AbstractDataTransferObject.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/ogm/AtlasSavedSearchDTO.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/ogm/AtlasUserProfileDTO.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/ogm/DTORegistry.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/ogm/DataAccess.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/ogm/DataTransferObject.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasAbstractDefStoreV1.java
2bf53a1d
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityChangeNotifier.java
6423aeab
repository/src/main/java/org/apache/atlas/repository/userprofile/UserProfileService.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/util/FilterUtil.java
54d6b405
repository/src/test/java/org/apache/atlas/TestModules.java af3189b9
repository/src/test/java/org/apache/atlas/repository/userprofile/UserProfileServiceTest.java
PRE-CREATION
webapp/src/main/java/org/apache/atlas/notification/NotificationEntityChangeListener.java
6eadc967
webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java 52258e3c
webapp/src/main/java/org/apache/atlas/web/rest/TypesREST.java c32f36ea
webapp/src/main/java/org/apache/atlas/web/util/Servlets.java 4a92763a
Diff: https://reviews.apache.org/r/62039/diff/7/
Changes: https://reviews.apache.org/r/62039/diff/6-7/
Testing
---
**Deployment**
- Copy _0010-base_model.json_ & _0011-user-profile_model.json_ to server
location: _/usr/hdp/current/atlas-server/models_
- Deploy _atlas.war_
**Unit tests**
Added unit tests for:
- _UserProfileService_.
**Functional testing**
- Verification using CURL calls.
- Verification via UI to ensure that internal entities do not show up.
File Attachments
Patch for Master
https://reviews.apache.org/media/uploaded/files/2017/09/15/ba9b07c2-5868-4f79-ad51-7d7c9ee39c44__ATLAS-2100-Master-DataAccess.patch
Thanks,
Ashutosh Mestry
Review Request 62390: ATLAS-2146: Remove Powermock Usage
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62390/ --- Review request for atlas, Apoorv Naik and Sarath Subramanian. Bugs: ATLAS-2146 https://issues.apache.org/jira/browse/ATLAS-2146 Repository: atlas Description --- Refactored to use _VisibleForTesting_ annotation instead of _powermock.whitebox_. This is type safe. Diffs - repository/pom.xml e1e41d7 repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java ad21ee4 repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java de48573 repository/src/main/java/org/apache/atlas/repository/impexp/TypeAttributeDifference.java b3c790a repository/src/test/java/org/apache/atlas/discovery/EntityDiscoveryServiceTest.java PRE-CREATION repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java 8eb7a51 repository/src/test/java/org/apache/atlas/repository/impexp/TypeAttributeDifferenceTest.java cf17ef2 Diff: https://reviews.apache.org/r/62390/diff/1/ Testing --- **Unit tests** - Executed impacted unit tests. - Build with tests. Thanks, Ashutosh Mestry
Re: Review Request 62420: ATLAS-2148: Enhancement to Saved Search
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62420/ --- (Updated Sept. 19, 2017, 10:40 p.m.) Review request for atlas, keval bhatt and Madhan Neethiraj. Changes --- Updates includes: - JSONs for new query & updated query. Bugs: ATLAS-2148 https://issues.apache.org/jira/browse/ATLAS-2148 Repository: atlas Description --- **Implementation** - _DataAccess.save_ now returns entity with GUID set. Logic uses _EntityMutationResponse_ for inferring the GUID. - REST APIs have undergone minor changes. CURL calls updated (see below). _Please used attached JSON for the examples below._ **Retrieve List of Saved Searches** ```javascript curl -g -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" "http://localhost:21000/api/atlas/v2/search/saved; ``` **Retrieve Specific Saved Searches** ```javascript curl -g -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" "http://localhost:21000/api/atlas/v2/search/saved/BASIC/my_search1; ``` ```javascript curl -g -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" "http://localhost:21000/api/atlas/v2/search/saved/ADVANCED/my_search3; ``` **Save** ```javascript curl -g -X POST -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" "http://localhost:21000/api/atlas/v2/search/saved; -d @../docs/savedSearch.json ``` **Update** ```javascript curl -g -X PUT -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" "http://localhost:21000/api/atlas/v2/search/saved; -d @../docs/savedSearch.json ``` **Delete** ```javascript curl -g -X DELETE -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" "http://localhost:21000/api/atlas/v2/search/saved/0d0f75ef-b101-466f-843e-60196b81c98e; ``` Diffs - addons/models/0010-base_model.json 8f38f596 intg/src/main/java/org/apache/atlas/model/profile/AtlasUserSavedSearch.java b0698fcf repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java baaee85d repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java ad21ee48 repository/src/main/java/org/apache/atlas/repository/ogm/AtlasSavedSearchDTO.java 26eee200 repository/src/main/java/org/apache/atlas/repository/ogm/DataAccess.java bc93cc6e repository/src/main/java/org/apache/atlas/repository/userprofile/UserProfileService.java 766d1c7d repository/src/test/java/org/apache/atlas/repository/userprofile/UserProfileServiceTest.java 4e832968 webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java ad595c81 Diff: https://reviews.apache.org/r/62420/diff/2/ Testing --- **Unit tests** Updated to reflect new logic. **Functional tests** >From the new UI. CURL calls. File Attachments (updated) saved Search https://reviews.apache.org/media/uploaded/files/2017/09/19/f9f6ec2d-28f6-487a-8126-aaf6fc8a1028__savedSearch.json saved search for update https://reviews.apache.org/media/uploaded/files/2017/09/19/2b1d4619-65d8-4435-87cb-9b11878a7420__savedSearch-for-update.json Thanks, Ashutosh Mestry
Re: Review Request 62420: ATLAS-2148: Enhancement to Saved Search
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62420/ --- (Updated Sept. 19, 2017, 11:33 p.m.) Review request for atlas, keval bhatt and Madhan Neethiraj. Changes --- Updates include: - Additional refactoring. Bugs: ATLAS-2148 https://issues.apache.org/jira/browse/ATLAS-2148 Repository: atlas Description --- **Implementation** - _DataAccess.save_ now returns entity with GUID set. Logic uses _EntityMutationResponse_ for inferring the GUID. - REST APIs have undergone minor changes. CURL calls updated (see below). _Please used attached JSON for the examples below._ **Retrieve List of Saved Searches** ```javascript curl -g -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" "http://localhost:21000/api/atlas/v2/search/saved; ``` **Retrieve Specific Saved Searches** ```javascript curl -g -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" "http://localhost:21000/api/atlas/v2/search/saved/BASIC/my_search1; ``` ```javascript curl -g -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" "http://localhost:21000/api/atlas/v2/search/saved/ADVANCED/my_search3; ``` **Save** ```javascript curl -g -X POST -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" "http://localhost:21000/api/atlas/v2/search/saved; -d @../docs/savedSearch.json ``` **Update** ```javascript curl -g -X PUT -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" "http://localhost:21000/api/atlas/v2/search/saved; -d @../docs/savedSearch.json ``` **Delete** ```javascript curl -g -X DELETE -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" "http://localhost:21000/api/atlas/v2/search/saved/0d0f75ef-b101-466f-843e-60196b81c98e; ``` Diffs (updated) - addons/models/0010-base_model.json 8f38f596 intg/src/main/java/org/apache/atlas/model/profile/AtlasUserSavedSearch.java b0698fcf repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java baaee85d repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java ad21ee48 repository/src/main/java/org/apache/atlas/repository/ogm/AtlasSavedSearchDTO.java 26eee200 repository/src/main/java/org/apache/atlas/repository/ogm/DataAccess.java bc93cc6e repository/src/main/java/org/apache/atlas/repository/userprofile/UserProfileService.java 766d1c7d repository/src/test/java/org/apache/atlas/repository/userprofile/UserProfileServiceTest.java 4e832968 webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java ad595c81 Diff: https://reviews.apache.org/r/62420/diff/3/ Changes: https://reviews.apache.org/r/62420/diff/2-3/ Testing --- **Unit tests** Updated to reflect new logic. **Functional tests** >From the new UI. CURL calls. File Attachments saved Search https://reviews.apache.org/media/uploaded/files/2017/09/19/f9f6ec2d-28f6-487a-8126-aaf6fc8a1028__savedSearch.json saved search for update https://reviews.apache.org/media/uploaded/files/2017/09/19/2b1d4619-65d8-4435-87cb-9b11878a7420__savedSearch-for-update.json Thanks, Ashutosh Mestry
Review Request 62420: ATLAS-2148: Enhancement to Saved Search
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62420/ --- Review request for atlas, keval bhatt and Madhan Neethiraj. Bugs: ATLAS-2148 https://issues.apache.org/jira/browse/ATLAS-2148 Repository: atlas Description --- **Implementation** - _DataAccess.save_ now returns entity with GUID set. Logic uses _EntityMutationResponse_ for inferring the GUID. - REST APIs have undergone minor changes. CURL calls updated (see below). _Please used attached JSON for the examples below._ **Retrieve List of Saved Searches** ```javascript curl -g -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" "http://localhost:21000/api/atlas/v2/search/saved; ``` **Retrieve Specific Saved Searches** ```javascript curl -g -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" "http://localhost:21000/api/atlas/v2/search/saved/BASIC/my_search1; ``` ```javascript curl -g -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" "http://localhost:21000/api/atlas/v2/search/saved/ADVANCED/my_search3; ``` **Save** ```javascript curl -g -X POST -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" "http://localhost:21000/api/atlas/v2/search/saved; -d @../docs/savedSearch.json ``` **Update** ```javascript curl -g -X PUT -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" "http://localhost:21000/api/atlas/v2/search/saved; -d @../docs/savedSearch.json ``` **Delete** ```javascript curl -g -X DELETE -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" "http://localhost:21000/api/atlas/v2/search/saved/0d0f75ef-b101-466f-843e-60196b81c98e; ``` Diffs - addons/models/0010-base_model.json 8f38f59 intg/src/main/java/org/apache/atlas/model/profile/AtlasUserSavedSearch.java b0698fc repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java baaee85 repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java ad21ee4 repository/src/main/java/org/apache/atlas/repository/ogm/AtlasSavedSearchDTO.java 26eee20 repository/src/main/java/org/apache/atlas/repository/ogm/DataAccess.java bc93cc6 repository/src/main/java/org/apache/atlas/repository/userprofile/UserProfileService.java 766d1c7 repository/src/test/java/org/apache/atlas/repository/userprofile/UserProfileServiceTest.java 4e83296 webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java ad595c8 Diff: https://reviews.apache.org/r/62420/diff/1/ Testing --- **Unit tests** Updated to reflect new logic. **Functional tests** >From the new UI. CURL calls. File Attachments ---- Saved Search JSON https://reviews.apache.org/media/uploaded/files/2017/09/19/f845e2cf-3354-48e4-bfed-76b09a581af2__savedSearch.json Thanks, Ashutosh Mestry
Re: Review Request 62420: ATLAS-2148: Enhancement to Saved Search
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62420/ --- (Updated Sept. 19, 2017, 10:37 p.m.) Review request for atlas, keval bhatt and Madhan Neethiraj. Changes --- Updates include: - Minor refactoring. Bugs: ATLAS-2148 https://issues.apache.org/jira/browse/ATLAS-2148 Repository: atlas Description --- **Implementation** - _DataAccess.save_ now returns entity with GUID set. Logic uses _EntityMutationResponse_ for inferring the GUID. - REST APIs have undergone minor changes. CURL calls updated (see below). _Please used attached JSON for the examples below._ **Retrieve List of Saved Searches** ```javascript curl -g -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" "http://localhost:21000/api/atlas/v2/search/saved; ``` **Retrieve Specific Saved Searches** ```javascript curl -g -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" "http://localhost:21000/api/atlas/v2/search/saved/BASIC/my_search1; ``` ```javascript curl -g -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" "http://localhost:21000/api/atlas/v2/search/saved/ADVANCED/my_search3; ``` **Save** ```javascript curl -g -X POST -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" "http://localhost:21000/api/atlas/v2/search/saved; -d @../docs/savedSearch.json ``` **Update** ```javascript curl -g -X PUT -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" "http://localhost:21000/api/atlas/v2/search/saved; -d @../docs/savedSearch.json ``` **Delete** ```javascript curl -g -X DELETE -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" "http://localhost:21000/api/atlas/v2/search/saved/0d0f75ef-b101-466f-843e-60196b81c98e; ``` Diffs (updated) - addons/models/0010-base_model.json 8f38f596 intg/src/main/java/org/apache/atlas/model/profile/AtlasUserSavedSearch.java b0698fcf repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java baaee85d repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java ad21ee48 repository/src/main/java/org/apache/atlas/repository/ogm/AtlasSavedSearchDTO.java 26eee200 repository/src/main/java/org/apache/atlas/repository/ogm/DataAccess.java bc93cc6e repository/src/main/java/org/apache/atlas/repository/userprofile/UserProfileService.java 766d1c7d repository/src/test/java/org/apache/atlas/repository/userprofile/UserProfileServiceTest.java 4e832968 webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java ad595c81 Diff: https://reviews.apache.org/r/62420/diff/2/ Changes: https://reviews.apache.org/r/62420/diff/1-2/ Testing --- **Unit tests** Updated to reflect new logic. **Functional tests** >From the new UI. CURL calls. File Attachments ---- Saved Search JSON https://reviews.apache.org/media/uploaded/files/2017/09/19/f845e2cf-3354-48e4-bfed-76b09a581af2__savedSearch.json Thanks, Ashutosh Mestry
Review Request 62484: ATLAS-2160: Execute Saved Searches
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62484/ --- Review request for atlas, keval bhatt and Madhan Neethiraj. Bugs: ATLAS-2160 https://issues.apache.org/jira/browse/ATLAS-2160 Repository: atlas Description --- **Implementation** There are 2 additional APIs: * Saved Search using _type_ and _name_. * Saved search using _guid_. Once _AtlasUserSavedSearch_ is executed, execute _basic_ or _advanced_ search using the _searchType_. The results are returned in _AtlasSearchResult_ form. **CURL** Search results using saved search using search name and search type: ```javascript curl -g -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" "http://localhost:21000/api/atlas/v2/search/saved/execute/BASIC/Tax%20Tables; ``` Search results using saved search GUID: ```javascript curl -g -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" "http://localhost:21000/api/atlas/v2/search/saved/execute/18a21a1b-af61-47c6-a50a-34e8ae81dad3; ``` Diffs - webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java aed82ae0 Diff: https://reviews.apache.org/r/62484/diff/1/ Testing --- **Unit tests** None **Functional tests** Executed APIs using CURL calls. Thanks, Ashutosh Mestry
Re: Review Request 62484: ATLAS-2160: Execute Saved Searches
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62484/ --- (Updated Sept. 22, 2017, 8:46 p.m.) Review request for atlas, keval bhatt and Madhan Neethiraj. Changes --- Updates include: - Added new REST APIs for execute saved search. - Removed notion of _BASIC_ and _ADVANCED_ in REST end points & core logic. - Minor refactoring. Bugs: ATLAS-2160 https://issues.apache.org/jira/browse/ATLAS-2160 Repository: atlas Description (updated) --- **Implementation** There are 2 additional APIs: * Saved Search using _type_ and _name_. * Saved search using _guid_. Once _AtlasUserSavedSearch_ is executed, execute _basic_ or _advanced_ search using the _searchType_. The results are returned in _AtlasSearchResult_ form. **CURL** Retrieve Saved Search: ```javascript curl -g -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" "http://localhost:21000/api/atlas/v2/search/saved/Tax%20Tables; ``` Search results using saved search using search name and search type: ```javascript curl -g -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" "http://localhost:21000/api/atlas/v2/search/saved/execute/Tax%20Tables; ``` Search results using saved search GUID: ```javascript curl -g -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" "http://localhost:21000/api/atlas/v2/search/saved/execute/guid/18a21a1b-af61-47c6-a50a-34e8ae81dad3; ``` Diffs - webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java aed82ae0 Diff: https://reviews.apache.org/r/62484/diff/1/ Testing (updated) --- **Unit tests** Executed existing tests. **Functional tests** Executed APIs using CURL calls. Thanks, Ashutosh Mestry
Re: Review Request 62484: ATLAS-2160: Execute Saved Searches
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62484/ --- (Updated Sept. 22, 2017, 8:49 p.m.) Review request for atlas, keval bhatt and Madhan Neethiraj. Changes --- Update include: - Uploaded new patch. Bugs: ATLAS-2160 https://issues.apache.org/jira/browse/ATLAS-2160 Repository: atlas Description --- **Implementation** There are 2 additional APIs: * Saved Search using _type_ and _name_. * Saved search using _guid_. Once _AtlasUserSavedSearch_ is executed, execute _basic_ or _advanced_ search using the _searchType_. The results are returned in _AtlasSearchResult_ form. **CURL** Retrieve Saved Search: ```javascript curl -g -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" "http://localhost:21000/api/atlas/v2/search/saved/Tax%20Tables; ``` Search results using saved search using search name and search type: ```javascript curl -g -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" "http://localhost:21000/api/atlas/v2/search/saved/execute/Tax%20Tables; ``` Search results using saved search GUID: ```javascript curl -g -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" "http://localhost:21000/api/atlas/v2/search/saved/execute/guid/18a21a1b-af61-47c6-a50a-34e8ae81dad3; ``` Diffs (updated) - repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java 630e776 repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 102c329 repository/src/main/java/org/apache/atlas/repository/ogm/AtlasSavedSearchDTO.java aa8fc39 repository/src/main/java/org/apache/atlas/repository/userprofile/UserProfileService.java 99d13af webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java aed82ae Diff: https://reviews.apache.org/r/62484/diff/2/ Changes: https://reviews.apache.org/r/62484/diff/1-2/ Testing --- **Unit tests** Executed existing tests. **Functional tests** Executed APIs using CURL calls. Thanks, Ashutosh Mestry
Re: Review Request 62484: ATLAS-2160: Execute Saved Searches
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62484/ --- (Updated Sept. 22, 2017, 9:53 p.m.) Review request for atlas, keval bhatt and Madhan Neethiraj. Changes --- Updates include: - Removed search types from method signatures. Bugs: ATLAS-2160 https://issues.apache.org/jira/browse/ATLAS-2160 Repository: atlas Description --- **Implementation** There are 2 additional APIs: * Saved Search using _type_ and _name_. * Saved search using _guid_. Once _AtlasUserSavedSearch_ is executed, execute _basic_ or _advanced_ search using the _searchType_. The results are returned in _AtlasSearchResult_ form. **CURL** Retrieve Saved Search: ```javascript curl -g -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" "http://localhost:21000/api/atlas/v2/search/saved/Tax%20Tables; ``` Search results using saved search using search name and search type: ```javascript curl -g -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" "http://localhost:21000/api/atlas/v2/search/saved/execute/Tax%20Tables; ``` Search results using saved search GUID: ```javascript curl -g -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" "http://localhost:21000/api/atlas/v2/search/saved/execute/guid/18a21a1b-af61-47c6-a50a-34e8ae81dad3; ``` Diffs (updated) - intg/src/main/java/org/apache/atlas/model/profile/AtlasUserSavedSearch.java 8ecd52d9 repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java 630e776a repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 102c3292 repository/src/main/java/org/apache/atlas/repository/ogm/AtlasSavedSearchDTO.java aa8fc399 repository/src/main/java/org/apache/atlas/repository/userprofile/UserProfileService.java 99d13afb repository/src/test/java/org/apache/atlas/repository/userprofile/UserProfileServiceTest.java 713dade2 webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java aed82ae0 Diff: https://reviews.apache.org/r/62484/diff/4/ Changes: https://reviews.apache.org/r/62484/diff/3-4/ Testing --- **Unit tests** Executed existing tests. **Functional tests** Executed APIs using CURL calls. Thanks, Ashutosh Mestry
Re: Review Request 61807: ATLAS-2075: Kafka Messages: Split & Combine Implementation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61807/ --- (Updated Sept. 20, 2017, 10:43 p.m.) Review request for atlas, Madhan Neethiraj and Nixon Rodrigues. Changes --- Updates include: - Implemented _VersionedMessageHeader_ for lighter deserialization. - Removed dependence on version number to identify message types. - Added/updated unit tests. - Simplified serialization process. Bugs: ATLAS-2075 https://issues.apache.org/jira/browse/ATLAS-2075 Repository: atlas Description --- **Approach** - Message size is checked. It is split into parts if it exceeds the size threshold. - New message type _Split_ derived from _VersionedMessage_ holds information of the individual parts of the message. - _MessageSplitter_ splits message into smaller packets and encloses them in a message of type _Split_. - _MessageCombiner_ combines array of split messages. - _VersionedMessageDeserializer_ is modified to support the new message type. **Additional configuration item** - _atlas.notification.message.size.threshold_ Needs to be added to Hive's atlas configuration as well as _Atlas_'s own _atlas-application.properties_. - Default is set to a number closer to 1MB. Diffs (updated) - intg/src/main/java/org/apache/atlas/type/AtlasType.java ae7044d3 intg/src/test/java/org/apache/atlas/TestAtlasTypeJSONSerialize.java PRE-CREATION notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java d3b4e49e notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 38889ef8 notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java cb44fc68 notification/src/main/java/org/apache/atlas/notification/MessageCombiner.java PRE-CREATION notification/src/main/java/org/apache/atlas/notification/MessageTransformer.java PRE-CREATION notification/src/main/java/org/apache/atlas/notification/MessageVersion.java 6ef407ac notification/src/main/java/org/apache/atlas/notification/VersionedMessage.java 1929eb46 notification/src/main/java/org/apache/atlas/notification/VersionedMessageDeserializer.java cc2099e7 notification/src/main/java/org/apache/atlas/notification/VersionedMessageHeader.java PRE-CREATION notification/src/main/java/org/apache/atlas/notification/messages/Compressed.java PRE-CREATION notification/src/main/java/org/apache/atlas/notification/messages/Split.java PRE-CREATION notification/src/main/java/org/apache/atlas/utils/StringSplitterCombiner.java PRE-CREATION notification/src/test/java/org/apache/atlas/hook/StringSplitCombineTest.java PRE-CREATION notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java 3b2a093b notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java 61107a9f notification/src/test/java/org/apache/atlas/notification/MessageCombinerTest.java PRE-CREATION notification/src/test/java/org/apache/atlas/notification/MessageSplitterTest.java PRE-CREATION notification/src/test/java/org/apache/atlas/notification/SplitCombinerUtils.java PRE-CREATION notification/src/test/java/org/apache/atlas/notification/entity/EntityMessageDeserializerTest.java be324277 notification/src/test/java/org/apache/atlas/notification/hook/HookMessageDeserializerTest.java 3724fd5f notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationTest.java dd3257e0 webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 2669c739 Diff: https://reviews.apache.org/r/61807/diff/2/ Changes: https://reviews.apache.org/r/61807/diff/1-2/ Testing --- **Unit tests** - Added unit tests for new functionality. - Refactored few of existing unit tests. - Added tests to verify backward compatibility. **Functional tests** - Verified regular flow. - Verified cases: - Case where server is shutdown before all the parts of a message are committed. Thanks, Ashutosh Mestry
Re: Review Request 62255: ATLAS-2129: Addressed Case Where Abrupt Shutdown of Atlas During Import Retains Entities Imported Before Shutdown
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62255/ --- (Updated Sept. 14, 2017, 10:23 p.m.) Review request for atlas and Madhan Neethiraj. Changes --- Updates includes: - Moved _getGraphTransactionInterceptor_ method to _AtlasGraphUtilsV1_. Bugs: ATLAS-2129 https://issues.apache.org/jira/browse/ATLAS-2129 Repository: atlas Description --- **Background** Existing implementation of _bulkImport_ was done in single transaction. This can potentially lead to case where abrupt shutdown of Atlas during Import will not retain entities already imported, this renders resume Import cability useless. Spring DI does not support case where from within a class annotated method is invoked. The method interception does not happen. This is a known limitation. In the current code base, _Guice_ is used as DI framework. This caused the case where the issue did not get detected from unit tests. Also, the UI does not render correctly when new code is deployed. This caused failure during visual inspection. Work around is to use a clean browser using _incognito_ mdoe for each test. Cache busting mechanism has been implemented, this should address this situation. **Implementation** - Removed _GraphTransaction_ from _bulkImport_. - New method _createOrUpdateForImport_ which is under trasaction scope. Diffs (updated) - repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java c6a4bbe repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java ca7fad0 repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java a4163f2 repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java dc6416a repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasGraphUtilsV1.java d1fd16f Diff: https://reviews.apache.org/r/62255/diff/3/ Changes: https://reviews.apache.org/r/62255/diff/2-3/ Testing --- **Unit tests** No new tests added. **Functional tests** Import API tested. Thanks, Ashutosh Mestry
Re: Review Request 62255: ATLAS-2129: Addressed Case Where Abrupt Shutdown of Atlas During Import Retains Entities Imported Before Shutdown
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62255/ --- (Updated Sept. 14, 2017, 11:41 p.m.) Review request for atlas and Madhan Neethiraj. Changes --- Updates include: - Changed approach to addressing the problem. - New interface & class _BulkImporter_ & _BulkImporterImpl_. - Refactored _ImportServiceTest_. Bugs: ATLAS-2129 https://issues.apache.org/jira/browse/ATLAS-2129 Repository: atlas Description --- **Background** Existing implementation of _bulkImport_ was done in single transaction. This can potentially lead to case where abrupt shutdown of Atlas during Import will not retain entities already imported, this renders resume Import cability useless. Spring DI does not support case where from within a class annotated method is invoked. The method interception does not happen. This is a known limitation. In the current code base, _Guice_ is used as DI framework. This caused the case where the issue did not get detected from unit tests. Also, the UI does not render correctly when new code is deployed. This caused failure during visual inspection. Work around is to use a clean browser using _incognito_ mdoe for each test. Cache busting mechanism has been implemented, this should address this situation. **Implementation** - Removed _GraphTransaction_ from _bulkImport_. - New method _createOrUpdateForImport_ which is under trasaction scope. Diffs (updated) - repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java bbe9ed0 repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java a4163f2 repository/src/main/java/org/apache/atlas/repository/store/graph/BulkImporter.java PRE-CREATION repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java dc6416a repository/src/main/java/org/apache/atlas/repository/store/graph/v1/BulkImporterImpl.java PRE-CREATION repository/src/test/java/org/apache/atlas/TestModules.java 9efbb77 repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java a3fb521 Diff: https://reviews.apache.org/r/62255/diff/4/ Changes: https://reviews.apache.org/r/62255/diff/3-4/ Testing (updated) --- **Unit tests** - Executed refactored unit tests. **Functional tests** - Import API tested. - Import validated via UI (on multiple browsers). - Verified abrupt abort scenario. Thanks, Ashutosh Mestry
Re: Review Request 62382: ATLAS-2144 Add Knox x-forwarded path to Atlas base URL when Atlas is access via knox proxy
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62382/#review186004 --- Ship it! Ship It! webapp/src/main/java/org/apache/atlas/web/filters/AtlasKnoxSSOAuthenticationFilter.java Lines 425 (patched) <https://reviews.apache.org/r/62382/#comment262352> You could remove this if you don't have corresponding tests. - Ashutosh Mestry On Sept. 22, 2017, 5:32 p.m., Nixon Rodrigues wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62382/ > --- > > (Updated Sept. 22, 2017, 5:32 p.m.) > > > Review request for atlas, Apoorv Naik, Ashutosh Mestry, keval bhatt, Madhan > Neethiraj, and Sarath Subramanian. > > > Bugs: ATLAS-2144 > https://issues.apache.org/jira/browse/ATLAS-2144 > > > Repository: atlas > > > Description > --- > > This patch includes fix to add Knox x-forwarded path to Atlas base URL when > Atlas is access via knox proxy. > > Also the changed the httpSecurity filter precendence between SSOFilter and > basicFilter. > When request dispatched from knox proxy is landed in atlas with basic header > and hadoopJwt cookie header, the basicFilter is invoked first before > ssoFilter causing issue in SSO login. > > > Diffs > - > > > webapp/src/main/java/org/apache/atlas/web/filters/AtlasKnoxSSOAuthenticationFilter.java > 665fa34 > webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java > 3bec838 > > > Diff: https://reviews.apache.org/r/62382/diff/2/ > > > Testing > --- > > Tested Atlas with knox authentication > Tested Atlas knox proxy with Form login and Knox SSO. > Tested Atlas form based Login > Tested Atlas api with basic and knox cookie header. > > > Thanks, > > Nixon Rodrigues > >
Re: Review Request 62797: ATLAS-2192: Split Join Messages Improved Logging
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62797/ --- (Updated Oct. 6, 2017, 10:48 p.m.) Review request for atlas and Madhan Neethiraj. Changes --- Updates include: - Added messages metrics. - Additional fields in _AtlasNotificationMessage_. - Updated hive hook to set new fields. Bugs: ATLAS-2190 https://issues.apache.org/jira/browse/ATLAS-2190 Repository: atlas Description --- **Implementation** - Added _SplitMessageAggregator_ that encapsulates some of the functionality of _AtlasNotificationStringMessage_ buffer. - Added mechanism to detect stale split messages and evict them with appropriate message in the log. Diffs (updated) - common/src/main/java/org/apache/atlas/AtlasConfiguration.java 451bd9d8 notification/src/main/java/org/apache/atlas/hook/AtlasHook.java a8609e6b notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java 6a707346 notification/src/main/java/org/apache/atlas/notification/AtlasNotificationMessage.java 2f6f9c71 notification/src/main/java/org/apache/atlas/notification/AtlasNotificationMessageDeserializer.java 3d802840 notification/src/main/java/org/apache/atlas/notification/AtlasNotificationStringMessage.java 193735c1 notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java a7878624 notification/src/main/java/org/apache/atlas/notification/SplitMessageAggregator.java PRE-CREATION notification/src/test/java/org/apache/atlas/notification/SplitMessageAggregatorTest.java PRE-CREATION Diff: https://reviews.apache.org/r/62797/diff/2/ Changes: https://reviews.apache.org/r/62797/diff/1-2/ Testing --- **Unit tests** - Added existing. - Executed existing. **Functional testing** - Executed tests with large messages. Thanks, Ashutosh Mestry
Review Request 62755: ATLAS-2186: Jetty Downgraded to v8
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62755/ --- Review request for atlas, Ayub Pathan, Madhan Neethiraj, Nixon Rodrigues, and Vishal Suvagia. Bugs: ATLAS-2816 https://issues.apache.org/jira/browse/ATLAS-2816 Repository: atlas Description --- **Background** Upgrading to newer version of Jetty, although fixes some of the knows problems with v9.2, also introduces new problems. The latest one being in kerberos environment. Newer version have reported problems with increased CPU usage in SSL environment. **Implementation** - Downgraded Jetty to v8. - Re-implemented _EmbeddedServer.getConnector_ to use the classes in the older version. - Updated _jetty-maven-plugin_. **Developer verification** Please ensure that _.m2_ repository is clean if not, some of the dependencies won't be resolved correctly leading to _' javax.servlet.jsp.JspApplicationContext's signer information does not match signer information of other classes in the same package'_ during startup. Diffs - pom.xml c8e7ee7a webapp/src/main/java/org/apache/atlas/LocalServletRequest.java 36a01b22 webapp/src/main/java/org/apache/atlas/web/service/EmbeddedServer.java 467571e8 webapp/src/main/java/org/apache/atlas/web/service/SecureEmbeddedServer.java fa02e9b9 Diff: https://reviews.apache.org/r/62755/diff/1/ Testing --- **Functional** - SSL - Non-SSL **Unit tests** - Existing tests. Thanks, Ashutosh Mestry
Review Request 62797: ATLAS-2192: Split Join Messages Improved Logging
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62797/ --- Review request for atlas and Madhan Neethiraj. Bugs: ATLAS-2190 https://issues.apache.org/jira/browse/ATLAS-2190 Repository: atlas Description --- **Implementation** - Added _SplitMessageAggregator_ that encapsulates some of the functionality of _AtlasNotificationStringMessage_ buffer. - Added mechanism to detect stale split messages and evict them with appropriate message in the log. Diffs - common/src/main/java/org/apache/atlas/AtlasConfiguration.java 451bd9d8 notification/src/main/java/org/apache/atlas/notification/AtlasNotificationMessageDeserializer.java 3d802840 notification/src/main/java/org/apache/atlas/notification/SplitMessageAggregator.java PRE-CREATION notification/src/test/java/org/apache/atlas/notification/SplitMessageAggregatorTest.java PRE-CREATION Diff: https://reviews.apache.org/r/62797/diff/1/ Testing --- **Unit tests** - Added existing. - Executed existing. **Functional testing** - Executed tests with large messages. Thanks, Ashutosh Mestry
Re: Review Request 62957: ATLAS-2195: Addressing Txn Leak During Startup
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62957/ --- (Updated Oct. 14, 2017, 1:31 a.m.) Review request for atlas, Madhan Neethiraj and Sarath Subramanian. Changes --- Updates include: - Part 2 of the patch. Bugs: ATLAS-2195 https://issues.apache.org/jira/browse/ATLAS-2195 Repository: atlas Description (updated) --- **Background** If entities and types are created simultaneously it was observed that some stale transactions are created. Titan runs an eviction thread every minute to evict these out of transaction cache. That is when the message mentioned in the bug appears. **Analysis** During startup when built-in types are added to the database, the process did not use _GraphTransactionInterceptor_ approach. This caused stale transactions to appear, which grew over time. ManagementSystem class in Titan creates a transaction object in its ctor. The user is expected to close this when done with using it. **Analysis for Part 2** In our _GraphBackedSearchIndexer.getVertexIndexKeys_ we use this to get index keys. This is invoked for every basic search. Interestingly, new object is not created for each basic search. Only 1 transaction stays open if user just keeps searching. Behavior changes when new tag is created. The count of open transactions keeps incrementing. **Instrumentation** Used _Titan's_ _StandardTitanGraph_ for adding instrumentation code. This is central point for transaction operations. Tracing the creation and co-relating them with destruction helped in narrowing down the area of focus. Diffs - repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 6e9ef685 Diff: https://reviews.apache.org/r/62957/diff/1/ Testing --- **Unit tests** Existing. **Other** Executed the attached bash scripts to create numerous types and entities simultaneously. Use this command: ``` bash create-type.sh & bash create-entity.sh & ``` File Attachments create type https://reviews.apache.org/media/uploaded/files/2017/10/12/755455cc-c08c-4885-9d07-5b5d9e3867e2__create-type.sh create entity https://reviews.apache.org/media/uploaded/files/2017/10/12/b20ddb13-7300-493d-8f43-5fcaf884c055__create-entity.sh Thanks, Ashutosh Mestry
Re: Review Request 62957: ATLAS-2195: Addressing Txn Leak During Startup
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62957/ --- (Updated Oct. 14, 2017, 1:34 a.m.) Review request for atlas, Madhan Neethiraj and Sarath Subramanian. Changes --- Updated patch. Bugs: ATLAS-2195 https://issues.apache.org/jira/browse/ATLAS-2195 Repository: atlas Description --- **Background** If entities and types are created simultaneously it was observed that some stale transactions are created. Titan runs an eviction thread every minute to evict these out of transaction cache. That is when the message mentioned in the bug appears. **Analysis** During startup when built-in types are added to the database, the process did not use _GraphTransactionInterceptor_ approach. This caused stale transactions to appear, which grew over time. ManagementSystem class in Titan creates a transaction object in its ctor. The user is expected to close this when done with using it. **Analysis for Part 2** In our _GraphBackedSearchIndexer.getVertexIndexKeys_ we use this to get index keys. This is invoked for every basic search. Interestingly, new object is not created for each basic search. Only 1 transaction stays open if user just keeps searching. Behavior changes when new tag is created. The count of open transactions keeps incrementing. **Instrumentation** Used _Titan's_ _StandardTitanGraph_ for adding instrumentation code. This is central point for transaction operations. Tracing the creation and co-relating them with destruction helped in narrowing down the area of focus. Diffs (updated) - repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 8aa22377 Diff: https://reviews.apache.org/r/62957/diff/2/ Changes: https://reviews.apache.org/r/62957/diff/1-2/ Testing --- **Unit tests** Existing. **Other** Executed the attached bash scripts to create numerous types and entities simultaneously. Use this command: ``` bash create-type.sh & bash create-entity.sh & ``` File Attachments create type https://reviews.apache.org/media/uploaded/files/2017/10/12/755455cc-c08c-4885-9d07-5b5d9e3867e2__create-type.sh create entity https://reviews.apache.org/media/uploaded/files/2017/10/12/b20ddb13-7300-493d-8f43-5fcaf884c055__create-entity.sh Thanks, Ashutosh Mestry
Re: Review Request 61200: ATLAS-1996: Log4J Default Configuration Updated
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61200/ --- (Updated Sept. 8, 2017, 4:28 p.m.) Review request for atlas and Madhan Neethiraj. Changes --- Updates include: - Addressed review comment. Bugs: ATLAS-1996 https://issues.apache.org/jira/browse/ATLAS-1996 Repository: atlas Description --- **Problem** Existing log4J configuration did not include max size parameter. This caused logs to grow making it hard for human log readers. Another contributing factor to log size growth is TitanDB's _IndexSerializer_. When this is used, the query conversion from graph query to indexed query is written as info log entry. With the recent changes, many more index queries are used, thus contributing to growth of log files. **Solution** Added maxFileSize parameter. Changed _com.thinkaurelius.titan_ log level to _warn_. **Additional Info** Defaults need to be set in _Ambari_ so that they reflect correctly as part of deployment. Diffs (updated) - addons/hive-bridge/src/main/resources/atlas-hive-import-log4j.xml c1184e5 distro/src/conf/atlas-log4j.xml a0ef6a9 typesystem/src/main/resources/atlas-log4j.xml c30403e Diff: https://reviews.apache.org/r/61200/diff/3/ Changes: https://reviews.apache.org/r/61200/diff/2-3/ Testing --- **Functional tests** Observed configuration changes in the logs generated. Thanks, Ashutosh Mestry
Re: Review Request 61665: ATLAS-2047: Exception Thrown by Kafka Consumer Ends up Filling Logs Due to Incorrect Handling
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61665/
---
(Updated Aug. 29, 2017, 9:01 p.m.)
Review request for atlas, Madhan Neethiraj and Nixon Rodrigues.
Changes
---
Updates include:
- _AdaptiveWaiter_ implementation.
- Added unit tests for new implementation.
Bugs: ATLAS-2047
https://issues.apache.org/jira/browse/ATLAS-2047
Repository: atlas
Description
---
Please refer to [ATLAS-2047](https://issues.apache.org/jira/browse/ATLAS-2047)
for background and analysis.
**Background**
The _IllegalStateException_ is thrown by _KafkaConsumer.aquire_. This method is
called at the beginning of almost every method in this class. The method checks
if the consumer is closed, if it is then it throws IllegalStateException.
Scenario may come about in this way:
- Shutdown has been initiated. Close on consumer is called.
- However, the consumer thread is just about to enter another poll cycle.
- Thus acquire sees that consumer is closed and throws the exception (2nd
bullet above).
Please take a look at this scala code. This is _ShutdownableThread_. The thread
does the job of handling all exceptions. Upon exception, it manages the
_shutdownLatch_ (from yesterday’s bug fix) and gets out of the _isRunning_ loop.
```scala
override def run(): Unit = {
info("Starting ")
try{
while(isRunning.get()){
doWork()
}
} catch{
case e: Throwable =>
if(isRunning.get())
error("Error due to ", e)
}
shutdownLatch.countDown()
info("Stopped ")
}
```
**Implementation**
Special treatment is given to _IllegalStateException_ by implementing pause &
retry logic:
- Modified _LOG_ to _debug_. That way logs are not filled during retry.
- _HookConsumer_ is more resilient. It handles exceptions resulting from
_Kafka_ and entity APIs.
Diffs (updated)
-
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
ef64c3b
webapp/src/test/java/org/apache/atlas/notification/AdaptiveWaiterTest.java
PRE-CREATION
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java
a6f58e8
Diff: https://reviews.apache.org/r/61665/diff/3/
Changes: https://reviews.apache.org/r/61665/diff/2-3/
Testing
---
**Unit tests**
Updated unit tests to reproduce the scenarios and verify the fix.
**Functional tests**
Verified regular notification scenarios.
Thanks,
Ashutosh Mestry
Re: Review Request 61940: ATLAS-2090 : UI - Cache busting for static content (css, js)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61940/#review184874 --- Ship it! Ship It! - Ashutosh Mestry On Aug. 28, 2017, 1:22 p.m., keval bhatt wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/61940/ > --- > > (Updated Aug. 28, 2017, 1:22 p.m.) > > > Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, > Nixon Rodrigues, and Sarath Subramanian. > > > Bugs: ATLAS-2090 > https://issues.apache.org/jira/browse/ATLAS-2090 > > > Repository: atlas > > > Description > --- > > Grunt task is updated to handle Cache busting issue, so now after every > successful build grunt task will add new time as the query string for static > content i.e .CSS and .JS > > > Diffs > - > > dashboardv2/gruntfile.js 900ee40 > dashboardv2/package.json 6068c79 > dashboardv2/public/index.html f572049 > dashboardv2/public/index.html.tpl PRE-CREATION > dashboardv2/public/js/main.js 63e9abc > > > Diff: https://reviews.apache.org/r/61940/diff/1/ > > > Testing > --- > > Build the atlas code and checked the query string for static content i.e.CSS > and .JS, its working as expected. > > > Thanks, > > keval bhatt > >
Re: Review Request 61940: ATLAS-2090 : UI - Cache busting for static content (css, js)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61940/#review184873 --- dashboardv2/public/index.html.tpl Lines 75 (patched) <https://reviews.apache.org/r/61940/#comment261068> Support _require.js_ with _bust_. ```javascript ``` - Ashutosh Mestry On Aug. 28, 2017, 1:22 p.m., keval bhatt wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/61940/ > --- > > (Updated Aug. 28, 2017, 1:22 p.m.) > > > Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, > Nixon Rodrigues, and Sarath Subramanian. > > > Bugs: ATLAS-2090 > https://issues.apache.org/jira/browse/ATLAS-2090 > > > Repository: atlas > > > Description > --- > > Grunt task is updated to handle Cache busting issue, so now after every > successful build grunt task will add new time as the query string for static > content i.e .CSS and .JS > > > Diffs > - > > dashboardv2/gruntfile.js 900ee40 > dashboardv2/package.json 6068c79 > dashboardv2/public/index.html f572049 > dashboardv2/public/index.html.tpl PRE-CREATION > dashboardv2/public/js/main.js 63e9abc > > > Diff: https://reviews.apache.org/r/61940/diff/1/ > > > Testing > --- > > Build the atlas code and checked the query string for static content i.e.CSS > and .JS, its working as expected. > > > Thanks, > > keval bhatt > >
Re: Review Request 61200: ATLAS-1996: Log4J Default Configuration Updated
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61200/ --- (Updated Sept. 7, 2017, 5:30 p.m.) Review request for atlas and Madhan Neethiraj. Changes --- Update includes: - Changed _DailyRollingFileAppender_ to _RollingFileAppender_. This honors max size. Bugs: ATLAS-1996 https://issues.apache.org/jira/browse/ATLAS-1996 Repository: atlas Description --- **Problem** Existing log4J configuration did not include max size parameter. This caused logs to grow making it hard for human log readers. Another contributing factor to log size growth is TitanDB's _IndexSerializer_. When this is used, the query conversion from graph query to indexed query is written as info log entry. With the recent changes, many more index queries are used, thus contributing to growth of log files. **Solution** Added maxFileSize parameter. Changed _com.thinkaurelius.titan_ log level to _warn_. **Additional Info** Defaults need to be set in _Ambari_ so that they reflect correctly as part of deployment. Diffs (updated) - addons/hive-bridge/src/main/resources/atlas-hive-import-log4j.xml c1184e5 distro/src/conf/atlas-log4j.xml a0ef6a9 typesystem/src/main/resources/atlas-log4j.xml c30403e Diff: https://reviews.apache.org/r/61200/diff/2/ Changes: https://reviews.apache.org/r/61200/diff/1-2/ Testing --- **Functional tests** Observed configuration changes in the logs generated. Thanks, Ashutosh Mestry
Review Request 62255: ATLAS-2129: Addressed Case Where Abrupt Shutdown of Atlas During Import Retains Entities Imported Before Shutdown
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62255/ --- Review request for atlas and Madhan Neethiraj. Bugs: ATLAS-2129 https://issues.apache.org/jira/browse/ATLAS-2129 Repository: atlas Description --- **Background** Existing implementation of _bulkImport_ was done in single transaction. This can potentially lead to case where abrupt shutdown of Atlas during Import will not retain entities already imported, this renders resume Import cability useless. **Implementation** - Removed _GraphTransaction_ from _bulkImport_. - New method _createOrUpdateForImport_ which is under trasaction scope. Diffs - repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java 1c168b4 Diff: https://reviews.apache.org/r/62255/diff/1/ Testing --- **Unit tests** No new tests added. **Functional tests** Import API tested. Thanks, Ashutosh Mestry
Review Request 62227: ATLAS-2120: Import API: Type Attribute Type Definition Matched
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62227/ --- Review request for atlas, Madhan Neethiraj and Sharmadha Sainath. Bugs: ATLAS-2120 https://issues.apache.org/jira/browse/ATLAS-2120 Repository: atlas Description --- This is a minor change, that performs additional check during import process. Import process now matches type definition for the types being imported. It fails and logs exception if types do not match. Diffs - docs/src/site/twiki/Import-API-Options.twiki cc09fc7 repository/src/main/java/org/apache/atlas/repository/impexp/TypeAttributeDifference.java 68bc583 repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 42f7011 repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java d80fab0 repository/src/test/resources/hdfs_path1.zip PRE-CREATION repository/src/test/resources/tag1.json PRE-CREATION Diff: https://reviews.apache.org/r/62227/diff/1/ Testing --- **Unit tests** - Additional failure unit tests added. **Functional tests** - Executed success and failure scenarios. Thanks, Ashutosh Mestry
Review Request 62251: ATLAS-2129: Loss of Imported Data In case of Abrupt Atlas Shutdown
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62251/ --- Review request for atlas and Madhan Neethiraj. Bugs: ATLAS-2129 https://issues.apache.org/jira/browse/ATLAS-2129 Repository: atlas Description --- **Background** Existing implementation adds a _GraphTransaction_ annotation on the _bulkImport_ method. This means that the transaction is open during entire import process. Also, existing implementation of _GraphTransaction_ does not handle nested transactions in the best way. **Implementation** This has an approach for dealing with nested transactions. Here are some highlights: - Maintains _ThreadLocal_ count of invocations. Each invocation increments the count, success in execution reduces the count. Commit is done when count reaches ZERO. - For rollback: Based on some research on transactions, there are 2 types viz. open and closed. * _Closed_: What we currently have, rollback is called when any transaction within the nesting encounters an error. * _Open_: Continues processing, and lets the current method deal with exceptions. The txn is closed when nesting reaches outer level. - I have added _GraphTransactionInterceptorTest_. This exercises capabilities of the _GraphTransasctionInterceptor_. Update to Import: _bulkImport_ method no longer is marked with _GraphTransaction_. This addresses the problem reported in the bug. Diffs - common/src/main/java/org/apache/atlas/annotation/GraphTransaction.java 7120166 common/src/main/java/org/apache/atlas/annotation/NestingType.java PRE-CREATION repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java c6a4bbe repository/src/test/java/org/apache/atlas/GraphTransactionInterceptorTest.java PRE-CREATION Diff: https://reviews.apache.org/r/62251/diff/1/ Testing --- **Unit tests** - New unit tests added to address problem with nested transactions, to test the new _GraphTransactionInterceptor_ functionality. - Executed existing unit tests. **Functional tests** Import functional tests. Thanks, Ashutosh Mestry
Re: Review Request 62577: [ATLAS-2176] NegativeSSLAndKerberosTest failing due to error in initialization
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62577/#review186288 --- Ship it! Ship It! - Ashutosh Mestry On Sept. 26, 2017, 3:06 p.m., Gergo Barna wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62577/ > --- > > (Updated Sept. 26, 2017, 3:06 p.m.) > > > Review request for atlas, Madhan Neethiraj and Nixon Rodrigues. > > > Repository: atlas > > > Description > --- > > ATLAS-2176 defending against NPE during initialization > (AtlasTypeDefStoreInitializer.loadBootstrapTypeDefs) > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java > a65ec3a9ead1ce2fa91232bc56e601b31ccf750e > > > Diff: https://reviews.apache.org/r/62577/diff/1/ > > > Testing > --- > > > Thanks, > > Gergo Barna > >
Re: Review Request 62681: ATLAS-2075: notification enhancement to handle large messages, using compression and multi-part messages
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62681/#review186617 --- I would have liked more unit tests, since this is core functionality. notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java Line 133 (original), 152 (patched) <https://reviews.apache.org/r/62681/#comment263312> I would suggest splitting this logic into methods, preferably classes. There is lot of logic that will help if there are tests. notification/src/main/java/org/apache/atlas/notification/AlasNotificationMessageDeserializer.java Lines 78 (patched) <https://reviews.apache.org/r/62681/#comment263313> Recommend splitting this in methods or classes. This logic can thus be testtable. notification/src/main/java/org/apache/atlas/notification/AtlasNotificationBaseMessage.java Lines 36 (patched) <https://reviews.apache.org/r/62681/#comment263315> This class seems to be doing 3 different things. Will help to model it that way. notification/src/main/java/org/apache/atlas/notification/AtlasNotificationBaseMessage.java Lines 130 (patched) <https://reviews.apache.org/r/62681/#comment263314> Consider encapsulating these methods into separate utility class. - Ashutosh Mestry On Sept. 28, 2017, 11:08 p.m., Madhan Neethiraj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62681/ > --- > > (Updated Sept. 28, 2017, 11:08 p.m.) > > > Review request for atlas, Apoorv Naik, Ashutosh Mestry, Abhay Kulkarni, > Sarath Subramanian, and Suma Shivaprasad. > > > Bugs: ATLAS-2075 > https://issues.apache.org/jira/browse/ATLAS-2075 > > > Repository: atlas > > > Description > --- > > Update notification module to compress and split large messages (> 1mb) that > can't be sent via Kafka. When such a message is sent, the message is first > compressed via GZip. If compressed message is within the size limit, the > message is sent via Kafka. If the compressed size is larger, then the message > is split into multiple small messages and sent via Kafka. > > Notification consumer is updated to merge and uncompress messages before the > message is processed further. > > > Diffs > - > > common/src/main/java/org/apache/atlas/AtlasConfiguration.java 9a9bb76a > notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java > d3b4e49e > notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java > 38889ef8 > > notification/src/main/java/org/apache/atlas/notification/AbstractMessageDeserializer.java > ec99372d > > notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java > cb44fc68 > > notification/src/main/java/org/apache/atlas/notification/AlasNotificationMessageDeserializer.java > PRE-CREATION > > notification/src/main/java/org/apache/atlas/notification/AtlasNotificationBaseMessage.java > PRE-CREATION > > notification/src/main/java/org/apache/atlas/notification/AtlasNotificationStringMessage.java > PRE-CREATION > > notification/src/main/java/org/apache/atlas/notification/MessageVersion.java > 6ef407ac > > notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java > 956c85e0 > > notification/src/main/java/org/apache/atlas/notification/VersionedMessage.java > 1929eb46 > > notification/src/main/java/org/apache/atlas/notification/VersionedMessageDeserializer.java > cc2099e7 > notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java > 9b712f44 > > notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java > b7474a0e > > notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java > 3b2a093b > > notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java > 61107a9f > > notification/src/test/java/org/apache/atlas/notification/VersionedMessageTest.java > 587b7ebd > > notification/src/test/java/org/apache/atlas/notification/entity/EntityMessageDeserializerTest.java > be324277 > > notification/src/test/java/org/apache/atlas/notification/hook/HookMessageDeserializerTest.java > 3724fd5f > > > Diff: https://reviews.apache.org/r/62681/diff/1/ > > > Testing > --- > > - verified that large messages are compressed and split by Atlas hook, before > sending to Kafka > - verified that Atlas server processes compressed, multi-part messages > correctly > > > Thanks, > > Madhan Neethiraj > >
Review Request 62691: ATLAS-2089: Upgrade Jetty to Newer Version
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62691/ --- Review request for atlas, Apoorv Naik, Madhan Neethiraj, Nixon Rodrigues, and Vishal Suvagia. Bugs: ATLAS-2089 https://issues.apache.org/jira/browse/ATLAS-2089 Repository: atlas Description --- **Background** Current Jetty version has known bug that causes timeout when serving static content. Please refer to [Bug 472621|https://bugs.eclipse.org/bugs/show_bug.cgi?id=472621] **Implementation** Jetty version has been upgraded to v9.4.x. Jetty-JSP needed an upgrade v9.2.x. Diffs - pom.xml f793321c Diff: https://reviews.apache.org/r/62691/diff/1/ Testing --- **Testing** - Functional testing. - Verification of security features viz. SSL, non-SSL, Knox integration. Thanks, Ashutosh Mestry
Re: Review Request 62681: ATLAS-2075: notification enhancement to handle large messages, using compression and multi-part messages
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62681/#review186755 --- Fix it, then Ship it! Ship It! notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java Lines 204 (patched) <https://reviews.apache.org/r/62681/#comment263565> Consider using Math.Ceil. - Ashutosh Mestry On Sept. 29, 2017, 10:32 p.m., Madhan Neethiraj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62681/ > --- > > (Updated Sept. 29, 2017, 10:32 p.m.) > > > Review request for atlas, Apoorv Naik, Ashutosh Mestry, Abhay Kulkarni, > Sarath Subramanian, and Suma Shivaprasad. > > > Bugs: ATLAS-2075 > https://issues.apache.org/jira/browse/ATLAS-2075 > > > Repository: atlas > > > Description > --- > > Update notification module to compress and split large messages (> 1mb) that > can't be sent via Kafka. When such a message is sent, the message is first > compressed via GZip. If compressed message is within the size limit, the > message is sent via Kafka. If the compressed size is larger, then the message > is split into multiple small messages and sent via Kafka. > > Notification consumer is updated to merge and uncompress messages before the > message is processed further. > > > Diffs > - > > common/src/main/java/org/apache/atlas/AtlasConfiguration.java 9a9bb76a > notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java > d3b4e49e > notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java > 38889ef8 > > notification/src/main/java/org/apache/atlas/notification/AbstractMessageDeserializer.java > ec99372d > > notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java > cb44fc68 > > notification/src/main/java/org/apache/atlas/notification/AtlasNotificationBaseMessage.java > PRE-CREATION > > notification/src/main/java/org/apache/atlas/notification/AtlasNotificationMessageDeserializer.java > PRE-CREATION > > notification/src/main/java/org/apache/atlas/notification/AtlasNotificationStringMessage.java > PRE-CREATION > > notification/src/main/java/org/apache/atlas/notification/MessageVersion.java > 6ef407ac > > notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java > 956c85e0 > > notification/src/main/java/org/apache/atlas/notification/VersionedMessage.java > 1929eb46 > > notification/src/main/java/org/apache/atlas/notification/VersionedMessageDeserializer.java > cc2099e7 > notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java > 9b712f44 > > notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java > b7474a0e > > notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java > 3b2a093b > > notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java > 61107a9f > > notification/src/test/java/org/apache/atlas/notification/VersionedMessageTest.java > 587b7ebd > > notification/src/test/java/org/apache/atlas/notification/entity/EntityMessageDeserializerTest.java > be324277 > > notification/src/test/java/org/apache/atlas/notification/hook/HookMessageDeserializerTest.java > 3724fd5f > > > Diff: https://reviews.apache.org/r/62681/diff/4/ > > > Testing > --- > > - verified that large messages are compressed and split by Atlas hook, before > sending to Kafka > - verified that Atlas server processes compressed, multi-part messages > correctly > > > Thanks, > > Madhan Neethiraj > >
Re: Review Request 61807: ATLAS-2075: Kafka Messages: Split & Combine Implementation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61807/ --- (Updated Sept. 25, 2017, 6:10 p.m.) Review request for atlas, Madhan Neethiraj and Nixon Rodrigues. Changes --- Updates include: - Moved changes related to JSON compression to _JsonUtils_. - Additional refactoring for simplicity. Bugs: ATLAS-2075 https://issues.apache.org/jira/browse/ATLAS-2075 Repository: atlas Description (updated) --- **Approach** - Message size is checked. It is split into parts if it exceeds the size threshold. - New message type _Split_ derived from _VersionedMessage_ holds information of the individual parts of the message. - _MessageSplitter_ splits message into smaller packets and encloses them in a message of type _Split_. - _MessageCombiner_ combines array of split messages. - _VersionedMessageDeserializer_ is modified to support the new message type. **Additional configuration item** - _atlas.notification.message.size.threshold_ Needs to be added to Hive's atlas configuration as well as _Atlas_'s own _atlas-application.properties_. - Default is set to a number closer to 1MB. **For testing** - Set threshold to a low number. - Update _atlas-application.properties_: - Add _atlas.notification.message.size.threshold_ to 1024. Diffs (updated) - notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java d3b4e49e notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 38889ef8 notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java cb44fc68 notification/src/main/java/org/apache/atlas/notification/MessageCombiner.java PRE-CREATION notification/src/main/java/org/apache/atlas/notification/MessageTransformer.java PRE-CREATION notification/src/main/java/org/apache/atlas/notification/MessageVersion.java 6ef407ac notification/src/main/java/org/apache/atlas/notification/VersionedMessage.java 1929eb46 notification/src/main/java/org/apache/atlas/notification/VersionedMessageDeserializer.java cc2099e7 notification/src/main/java/org/apache/atlas/notification/VersionedMessageHeader.java PRE-CREATION notification/src/main/java/org/apache/atlas/notification/messages/Compressed.java PRE-CREATION notification/src/main/java/org/apache/atlas/notification/messages/Split.java PRE-CREATION notification/src/main/java/org/apache/atlas/utils/JsonUtils.java PRE-CREATION notification/src/main/java/org/apache/atlas/utils/StringSplitterCombiner.java PRE-CREATION notification/src/test/java/org/apache/atlas/hook/StringSplitCombineTest.java PRE-CREATION notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java 3b2a093b notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java 61107a9f notification/src/test/java/org/apache/atlas/notification/MessageCombinerTest.java PRE-CREATION notification/src/test/java/org/apache/atlas/notification/MessageSplitterTest.java PRE-CREATION notification/src/test/java/org/apache/atlas/notification/SplitCombinerUtils.java PRE-CREATION notification/src/test/java/org/apache/atlas/notification/entity/EntityMessageDeserializerTest.java be324277 notification/src/test/java/org/apache/atlas/notification/hook/HookMessageDeserializerTest.java 3724fd5f notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationTest.java dd3257e0 notification/src/test/java/org/apache/atlas/utils/TestJsonUtils.java PRE-CREATION webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 2669c739 webapp/src/test/java/org/apache/atlas/util/JsonBenchmarkTest.java PRE-CREATION Diff: https://reviews.apache.org/r/61807/diff/3/ Changes: https://reviews.apache.org/r/61807/diff/2-3/ Testing --- **Unit tests** - Added unit tests for new functionality. - Refactored few of existing unit tests. - Added tests to verify backward compatibility. **Functional tests** - Verified regular flow. - Verified cases: - Case where server is shutdown before all the parts of a message are committed. Thanks, Ashutosh Mestry
Review Request 61807: ATLAS-2075: Kafka Messages: Split & Combine Implementation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61807/ --- Review request for atlas, Madhan Neethiraj and Nixon Rodrigues. Bugs: ATLAS-2075 https://issues.apache.org/jira/browse/ATLAS-2075 Repository: atlas Description --- **Approach** - Message size is checked. It is split into parts if it exceeds the size threshold. - New message type _Split_ derived from _VersionedMessage_ holds information of the individual parts of the message. - _MessageSplitter_ splits message into smaller packets and encloses them in a message of type _Split_. - _MessageCombiner_ combines array of split messages. - _VersionedMessageDeserializer_ is modified to support the new message type. **Additional configuration item** - _atlas.notification.message.size.threshold_ Needs to be added to Hive's atlas configuration as well as _Atlas_'s own _atlas-application.properties_. - Default is set to a number closer to 1MB. Diffs - intg/src/main/java/org/apache/atlas/type/AtlasType.java c99eb7f intg/src/test/java/org/apache/atlas/TestAtlasTypeJSONSerialize.java PRE-CREATION notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java d3b4e49 notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 38889ef notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java cb44fc6 notification/src/main/java/org/apache/atlas/notification/MessageCombiner.java PRE-CREATION notification/src/main/java/org/apache/atlas/notification/MessageSplitter.java PRE-CREATION notification/src/main/java/org/apache/atlas/notification/MessageVersion.java 6ef407a notification/src/main/java/org/apache/atlas/notification/VersionedMessage.java 1929eb4 notification/src/main/java/org/apache/atlas/notification/VersionedMessageDeserializer.java cc2099e notification/src/main/java/org/apache/atlas/notification/messages/CompressedVersioned.java PRE-CREATION notification/src/main/java/org/apache/atlas/notification/messages/Split.java PRE-CREATION notification/src/main/java/org/apache/atlas/utils/StringSplitterCombiner.java PRE-CREATION notification/src/test/java/org/apache/atlas/hook/StringSplitCombineTest.java PRE-CREATION notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java 3b2a093 notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java 61107a9 notification/src/test/java/org/apache/atlas/notification/MessageCombinerTest.java PRE-CREATION notification/src/test/java/org/apache/atlas/notification/MessageSplitterTest.java PRE-CREATION notification/src/test/java/org/apache/atlas/notification/SplitCombinerUtils.java PRE-CREATION notification/src/test/java/org/apache/atlas/notification/entity/EntityMessageDeserializerTest.java be32427 notification/src/test/java/org/apache/atlas/notification/hook/HookMessageDeserializerTest.java 3724fd5 notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationTest.java dd3257e webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java ef64c3b webapp/src/test/java/org/apache/atlas/web/filters/AtlasAuthenticationSimpleFilterIT.java 25414fd Diff: https://reviews.apache.org/r/61807/diff/1/ Testing --- **Unit tests** - Added unit tests for new functionality. - Refactored few of existing unit tests. - Added tests to verify backward compatibility. **Functional tests** - Verified regular flow. - Verified cases: - Case where server is shutdown before all the parts of a message are committed. Thanks, Ashutosh Mestry
Re: Review Request 61940: ATLAS-2090 : UI - Cache busting for static content (css, js)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61940/#review183933 --- This is an elegant solution. It will ensure that JS libraries & templates are loaded from the server for each build that is deployed. I verified: - Firefox - Chrome Played around with the UI while noting the URLs requested from server. Good thing is that the _bust_ value remains constant across browsers (re-stating the point above). - Ashutosh Mestry On Aug. 28, 2017, 1:22 p.m., keval bhatt wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/61940/ > --- > > (Updated Aug. 28, 2017, 1:22 p.m.) > > > Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, > Nixon Rodrigues, and Sarath Subramanian. > > > Bugs: ATLAS-2090 > https://issues.apache.org/jira/browse/ATLAS-2090 > > > Repository: atlas > > > Description > --- > > Grunt task is updated to handle Cache busting issue, so now after every > successful build grunt task will add new time as the query string for static > content i.e .CSS and .JS > > > Diffs > - > > dashboardv2/gruntfile.js 900ee40 > dashboardv2/package.json 6068c79 > dashboardv2/public/index.html f572049 > dashboardv2/public/index.html.tpl PRE-CREATION > dashboardv2/public/js/main.js 63e9abc > > > Diff: https://reviews.apache.org/r/61940/diff/1/ > > > Testing > --- > > Build the atlas code and checked the query string for static content i.e.CSS > and .JS, its working as expected. > > > Thanks, > > keval bhatt > >
Re: Review Request 61940: ATLAS-2090 : UI - Cache busting for static content (css, js)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61940/#review183936 --- Question: _require.js_ does not have bust parameter. Is this intentional? If yes, what is the rationale? http://localhost:21000/js/libs/requirejs/require.js - Ashutosh Mestry On Aug. 28, 2017, 1:22 p.m., keval bhatt wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/61940/ > --- > > (Updated Aug. 28, 2017, 1:22 p.m.) > > > Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, > Nixon Rodrigues, and Sarath Subramanian. > > > Bugs: ATLAS-2090 > https://issues.apache.org/jira/browse/ATLAS-2090 > > > Repository: atlas > > > Description > --- > > Grunt task is updated to handle Cache busting issue, so now after every > successful build grunt task will add new time as the query string for static > content i.e .CSS and .JS > > > Diffs > - > > dashboardv2/gruntfile.js 900ee40 > dashboardv2/package.json 6068c79 > dashboardv2/public/index.html f572049 > dashboardv2/public/index.html.tpl PRE-CREATION > dashboardv2/public/js/main.js 63e9abc > > > Diff: https://reviews.apache.org/r/61940/diff/1/ > > > Testing > --- > > Build the atlas code and checked the query string for static content i.e.CSS > and .JS, its working as expected. > > > Thanks, > > keval bhatt > >
Re: [VOTE] Release Apache Atlas 0.8.1 - release candidate 1
+1 for the release. Sarath: Thanks for the efforts on the release! I have verified the build and artifacts. I did functional testing on search and Export-Import APIs. ~ ashutosh Ashutosh Mestry (ames...@apache.org) ... I get up each morning determined both to change the world and have one hell of a good time. Sometimes this makes planning my day difficult. – E.B. White, Essayist On 8/24/17, 6:26 AM, "Keval Bhatt" <kevalbhat...@gmail.com> wrote: Hi, Sarath thanks for the release work. I verified the Atlas build and Atlas UI it's working as expected. https://dist.apache.org/repos/dist/dev/atlas/0.8.1-rc1/ apache-atlas-0.8.1-sources.tar.gz +1 for the release. Thanks, Keval Bhatt On Thu, Aug 24, 2017 at 6:42 PM, Nixon Rodrigues < nixon.rodrig...@freestoneinfotech.com> wrote: > Thanks Sarath for putting Atlas 0.8.1 for release. > > I verified and tested following with the source provided > https://dist.apache.org/repos/dist/dev/atlas/0.8.1-rc1/apache-atlas-0.8.1- > sources.tar.gz > > >- Verified the signature and md5 hash keys >- Build is going fine. >- Unit test cases are passing >- Rat check is good >- Release tag is proper on git repo. > > >- Atlas UI looks good. >- Hive Entities from hook are reflecting properly in Atlas UI. >- Tags/Entities are created and rendered properly on UI. >- Advance / Basic search with attribute is working fine.(Nice feature) > ># There was one IT failure in HiveHookIT due timeout issue > > *+1* for the rc1 for Atlas 0.8.1 release. > > > Regards, > Nixon > > > > On Thu, Aug 24, 2017 at 5:18 AM, Sarath Subramanian <sar...@apache.org> > wrote: > > > Atlas team, > > > > I have created a build for Apache Atlas 0.8.1 - release candidate #1. > > Thanks to everyone who have contributed to this release and tested > > this. > > > > > > Changes since last release-candidate: > > > >- ATLAS-2038: Minor refactoring to use > > org.apache.commons.codec.binary.Base64. > >- ATLAS-2073: UI - removing tag or type from drop-down will not > > exclude respective filters > >- ATLAS-2074: AtlasType.resolveReferences() method made > package-private > >- ATLAS-2078: Type update using v1 API doesn't route to > > createUpdateTypesDef in v2 API > >- ATLAS-2079: Fix coverity scan issue and IT failures introduced by > > ATLAS-2062 > >- ATLAS-2080: fix - tag filters should not be added in the results > > attributes list > >- ATLAS-2082: basic-search fails when in-memory filters all results > > from index-query > > > > Git tag for the release: > > > > https://github.com/apache/atlas/tree/release-0.8.1-rc1 > > > > > > Sources for the release: > > > > https://dist.apache.org/repos/dist/dev/atlas/0.8.1-rc1/apach > > e-atlas-0.8.1-sources.tar.gz > > > > Source release verification: > > PGP Signature : > > https://dist.apache.org/repos/dist/dev/atlas/0.8.1-rc1/apach > > e-atlas-0.8.1-sources.tar.gz.asc > > MD5 Checksum : > > https://dist.apache.org/repos/dist/dev/atlas/0.8.1-rc1/apach > > e-atlas-0.8.1-sources.tar.gz.md5 > > > > SHA512 Checksum : > > https://dist.apache.org/repos/dist/dev/atlas/0.8.1-rc1/apach > > e-atlas-0.8.1-sources.tar.gz.sha512 > > > > Keys to verify the signature of the release artifacts are available at: > > > > https://dist.apache.org/repos/dist/dev/atlas/KEYS > > > > > > List of improvements and issues addressed in this release: > > > > https://issues.apache.org/jira/browse/ATLAS-2066?jql=project > > %20%3D%20ATLAS%20AND%20status%20%3D%20Resolved%20AND%20resol > > ution%20%3D%20Fixed%20AND%20fixVersion%20%3D%200.8.1- > > incubating%20ORDER%20BY%20key%20DESC > > > > > > Note that this is a source only release and we are voting on the > > source *release-0.8.1-rc1*. Please download, test, and try it out. > > > > Voting will be open for at least *72 hours* (until the required number > > of votes are obtained) till 26th August, 2017 04:45 PM PT. > > > > [ ] +1 approve > > [ ] +0 no opinion > > [ ] -1 disapprove (and reason why) > > > > +1 from my side for the release. > > > > > > Thanks, > > > > Sarath Subramanian > > >
Re: Review Request 61552: ATLAS-2038 : Unit Test cases for AtlasAuthentication and Authorization Filter
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61552/#review183352 --- Ship it! Ship It! - Ashutosh Mestry On Aug. 11, 2017, 4:25 p.m., Nixon Rodrigues wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/61552/ > --- > > (Updated Aug. 11, 2017, 4:25 p.m.) > > > Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and > Sarath Subramanian. > > > Bugs: ATLAS-2038 > https://issues.apache.org/jira/browse/ATLAS-2038 > > > Repository: atlas > > > Description > --- > > This patch enable the Unittest from this testclass and includes unit test > cases for testing Authentication and Authorization Filters which can fail if > any regression is introduced in this Filters and authentication/authorization > process. ( eg bugs ATLAS-2009). > > > Diffs > - > > > webapp/src/test/java/org/apache/atlas/web/filters/AtlasAuthenticationSimpleFilterIT.java > PRE-CREATION > > webapp/src/test/java/org/apache/atlas/web/filters/AtlasAuthenticationSimpleFilterTest.java > 389eefe > > > Diff: https://reviews.apache.org/r/61552/diff/2/ > > > Testing > --- > > Excecuted unit and Integration test from mvn clean install > > > Thanks, > > Nixon Rodrigues > >
Re: Review Request 61788: ATLAS-2074: AtlasType.resolveReferences() method made package-private
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61788/#review183357 --- Ship it! Ship It! - Ashutosh Mestry On Aug. 21, 2017, 6:05 p.m., Madhan Neethiraj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/61788/ > --- > > (Updated Aug. 21, 2017, 6:05 p.m.) > > > Review request for atlas. > > > Bugs: ATLAS-2074 > https://issues.apache.org/jira/browse/ATLAS-2074 > > > Repository: atlas > > > Description > --- > > Currently AtlasType.resolveReferences() method is marked as public. > Implementations in sub-types likely to update the state of the type object; > hence this method is only called during initialization/updates to > type-registry. To prevent potential misuse of this method, which can result > in errors, resolveReference() method is made package-private. > > > Diffs > - > > intg/src/main/java/org/apache/atlas/type/AtlasArrayType.java de3394f1 > intg/src/main/java/org/apache/atlas/type/AtlasClassificationType.java > 56c3ed38 > intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 9dad95df > intg/src/main/java/org/apache/atlas/type/AtlasEnumType.java 1cd27b39 > intg/src/main/java/org/apache/atlas/type/AtlasMapType.java 90c898a3 > intg/src/main/java/org/apache/atlas/type/AtlasRelationshipType.java > 934dffce > intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 277d0fa2 > intg/src/main/java/org/apache/atlas/type/AtlasType.java c99eb7f1 > > > Diff: https://reviews.apache.org/r/61788/diff/1/ > > > Testing > --- > > Verified that Atlas builds successfully after this change in method accessor. > > > Thanks, > > Madhan Neethiraj > >
Re: Review Request 62009: ATLAS-2101: Update Implementation to Eliminate Use of Stopwatch
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62009/ --- (Updated Sept. 1, 2017, 1:47 a.m.) Review request for atlas and Apoorv Naik. Changes --- Updates include: - Added license header. Bugs: ATLAS-2101 https://issues.apache.org/jira/browse/ATLAS-2101 Repository: atlas Description --- **Details** - Added _StandardIDPool_ to _Titan0_ Shaded JAR. - Replace use of _Stopwatch_ with custom logic. Diffs (updated) - graphdb/titan0/src/main/java/com/thinkaurelius/titan/graphdb/database/idassigner/StandardIDPool.java PRE-CREATION Diff: https://reviews.apache.org/r/62009/diff/2/ Changes: https://reviews.apache.org/r/62009/diff/1-2/ Testing --- Thanks, Ashutosh Mestry
Re: Review Request 62039: ATLAS-2100: UserProfile & SavedSearch Implementation
"fullName" : "Admin Admin"
"savedSearchParameters": [
]
}
```
_AtlasSavedSearch_
```javascript
{
"name": "my_search1",
"ownerUserName": "admin",
"searchParameters":
}
```
Diffs (updated)
-
addons/models/0010-base_model.json 7f64d85
addons/models/0011-user-profile_model.json PRE-CREATION
intg/src/main/java/org/apache/atlas/AtlasErrorCode.java aa8e3f4
intg/src/main/java/org/apache/atlas/model/AtlasBaseModelObject.java
PRE-CREATION
intg/src/main/java/org/apache/atlas/model/SearchFilter.java 7dccf5e
intg/src/main/java/org/apache/atlas/model/profile/AtlasUserProfile.java
PRE-CREATION
intg/src/main/java/org/apache/atlas/model/profile/AtlasUserSavedSearch.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java
8196a67
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
1e68835
repository/src/main/java/org/apache/atlas/repository/orm/DataAccess.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/orm/DataAccessHelper.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/userprofile/UserProfileService.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/util/FilterUtil.java
54d6b40
repository/src/test/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStoreTest.java
728d418
repository/src/test/java/org/apache/atlas/repository/userprofile/UserProfileServiceTest.java
PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java 52258e3
webapp/src/main/java/org/apache/atlas/web/rest/TypesREST.java c32f36e
webapp/src/main/java/org/apache/atlas/web/util/Servlets.java 4a92763
Diff: https://reviews.apache.org/r/62039/diff/2/
Changes: https://reviews.apache.org/r/62039/diff/1-2/
Testing
---
**Deployment**
- Copy _0010-base_model.json_ & _0011-user-profile_model.json_ to server
location: _/usr/hdp/current/atlas-server/models_
- Deploy _atlas.war_
**Unit tests**
Added unit tests for:
- _UserProfileService_.
**Functional testing**
- Verification using CURL calls.
Thanks,
Ashutosh Mestry
Review Request 62039: ATLAS-2100: UserProfile & SavedSearch Implementation
SavedSearch_
```javascript
{
"name": "my_search1",
"ownerUserName": "admin",
"searchParameters":
}
```
Diffs
-
addons/models/0010-base_model.json 7f64d85
addons/models/0011-user-profile_model.json PRE-CREATION
graphdb/titan0/src/main/java/com/thinkaurelius/titan/graphdb/database/idassigner/StandardIDPool.java
PRE-CREATION
intg/src/main/java/org/apache/atlas/AtlasErrorCode.java aa8e3f4
intg/src/main/java/org/apache/atlas/model/AtlasBaseModelObject.java
PRE-CREATION
intg/src/main/java/org/apache/atlas/model/SearchFilter.java 7dccf5e
intg/src/main/java/org/apache/atlas/model/profile/AtlasUserProfile.java
PRE-CREATION
intg/src/main/java/org/apache/atlas/model/profile/AtlasUserSavedSearch.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java
8196a67
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
1e68835
repository/src/main/java/org/apache/atlas/repository/orm/DataAccess.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/orm/DataAccessHelper.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/userprofile/UserProfileService.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/util/FilterUtil.java
54d6b40
repository/src/test/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStoreTest.java
728d418
repository/src/test/java/org/apache/atlas/repository/userprofile/UserProfileServiceTest.java
PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java 52258e3
webapp/src/main/java/org/apache/atlas/web/rest/TypesREST.java c32f36e
webapp/src/main/java/org/apache/atlas/web/util/Servlets.java 4a92763
Diff: https://reviews.apache.org/r/62039/diff/1/
Testing
---
**Deployment**
- Copy _0010-base_model.json_ & _0011-user-profile_model.json_ to server
location: _/usr/hdp/current/atlas-server/models_
- Deploy _atlas.war_
**Unit tests**
Added unit tests for:
- _UserProfileService_.
**Functional testing**
- Verification using CURL calls.
Thanks,
Ashutosh Mestry
Review Request 62009: ATLAS-2101: Update Implementation to Eliminate Use of Stopwatch
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62009/ --- Review request for atlas and Apoorv Naik. Bugs: ATLAS-2101 https://issues.apache.org/jira/browse/ATLAS-2101 Repository: atlas Description --- **Details** - Added _StandardIDPool_ to _Titan0_ Shaded JAR. - Replace use of _Stopwatch_ with custom logic. Diffs - graphdb/titan0/src/main/java/com/thinkaurelius/titan/graphdb/database/idassigner/StandardIDPool.java PRE-CREATION Diff: https://reviews.apache.org/r/62009/diff/1/ Testing --- Thanks, Ashutosh Mestry
Re: Review Request 61665: ATLAS-2047: Exception Thrown by Kafka Consumer Ends up Filling Logs Due to Incorrect Handling
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61665/
---
(Updated Aug. 31, 2017, 4:23 a.m.)
Review request for atlas, Madhan Neethiraj and Nixon Rodrigues.
Changes
---
Updates include:
- Addressed review comments.
- Updated unit test.
Bugs: ATLAS-2047
https://issues.apache.org/jira/browse/ATLAS-2047
Repository: atlas
Description
---
Please refer to [ATLAS-2047](https://issues.apache.org/jira/browse/ATLAS-2047)
for background and analysis.
**Background**
The _IllegalStateException_ is thrown by _KafkaConsumer.aquire_. This method is
called at the beginning of almost every method in this class. The method checks
if the consumer is closed, if it is then it throws IllegalStateException.
Scenario may come about in this way:
- Shutdown has been initiated. Close on consumer is called.
- However, the consumer thread is just about to enter another poll cycle.
- Thus acquire sees that consumer is closed and throws the exception (2nd
bullet above).
Please take a look at this scala code. This is _ShutdownableThread_. The thread
does the job of handling all exceptions. Upon exception, it manages the
_shutdownLatch_ (from yesterday’s bug fix) and gets out of the _isRunning_ loop.
```scala
override def run(): Unit = {
info("Starting ")
try{
while(isRunning.get()){
doWork()
}
} catch{
case e: Throwable =>
if(isRunning.get())
error("Error due to ", e)
}
shutdownLatch.countDown()
info("Stopped ")
}
```
**Implementation**
Special treatment is given to _IllegalStateException_ by implementing pause &
retry logic:
- Modified _LOG_ to _debug_. That way logs are not filled during retry.
- _HookConsumer_ is more resilient. It handles exceptions resulting from
_Kafka_ and entity APIs.
Diffs (updated)
-
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
ef64c3b
webapp/src/test/java/org/apache/atlas/notification/AdaptiveWaiterTest.java
PRE-CREATION
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java
a6f58e8
Diff: https://reviews.apache.org/r/61665/diff/5/
Changes: https://reviews.apache.org/r/61665/diff/4-5/
Testing
---
**Unit tests**
Updated unit tests to reproduce the scenarios and verify the fix.
**Functional tests**
Verified regular notification scenarios.
Thanks,
Ashutosh Mestry
Review Request 62957: ATLAS-2195: Addressing Txn Leak During Startup
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62957/ --- Review request for atlas, Madhan Neethiraj and Sarath Subramanian. Bugs: ATLAS-2195 https://issues.apache.org/jira/browse/ATLAS-2195 Repository: atlas Description --- **Background** If entities and types are created simultaneously it was observed that some stale transactions are created. Titan runs an eviction thread every minute to evict these out of transaction cache. That is when the message mentioned in the bug appears. **Analysis** During startup when built-in types are added to the database, the process did not use _GraphTransactionInterceptor_ approach. This caused stale transactions to appear, which grew over time. **Instrumentation** Used _Titan's_ _StandardTitanGraph_ for adding instrumentation code. This is central point for transaction operations. Tracing the creation and co-relating them with destruction helped in narrowing down the area of focus. Diffs - repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 6e9ef685 Diff: https://reviews.apache.org/r/62957/diff/1/ Testing --- **Unit tests** Existing. **Other** Executed the attached bash scripts to create numerous types and entities simultaneously. Use this command: ``` bash create-type.sh & bash create-entity.sh & ``` File Attachments create type https://reviews.apache.org/media/uploaded/files/2017/10/12/755455cc-c08c-4885-9d07-5b5d9e3867e2__create-type.sh create entity https://reviews.apache.org/media/uploaded/files/2017/10/12/b20ddb13-7300-493d-8f43-5fcaf884c055__create-entity.sh Thanks, Ashutosh Mestry
Re: Review Request 63136: ATLAS-2218: Separate UT and IT in webapp module
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63136/#review189000 --- Ship it! - Ashutosh Mestry On Oct. 19, 2017, 4:59 a.m., Apoorv Naik wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/63136/ > --- > > (Updated Oct. 19, 2017, 4:59 a.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath > Subramanian. > > > Bugs: ATLAS-2218 > https://issues.apache.org/jira/browse/ATLAS-2218 > > > Repository: atlas > > > Description > --- > > Currently UTs and ITs run under a single mvn goal "pre-integration-test", > this change allows certain tests to run as unit-tests and the remainder run > as integration-tests > > > Diffs > - > > webapp/pom.xml 7a868086 > > > Diff: https://reviews.apache.org/r/63136/diff/1/ > > > Testing > --- > > mvn clean package run as expected > > mvn test -DskipITs don't run any Secure server tests or SSL tests (from > webapp module) > mvn verify -pl webapp -DskipITs run all unit + pre-integration tests > > mvn test runs only UTs and no secure server, ssl or kerberos tests > mvn verify -pl webapp runs every single test under webapp > > > Thanks, > > Apoorv Naik > >
Re: Review Request 63136: ATLAS-2218: Separate UT and IT in webapp module
> On Oct. 23, 2017, 11:08 p.m., Ashutosh Mestry wrote: > > I was able to execute ITs and UTs indepdenently. - Ashutosh --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63136/#review189000 --- On Oct. 19, 2017, 4:59 a.m., Apoorv Naik wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/63136/ > --- > > (Updated Oct. 19, 2017, 4:59 a.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath > Subramanian. > > > Bugs: ATLAS-2218 > https://issues.apache.org/jira/browse/ATLAS-2218 > > > Repository: atlas > > > Description > --- > > Currently UTs and ITs run under a single mvn goal "pre-integration-test", > this change allows certain tests to run as unit-tests and the remainder run > as integration-tests > > > Diffs > - > > webapp/pom.xml 7a868086 > > > Diff: https://reviews.apache.org/r/63136/diff/1/ > > > Testing > --- > > mvn clean package run as expected > > mvn test -DskipITs don't run any Secure server tests or SSL tests (from > webapp module) > mvn verify -pl webapp -DskipITs run all unit + pre-integration tests > > mvn test runs only UTs and no secure server, ssl or kerberos tests > mvn verify -pl webapp runs every single test under webapp > > > Thanks, > > Apoorv Naik > >
Re: Review Request 63136: ATLAS-2218: Separate UT and IT in webapp module
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63136/#review188999 --- I was able to execute ITs and UTs indepdenently. - Ashutosh Mestry On Oct. 19, 2017, 4:59 a.m., Apoorv Naik wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/63136/ > --- > > (Updated Oct. 19, 2017, 4:59 a.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath > Subramanian. > > > Bugs: ATLAS-2218 > https://issues.apache.org/jira/browse/ATLAS-2218 > > > Repository: atlas > > > Description > --- > > Currently UTs and ITs run under a single mvn goal "pre-integration-test", > this change allows certain tests to run as unit-tests and the remainder run > as integration-tests > > > Diffs > - > > webapp/pom.xml 7a868086 > > > Diff: https://reviews.apache.org/r/63136/diff/1/ > > > Testing > --- > > mvn clean package run as expected > > mvn test -DskipITs don't run any Secure server tests or SSL tests (from > webapp module) > mvn verify -pl webapp -DskipITs run all unit + pre-integration tests > > mvn test runs only UTs and no secure server, ssl or kerberos tests > mvn verify -pl webapp runs every single test under webapp > > > Thanks, > > Apoorv Naik > >
Re: Review Request 63152: ATLAS-2218: TestMetadata.py fix
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63152/#review188996 --- Ship it! Ship It! - Ashutosh Mestry On Oct. 19, 2017, 4:32 p.m., Apoorv Naik wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/63152/ > --- > > (Updated Oct. 19, 2017, 4:32 p.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath > Subramanian. > > > Bugs: ATLAS-2218 > https://issues.apache.org/jira/browse/ATLAS-2218 > > > Repository: atlas > > > Description > --- > > Recent changes to the startup script introduced a polling mechanism which > waits for atlas server's response (heartbeat check). As a side effect the > python unit tests took longer to execute because of the polling logic. > > This change introduces the mocks for getConfig, getConfigWithDefault and > wait_for_startup to reduce the test time from 11 mins to ~5 secs > > > Diffs > - > > distro/src/test/python/scripts/TestMetadata.py 1fd9a8cf > > > Diff: https://reviews.apache.org/r/63152/diff/1/ > > > Testing > --- > > mvn clean package executes successfully. > > > Thanks, > > Apoorv Naik > >
Re: Review Request 64072: ATLAS-2274, ATLAS-2277: search for classification '*' - doesn't honor excludeDeletedEntities=true; fails when typeName filter is specified
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64072/#review191919 --- Ship it! Ship It! - Ashutosh Mestry On Nov. 26, 2017, 7:34 a.m., Madhan Neethiraj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/64072/ > --- > > (Updated Nov. 26, 2017, 7:34 a.m.) > > > Review request for atlas, Apoorv Naik, Ashutosh Mestry, Nixon Rodrigues, > Sarath Subramanian, and Sharmadha Sainath. > > > Bugs: ATLAS-2274 and ATLAS-2277 > https://issues.apache.org/jira/browse/ATLAS-2274 > https://issues.apache.org/jira/browse/ATLAS-2277 > > > Repository: atlas > > > Description > --- > > Updates to support searching for tagged entities (i.e. classification='*'), > ATLAS-2267, didn't correctly handle excludeDeletedEntities flag and typeName > filter. Fixed basic-search to address these issues. > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java > 0a9e8462 > > repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java > 87efed70 > > > Diff: https://reviews.apache.org/r/64072/diff/1/ > > > Testing > --- > > Verified that basic-search doesn't return deleted entities when > excludeDeletedEntities=true (ATLAS-2277); and also doesn't fail when > typeName filter is specified (ATLAS-2274) > > > Thanks, > > Madhan Neethiraj > >
Re: Review Request 64048: ATLAS-2275: fix for incorrect serialization of date type trait attributes in entity notification
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64048/#review191809 --- Ship it! Ship It! - Ashutosh Mestry On Nov. 23, 2017, 10:24 a.m., Madhan Neethiraj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/64048/ > --- > > (Updated Nov. 23, 2017, 10:24 a.m.) > > > Review request for atlas, Apoorv Naik, Ashutosh Mestry, Abhay Kulkarni, and > Sarath Subramanian. > > > Bugs: ATLAS-2275 > https://issues.apache.org/jira/browse/ATLAS-2275 > > > Repository: atlas > > > Description > --- > > Entity seraialization didn't handle IStruct correctly, casuing this issue. > This patch fixes the issue. > > > Diffs > - > > > notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java > 4f56bd8d > > > Diff: https://reviews.apache.org/r/64048/diff/1/ > > > Testing > --- > > Verified that date type trait attributes are serialzied in correct format in > entity notifications. > > > Thanks, > > Madhan Neethiraj > >
Review Request 64721: DSL: Phase 2: Operators and Test Suite
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64721/ --- Review request for atlas, Apoorv Naik, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-2229 https://issues.apache.org/jira/browse/ATLAS-2229 Repository: atlas Description --- **Background** This implementation builds on top of the earlier Antlr-based DSL implementation. **Scope** Includes: - New framework. - Operators. - Enhancements to _select_ clause. - V1 REST APIs now integrated to call this implementation. - Unit test suite. Diffs - intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java b816e805 repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java cbc08b8c repository/src/main/java/org/apache/atlas/query/DSLVisitor.java a2c6b983 repository/src/main/java/org/apache/atlas/query/IdentifierHelper.java PRE-CREATION repository/src/main/java/org/apache/atlas/query/Lookup.java PRE-CREATION repository/src/main/java/org/apache/atlas/query/QueryProcessor.java 60480a10 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.java 85f8d61f repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.g4 bf6b7e30 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.java 73627a3f repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserBaseVisitor.java 4495f779 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserVisitor.java 4985f8a1 repository/src/test/java/org/apache/atlas/query/BasicTestSetup.java 2aecf2b4 repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java a794a2ae repository/src/test/java/org/apache/atlas/query/QueryProcessorTest.java d1a3d108 webapp/src/main/java/org/apache/atlas/web/resources/MetadataDiscoveryResource.java 0ac4ad56 Diff: https://reviews.apache.org/r/64721/diff/1/ Testing --- **Unit tests** - Added 213 suite of tests that exercise the implementation. - Updated _QueryProcessTest_ to include additional scenarios. **Functional tests** - Verified scenarios from UI. Thanks, Ashutosh Mestry
Re: Review Request 63721: ATLAS-1964 :UI, Support to resize and order columns in Search table
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63721/#review190973 --- Fix it, then Ship it! dashboardv2/gruntfile.js Lines 168 (patched) <https://reviews.apache.org/r/63721/#comment268565> Where is the file with attributations? - Ashutosh Mestry On Nov. 14, 2017, 6:40 a.m., pratik pandey wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/63721/ > --- > > (Updated Nov. 14, 2017, 6:40 a.m.) > > > Review request for atlas, Ashutosh Mestry, keval bhatt, Madhan Neethiraj, and > Nixon Rodrigues. > > > Bugs: ATLAS-1964 > https://issues.apache.org/jira/browse/ATLAS-1964 > > > Repository: atlas > > > Description > --- > > Add support for drag and drop column. > > > Diffs > - > > dashboardv2/gruntfile.js ba9ea0f > dashboardv2/package.json 957e916 > dashboardv2/public/css/scss/search.scss 26f1505 > dashboardv2/public/js/main.js f6c0157 > dashboardv2/public/js/utils/Utils.js 47e9ac2 > dashboardv2/public/js/views/search/SearchResultLayoutView.js 2e8c165 > > > Diff: https://reviews.apache.org/r/63721/diff/1/ > > > Testing > --- > > Done one round of sanity testing. > > > Thanks, > > pratik pandey > >
Re: Review Request 64476: DSL Implementation using Antlr
** _QueryProcessorTest_ implements the testing framework for DSL. It has relevant code that setup up environment to verify various clauses implemented. **Verification** Generated Gremlin queries have been verified using JanusGraph shell customized for Atlas. Thanks, Ashutosh Mestry
Re: Review Request 64476: DSL Implementation using Antlr
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64476/ --- (Updated Dec. 12, 2017, 1:19 a.m.) Review request for atlas, Apoorv Naik, Madhan Neethiraj, and Sarath Subramanian. Changes --- Updates include: - QuickStart clauses. - Additional unit tests. Bugs: ATLAS-2229 https://issues.apache.org/jira/browse/ATLAS-2229 Repository: atlas Description --- **Background** Atlas' DSL implementation was removed with the removal of the older type system implementation, thus creating gap in functionality. The patch implements Atlas' DSL using Antlr and a new from-scratch implementation. **Approach** Highlights of the implementation: - Does not use any of the old scala-based implementation of lexer and parser. It uses Antlr library for this purpose. - Generated Gremlin queries are much simpler than the earlier implementation. - Interface with rest of the application has been streamlined. The Antlr's G4 (_Atlas.G4_) grammer file implements the DSL grammer. Antlr code-generates the requisit lexer and parser. The _DSLVisitor_ class is the concrete visitor implementation for the generated syntax tree. _QueryProcessor_ maintains sequence of Gremlin clauses that are found during the abstract syntax tree traversal. It uses _TypeRegistry_ for creating qualified names. Integration with rest of the application has had little impact. Diffs (updated) - 3party-licenses/Antlr-LICENSE PRE-CREATION pom.xml 4ee2d9ec repository/pom.xml 35b74e32 repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 67aeb308 repository/src/main/java/org/apache/atlas/query/DSLVisitor.java PRE-CREATION repository/src/main/java/org/apache/atlas/query/Expressions.java 9e93ce41 repository/src/main/java/org/apache/atlas/query/GremlinQuery.java fcb1f483 repository/src/main/java/org/apache/atlas/query/GremlinTranslator.java 5395ddd3 repository/src/main/java/org/apache/atlas/query/QueryParser.java 1e5e5ff9 repository/src/main/java/org/apache/atlas/query/QueryProcessor.java 04cf0b4d repository/src/main/java/org/apache/atlas/query/SelectExpressionHelper.java a8748efa repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.g4 PRE-CREATION repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.java PRE-CREATION repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.tokens PRE-CREATION repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.g4 PRE-CREATION repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.java PRE-CREATION repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.tokens PRE-CREATION repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserBaseVisitor.java PRE-CREATION repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserVisitor.java PRE-CREATION repository/src/test/java/org/apache/atlas/query/QueryProcessorTest.java PRE-CREATION repository/src/test/java/org/apache/atlas/services/EntityDiscoveryServiceTest.java PRE-CREATION webapp/src/main/java/org/apache/atlas/examples/QuickStartV2.java a338ae2a Diff: https://reviews.apache.org/r/64476/diff/6/ Changes: https://reviews.apache.org/r/64476/diff/5-6/ Testing --- **Unit tests** _QueryProcessorTest_ implements the testing framework for DSL. It has relevant code that setup up environment to verify various clauses implemented. **Functional testing** - Verified from web UI. - Verified using '_Saved Search_' feature. **Additional verification** Generated Gremlin queries have been verified using JanusGraph shell customized for Atlas. Thanks, Ashutosh Mestry
Re: Review Request 64476: DSL Implementation using Antlr
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64476/ --- (Updated Dec. 12, 2017, 3:23 a.m.) Review request for atlas, Apoorv Naik, Madhan Neethiraj, and Sarath Subramanian. Changes --- Updates include: - Update G4 files. Updates to grammer to handle edge cases, and quoted identifiers. - Updated unit tests. Bugs: ATLAS-2229 https://issues.apache.org/jira/browse/ATLAS-2229 Repository: atlas Description --- **Background** Atlas' DSL implementation was removed with the removal of the older type system implementation, thus creating gap in functionality. The patch implements Atlas' DSL using Antlr and a new from-scratch implementation. **Approach** Highlights of the implementation: - Does not use any of the old scala-based implementation of lexer and parser. It uses Antlr library for this purpose. - Generated Gremlin queries are much simpler than the earlier implementation. - Interface with rest of the application has been streamlined. The Antlr's G4 (_Atlas.G4_) grammer file implements the DSL grammer. Antlr code-generates the requisit lexer and parser. The _DSLVisitor_ class is the concrete visitor implementation for the generated syntax tree. _QueryProcessor_ maintains sequence of Gremlin clauses that are found during the abstract syntax tree traversal. It uses _TypeRegistry_ for creating qualified names. Integration with rest of the application has had little impact. Diffs (updated) - 3party-licenses/Antlr-LICENSE PRE-CREATION pom.xml 233506c8 repository/pom.xml 35b74e32 repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 67aeb308 repository/src/main/java/org/apache/atlas/query/DSLVisitor.java PRE-CREATION repository/src/main/java/org/apache/atlas/query/Expressions.java 9e93ce41 repository/src/main/java/org/apache/atlas/query/GremlinQuery.java fcb1f483 repository/src/main/java/org/apache/atlas/query/GremlinTranslator.java 5395ddd3 repository/src/main/java/org/apache/atlas/query/QueryParser.java 1e5e5ff9 repository/src/main/java/org/apache/atlas/query/QueryProcessor.java 04cf0b4d repository/src/main/java/org/apache/atlas/query/SelectExpressionHelper.java a8748efa repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.g4 PRE-CREATION repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.java PRE-CREATION repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.tokens PRE-CREATION repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.g4 PRE-CREATION repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.java PRE-CREATION repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.tokens PRE-CREATION repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserBaseVisitor.java PRE-CREATION repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserVisitor.java PRE-CREATION repository/src/test/java/org/apache/atlas/query/BasicTestSetup.java PRE-CREATION repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java PRE-CREATION repository/src/test/java/org/apache/atlas/query/QueryProcessorTest.java PRE-CREATION repository/src/test/java/org/apache/atlas/services/EntityDiscoveryServiceTest.java PRE-CREATION webapp/src/main/java/org/apache/atlas/examples/QuickStartV2.java a338ae2a Diff: https://reviews.apache.org/r/64476/diff/7/ Changes: https://reviews.apache.org/r/64476/diff/6-7/ Testing --- **Unit tests** _QueryProcessorTest_ implements the testing framework for DSL. It has relevant code that setup up environment to verify various clauses implemented. **Functional testing** - Verified from web UI. - Verified using '_Saved Search_' feature. **Additional verification** Generated Gremlin queries have been verified using JanusGraph shell customized for Atlas. Thanks, Ashutosh Mestry
Re: Review Request 64476: DSL Implementation using Antlr
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64476/ --- (Updated Dec. 12, 2017, 4:27 a.m.) Review request for atlas, Apoorv Naik, Madhan Neethiraj, and Sarath Subramanian. Changes --- Updates include: - Modification to _QuickStart_ to execute currently supported queries. - Added tests for DSL that were removed as part of earlier clean-up exercise. Bugs: ATLAS-2229 https://issues.apache.org/jira/browse/ATLAS-2229 Repository: atlas Description --- **Background** Atlas' DSL implementation was removed with the removal of the older type system implementation, thus creating gap in functionality. The patch implements Atlas' DSL using Antlr and a new from-scratch implementation. **Approach** Highlights of the implementation: - Does not use any of the old scala-based implementation of lexer and parser. It uses Antlr library for this purpose. - Generated Gremlin queries are much simpler than the earlier implementation. - Interface with rest of the application has been streamlined. The Antlr's G4 (_Atlas.G4_) grammer file implements the DSL grammer. Antlr code-generates the requisit lexer and parser. The _DSLVisitor_ class is the concrete visitor implementation for the generated syntax tree. _QueryProcessor_ maintains sequence of Gremlin clauses that are found during the abstract syntax tree traversal. It uses _TypeRegistry_ for creating qualified names. Integration with rest of the application has had little impact. Diffs (updated) - 3party-licenses/Antlr-LICENSE PRE-CREATION pom.xml 233506c8 repository/pom.xml 35b74e32 repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 67aeb308 repository/src/main/java/org/apache/atlas/query/DSLVisitor.java PRE-CREATION repository/src/main/java/org/apache/atlas/query/Expressions.java 9e93ce41 repository/src/main/java/org/apache/atlas/query/GremlinQuery.java fcb1f483 repository/src/main/java/org/apache/atlas/query/GremlinTranslator.java 5395ddd3 repository/src/main/java/org/apache/atlas/query/QueryParser.java 1e5e5ff9 repository/src/main/java/org/apache/atlas/query/QueryProcessor.java 04cf0b4d repository/src/main/java/org/apache/atlas/query/SelectExpressionHelper.java a8748efa repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.g4 PRE-CREATION repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.java PRE-CREATION repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.tokens PRE-CREATION repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.g4 PRE-CREATION repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.java PRE-CREATION repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.tokens PRE-CREATION repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserBaseVisitor.java PRE-CREATION repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserVisitor.java PRE-CREATION repository/src/test/java/org/apache/atlas/query/BasicTestSetup.java PRE-CREATION repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java PRE-CREATION repository/src/test/java/org/apache/atlas/query/QueryProcessorTest.java PRE-CREATION repository/src/test/java/org/apache/atlas/services/EntityDiscoveryServiceTest.java PRE-CREATION webapp/src/main/java/org/apache/atlas/examples/QuickStartV2.java a338ae2a Diff: https://reviews.apache.org/r/64476/diff/8/ Changes: https://reviews.apache.org/r/64476/diff/7-8/ Testing --- **Unit tests** _QueryProcessorTest_ implements the testing framework for DSL. It has relevant code that setup up environment to verify various clauses implemented. **Functional testing** - Verified from web UI. - Verified using '_Saved Search_' feature. **Additional verification** Generated Gremlin queries have been verified using JanusGraph shell customized for Atlas. Thanks, Ashutosh Mestry
Re: Review Request 64476: DSL Implementation using Antlr
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64476/ --- (Updated Dec. 12, 2017, 7:53 a.m.) Review request for atlas, Apoorv Naik, Madhan Neethiraj, and Sarath Subramanian. Changes --- Updates include: - Operators: LT, GT, LTE, GTE, EQ, NEQ, IN implemented. - Added support for traits with space in their name. - Updated unit tests. Bugs: ATLAS-2229 https://issues.apache.org/jira/browse/ATLAS-2229 Repository: atlas Description --- **Background** Atlas' DSL implementation was removed with the removal of the older type system implementation, thus creating gap in functionality. The patch implements Atlas' DSL using Antlr and a new from-scratch implementation. **Approach** Highlights of the implementation: - Does not use any of the old scala-based implementation of lexer and parser. It uses Antlr library for this purpose. - Generated Gremlin queries are much simpler than the earlier implementation. - Interface with rest of the application has been streamlined. The Antlr's G4 (_Atlas.G4_) grammer file implements the DSL grammer. Antlr code-generates the requisit lexer and parser. The _DSLVisitor_ class is the concrete visitor implementation for the generated syntax tree. _QueryProcessor_ maintains sequence of Gremlin clauses that are found during the abstract syntax tree traversal. It uses _TypeRegistry_ for creating qualified names. Integration with rest of the application has had little impact. Diffs (updated) - 3party-licenses/Antlr-LICENSE PRE-CREATION pom.xml 233506c8 repository/pom.xml 35b74e32 repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 67aeb308 repository/src/main/java/org/apache/atlas/query/DSLVisitor.java PRE-CREATION repository/src/main/java/org/apache/atlas/query/Expressions.java 9e93ce41 repository/src/main/java/org/apache/atlas/query/GremlinQuery.java fcb1f483 repository/src/main/java/org/apache/atlas/query/GremlinTranslator.java 5395ddd3 repository/src/main/java/org/apache/atlas/query/QueryParser.java 1e5e5ff9 repository/src/main/java/org/apache/atlas/query/QueryProcessor.java 04cf0b4d repository/src/main/java/org/apache/atlas/query/SelectExpressionHelper.java a8748efa repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.g4 PRE-CREATION repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.java PRE-CREATION repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.tokens PRE-CREATION repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.g4 PRE-CREATION repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.java PRE-CREATION repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.tokens PRE-CREATION repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserBaseVisitor.java PRE-CREATION repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserVisitor.java PRE-CREATION repository/src/test/java/org/apache/atlas/query/BasicTestSetup.java PRE-CREATION repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java PRE-CREATION repository/src/test/java/org/apache/atlas/query/QueryProcessorTest.java PRE-CREATION repository/src/test/java/org/apache/atlas/services/EntityDiscoveryServiceTest.java PRE-CREATION webapp/src/main/java/org/apache/atlas/examples/QuickStartV2.java a338ae2a Diff: https://reviews.apache.org/r/64476/diff/9/ Changes: https://reviews.apache.org/r/64476/diff/8-9/ Testing --- **Unit tests** _QueryProcessorTest_ implements the testing framework for DSL. It has relevant code that setup up environment to verify various clauses implemented. **Functional testing** - Verified from web UI. - Verified using '_Saved Search_' feature. **Additional verification** Generated Gremlin queries have been verified using JanusGraph shell customized for Atlas. Thanks, Ashutosh Mestry
Re: Review Request 66899: Large Notification Messages: Avoid Duplicate Processing
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66899/ --- (Updated May 4, 2018, 4:33 p.m.) Review request for atlas and Madhan Neethiraj. Changes --- Updates include: - Addressed review comments. - Updated unit tests. Bugs: ATLAS-2634 https://issues.apache.org/jira/browse/ATLAS-2634 Repository: atlas Description --- **Approach** Simplified approach. **Algorithm** - Modified: _NotificationHookConsumer_: - when commit() fails, record current offset of the message. - on receiving a message - if _offset_ is same as that of the received message, skip processing the message, go ahead and commit. - reset recorded offset. Added appropriate log messages. **Additional Refactoring** None. Diffs (updated) - notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 80dc51423 webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 7a4596a7e webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java d2b3dfdab Diff: https://reviews.apache.org/r/66899/diff/4/ Changes: https://reviews.apache.org/r/66899/diff/3-4/ Testing --- **Unit tests** Additional unit tests to test specific scenarios. Thanks, Ashutosh Mestry
Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66928/ --- (Updated May 15, 2018, 4:18 a.m.) Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath Subramanian. Changes --- Updates include: - Handling primitive and non-primitive Map. - Handling non-primitive Arrays. - Refactoring. - Fixed failing UTs. Bugs: ATLAS-2637 https://issues.apache.org/jira/browse/ATLAS-2637 Repository: atlas Description --- **Approach** New class: _TypesStoringEdgeIds_: Navigates through all the types in the _typeRegistry_, returns map of entity type and properties that store edge ids. Modified: _DataMigrationService_: Uses output from class above and passes it down to migration. Modified: _PostProcessManager_: Uses the map generated above and uses it for post processing. Modified: _GraphSONUtility_: Improvement to check for vertex of type. This avoids potential exeception when a non-existent property is checked for presence. Added PostProcess framework. Added logic for handling new Array and Map representation. Diffs (updated) - graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 607baf664 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java c0b9c1741 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java 16aecd5e2 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java aa0b6375d graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java ec320b03e graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java d0a65f7b1 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java e4e82649b graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java PRE-CREATION graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessMapProperty.java PRE-CREATION graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessor.java PRE-CREATION graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java de43e0f41 graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java 4d73c78ef graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java 794b5471e graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCacheTest.java cac09d229 graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java PRE-CREATION graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessMapPropertyTest.java PRE-CREATION graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION graphdb/janus/src/test/resources/table-v-147504.json 898dce5df intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b intg/src/test/java/org/apache/atlas/TestUtilsV2.java 886ce77f5 repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java f289365f5 repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java 22cd55217 repository/src/main/java/org/apache/atlas/repository/migration/TypesStoringEdgeIds.java PRE-CREATION repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 3c84e3c22 repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java ec6e64a25 repository/src/test/java/org/apache/atlas/repository/migration/TypesStoringEdgeIdsTest.java PRE-CREATION repository/src/test/resources/parts_db/atlas-migration-data.json 1414ea160 Diff: https://reviews.apache.org/r/66928/diff/9/ Changes: https://reviews.apache.org/r/66928/diff/8-9/ Testing (updated) --- **Unit tests** Additional tests added. **Functional tests** Regular flow verified. **[Pre-commit build](https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/404/)** Thanks, Ashutosh Mestry
Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with JanusGraph 0.2.0
> On April 13, 2018, 12:34 a.m., Ashutosh Mestry wrote: > > Can you please add instructions on how to use this profile? I tried the > > patch today and I am still having trouble getting Atlas up. > > Pierre Padovani wrote: > Let me rebase the patch and make some changes, and retest. Master has > moved a bit since I generated the original patch. I'll update as soon as I > have it done. > > Pierre Padovani wrote: > I updated the patch four days ago, can you please retest? > > Ashutosh Mestry wrote: > I will retest and let you know. Please give me a couple of days. > > Pierre Padovani wrote: > Any updates on this? > > Thanks! Can you please rebase this with latest master? - Ashutosh --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66064/#review201069 --- On April 13, 2018, 2:55 a.m., Pierre Padovani wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/66064/ > --- > > (Updated April 13, 2018, 2:55 a.m.) > > > Review request for atlas, David Radley, Madhan Neethiraj, and Sarath > Subramanian. > > > Bugs: ATLAS-2478 > https://issues.apache.org/jira/browse/ATLAS-2478 > > > Repository: atlas > > > Description > --- > > This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates > documentation. > > Included with this patch is an update to the berkley-elasticsearch profile to > automatically download and include elasticsearch as a side application much > like solr is. Updates to the start/stop/conf scripts are included as well. > > NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to > /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There > are six constants that are incorrectly named with a '.' (dot). This is not > supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** > the field names can be collectively thought of as an object. In the case of > the fields defined in the Constants.java file, 'type' is defined as a string > field, and 'type.name' is also defined as a string field. Elasticsearch sees > this as an error, since it cannot convert type to an object. The fix included > simply changes the field names from using a '.' (dot) to an '_' (underscore). > This should NOT affect compatibility with hbase/solr for new installs. For > existing installations, a reindex will be required as the field names will > have changed. > > **Query**: There is a way we can simplify integration/unit tests for the > in-memory graph store by using a maven plugin that will download and run an > elasticsearch node. This is nothing more than a maven change, and change to > the atlas-application.properties to switch to elasticsearch from solr. I did > not implement this, but am curious if this change would be desired. If so, > this can be done with a separate ticket. > > > Diffs > - > > common/src/main/java/org/apache/atlas/repository/Constants.java 01e49157d > distro/pom.xml 1f4c6d557 > distro/src/bin/atlas_config.py 9062da649 > distro/src/bin/atlas_start.py 61d69eb21 > distro/src/bin/atlas_stop.py 94c3d6d46 > distro/src/conf/atlas-env.sh 9213f488b > distro/src/main/assemblies/standalone-package.xml dc2a66ba9 > docs/src/site/twiki/Configuration.twiki 63c3fce96 > docs/src/site/twiki/HighAvailability.twiki 4270d0974 > docs/src/site/twiki/InstallationSteps.twiki dca0618e3 > graphdb/janus/pom.xml 016a09c33 > pom.xml 4d2ab7391 > webapp/pom.xml 284f538f7 > > > Diff: https://reviews.apache.org/r/66064/diff/2/ > > > Testing > --- > > We currently use this fix with our Atlas setup in a fork of the Atlas code, > and have found no issues with it. Additionally, with the update to the > berkley-elasticsearch profile, I have extensively tested that profile to make > sure that management of Elasticsearch functioned correctly. > > > Thanks, > > Pierre Padovani > >
Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with JanusGraph 0.2.0
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66064/#review202602 --- graphdb/janus/pom.xml Lines 121 (patched) <https://reviews.apache.org/r/66064/#comment284490> This is getting included even when the profile is not selected. - Ashutosh Mestry On May 7, 2018, 9:28 p.m., Pierre Padovani wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/66064/ > --- > > (Updated May 7, 2018, 9:28 p.m.) > > > Review request for atlas, David Radley, Madhan Neethiraj, and Sarath > Subramanian. > > > Bugs: ATLAS-2478 > https://issues.apache.org/jira/browse/ATLAS-2478 > > > Repository: atlas > > > Description > --- > > This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates > documentation. > > Included with this patch is an update to the berkley-elasticsearch profile to > automatically download and include elasticsearch as a side application much > like solr is. Updates to the start/stop/conf scripts are included as well. > > NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to > /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There > are six constants that are incorrectly named with a '.' (dot). This is not > supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** > the field names can be collectively thought of as an object. In the case of > the fields defined in the Constants.java file, 'type' is defined as a string > field, and 'type.name' is also defined as a string field. Elasticsearch sees > this as an error, since it cannot convert type to an object. The fix included > simply changes the field names from using a '.' (dot) to an '_' (underscore). > This should NOT affect compatibility with hbase/solr for new installs. For > existing installations, a reindex will be required as the field names will > have changed. > > **Query**: There is a way we can simplify integration/unit tests for the > in-memory graph store by using a maven plugin that will download and run an > elasticsearch node. This is nothing more than a maven change, and change to > the atlas-application.properties to switch to elasticsearch from solr. I did > not implement this, but am curious if this change would be desired. If so, > this can be done with a separate ticket. > > > Diffs > - > > common/src/main/java/org/apache/atlas/repository/Constants.java 16a3aaa15 > distro/pom.xml 6431fd86d > distro/src/bin/atlas_config.py 9062da649 > distro/src/bin/atlas_start.py 61d69eb21 > distro/src/bin/atlas_stop.py 94c3d6d46 > distro/src/conf/atlas-env.sh 053cbd500 > distro/src/main/assemblies/standalone-package.xml dc88add56 > docs/src/site/twiki/Configuration.twiki 63c3fce96 > docs/src/site/twiki/HighAvailability.twiki 4270d0974 > docs/src/site/twiki/InstallationSteps.twiki dca0618e3 > graphdb/janus/pom.xml 016a09c33 > pom.xml 3469a393a > webapp/pom.xml 03b84087f > > > Diff: https://reviews.apache.org/r/66064/diff/3/ > > > Testing > --- > > We currently use this fix with our Atlas setup in a fork of the Atlas code, > and have found no issues with it. Additionally, with the update to the > berkley-elasticsearch profile, I have extensively tested that profile to make > sure that management of Elasticsearch functioned correctly. > > > Thanks, > > Pierre Padovani > >
Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with JanusGraph 0.2.0
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review202603
---
On executing ./bin/atlas_start.py displays these messages:
./target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/conf/atlas-env.sh:
line 65: MANAGE_EMBEDDED_CASSANDRA=${cassandra.embedded}: bad substitution
./target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/conf/atlas-env.sh:
line 68: MANAGE_LOCAL_ELASTICSEARCH=${elasticsearch.managed}: bad substitution
Exception: ('Could not find hbase-site.xml in %s. Please set env var
HBASE_CONF_DIR to the hbase client conf dir',
'./target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/hbase/conf')
Traceback (most recent call last):
File "./bin/atlas_start.py", line 163, in
File "./bin/atlas_start.py", line 92, in main
Exception: ('Could not find hbase-site.xml in %s. Please set env var
HBASE_CONF_DIR to the hbase client conf dir',
'./target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/hbase/conf')
- Ashutosh Mestry
On May 7, 2018, 9:28 p.m., Pierre Padovani wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> ---
>
> (Updated May 7, 2018, 9:28 p.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath
> Subramanian.
>
>
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
>
>
> Repository: atlas
>
>
> Description
> ---
>
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates
> documentation.
>
> Included with this patch is an update to the berkley-elasticsearch profile to
> automatically download and include elasticsearch as a side application much
> like solr is. Updates to the start/stop/conf scripts are included as well.
>
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to
> /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There
> are six constants that are incorrectly named with a '.' (dot). This is not
> supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS**
> the field names can be collectively thought of as an object. In the case of
> the fields defined in the Constants.java file, 'type' is defined as a string
> field, and 'type.name' is also defined as a string field. Elasticsearch sees
> this as an error, since it cannot convert type to an object. The fix included
> simply changes the field names from using a '.' (dot) to an '_' (underscore).
> This should NOT affect compatibility with hbase/solr for new installs. For
> existing installations, a reindex will be required as the field names will
> have changed.
>
> **Query**: There is a way we can simplify integration/unit tests for the
> in-memory graph store by using a maven plugin that will download and run an
> elasticsearch node. This is nothing more than a maven change, and change to
> the atlas-application.properties to switch to elasticsearch from solr. I did
> not implement this, but am curious if this change would be desired. If so,
> this can be done with a separate ticket.
>
>
> Diffs
> -
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 16a3aaa15
> distro/pom.xml 6431fd86d
> distro/src/bin/atlas_config.py 9062da649
> distro/src/bin/atlas_start.py 61d69eb21
> distro/src/bin/atlas_stop.py 94c3d6d46
> distro/src/conf/atlas-env.sh 053cbd500
> distro/src/main/assemblies/standalone-package.xml dc88add56
> docs/src/site/twiki/Configuration.twiki 63c3fce96
> docs/src/site/twiki/HighAvailability.twiki 4270d0974
> docs/src/site/twiki/InstallationSteps.twiki dca0618e3
> graphdb/janus/pom.xml 016a09c33
> pom.xml 3469a393a
> webapp/pom.xml 03b84087f
>
>
> Diff: https://reviews.apache.org/r/66064/diff/3/
>
>
> Testing
> ---
>
> We currently use this fix with our Atlas setup in a fork of the Atlas code,
> and have found no issues with it. Additionally, with the update to the
> berkley-elasticsearch profile, I have extensively tested that profile to make
> sure that management of Elasticsearch functioned correctly.
>
>
> Thanks,
>
> Pierre Padovani
>
>
Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with JanusGraph 0.2.0
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66064/#review202597 --- Ship it! Ship It! - Ashutosh Mestry On May 7, 2018, 9:28 p.m., Pierre Padovani wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/66064/ > --- > > (Updated May 7, 2018, 9:28 p.m.) > > > Review request for atlas, David Radley, Madhan Neethiraj, and Sarath > Subramanian. > > > Bugs: ATLAS-2478 > https://issues.apache.org/jira/browse/ATLAS-2478 > > > Repository: atlas > > > Description > --- > > This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates > documentation. > > Included with this patch is an update to the berkley-elasticsearch profile to > automatically download and include elasticsearch as a side application much > like solr is. Updates to the start/stop/conf scripts are included as well. > > NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to > /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There > are six constants that are incorrectly named with a '.' (dot). This is not > supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** > the field names can be collectively thought of as an object. In the case of > the fields defined in the Constants.java file, 'type' is defined as a string > field, and 'type.name' is also defined as a string field. Elasticsearch sees > this as an error, since it cannot convert type to an object. The fix included > simply changes the field names from using a '.' (dot) to an '_' (underscore). > This should NOT affect compatibility with hbase/solr for new installs. For > existing installations, a reindex will be required as the field names will > have changed. > > **Query**: There is a way we can simplify integration/unit tests for the > in-memory graph store by using a maven plugin that will download and run an > elasticsearch node. This is nothing more than a maven change, and change to > the atlas-application.properties to switch to elasticsearch from solr. I did > not implement this, but am curious if this change would be desired. If so, > this can be done with a separate ticket. > > > Diffs > - > > common/src/main/java/org/apache/atlas/repository/Constants.java 16a3aaa15 > distro/pom.xml 6431fd86d > distro/src/bin/atlas_config.py 9062da649 > distro/src/bin/atlas_start.py 61d69eb21 > distro/src/bin/atlas_stop.py 94c3d6d46 > distro/src/conf/atlas-env.sh 053cbd500 > distro/src/main/assemblies/standalone-package.xml dc88add56 > docs/src/site/twiki/Configuration.twiki 63c3fce96 > docs/src/site/twiki/HighAvailability.twiki 4270d0974 > docs/src/site/twiki/InstallationSteps.twiki dca0618e3 > graphdb/janus/pom.xml 016a09c33 > pom.xml 3469a393a > webapp/pom.xml 03b84087f > > > Diff: https://reviews.apache.org/r/66064/diff/3/ > > > Testing > --- > > We currently use this fix with our Atlas setup in a fork of the Atlas code, > and have found no issues with it. Additionally, with the update to the > berkley-elasticsearch profile, I have extensively tested that profile to make > sure that management of Elasticsearch functioned correctly. > > > Thanks, > > Pierre Padovani > >
Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66928/ --- (Updated May 11, 2018, 12:40 a.m.) Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath Subramanian. Changes --- Updates include: Array and Map type handling. Bugs: ATLAS-2637 https://issues.apache.org/jira/browse/ATLAS-2637 Repository: atlas Description (updated) --- **Approach** New class: _TypesStoringEdgeIds_: Navigates through all the types in the _typeRegistry_, returns map of entity type and properties that store edge ids. Modified: _DataMigrationService_: Uses output from class above and passes it down to migration. Modified: _PostProcessManager_: Uses the map generated above and uses it for post processing. Modified: _GraphSONUtility_: Improvement to check for vertex of type. This avoids potential exeception when a non-existent property is checked for presence. Added PostProcess framework. Added logic for handling new Array and Map representation. Diffs (updated) - graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 607baf664 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java c0b9c1741 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java 3636ca65a graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java aa0b6375d graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java ec320b03e graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java d0a65f7b1 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java PRE-CREATION graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessMapProperty.java PRE-CREATION graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessor.java PRE-CREATION graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java de43e0f41 graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java 4d73c78ef graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java PRE-CREATION graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessMapPropertyTest.java PRE-CREATION graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION graphdb/janus/src/test/resources/table-v-147504.json 898dce5df intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java 22cd55217 repository/src/main/java/org/apache/atlas/repository/migration/TypesStoringEdgeIds.java PRE-CREATION repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 3c84e3c22 repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java ec6e64a25 repository/src/test/java/org/apache/atlas/repository/migration/TypesStoringEdgeIdsTest.java PRE-CREATION Diff: https://reviews.apache.org/r/66928/diff/5/ Changes: https://reviews.apache.org/r/66928/diff/4-5/ Testing --- **Unit tests** Additional tests added. **Functional tests** Regular flow verified. Thanks, Ashutosh Mestry
Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66928/ --- (Updated May 11, 2018, 1:50 a.m.) Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath Subramanian. Changes --- Updates include: Fixes based on testing. Bugs: ATLAS-2637 https://issues.apache.org/jira/browse/ATLAS-2637 Repository: atlas Description --- **Approach** New class: _TypesStoringEdgeIds_: Navigates through all the types in the _typeRegistry_, returns map of entity type and properties that store edge ids. Modified: _DataMigrationService_: Uses output from class above and passes it down to migration. Modified: _PostProcessManager_: Uses the map generated above and uses it for post processing. Modified: _GraphSONUtility_: Improvement to check for vertex of type. This avoids potential exeception when a non-existent property is checked for presence. Added PostProcess framework. Added logic for handling new Array and Map representation. Diffs (updated) - graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 607baf664 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java c0b9c1741 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java 3636ca65a graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java aa0b6375d graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java ec320b03e graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java d0a65f7b1 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java PRE-CREATION graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessMapProperty.java PRE-CREATION graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessor.java PRE-CREATION graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java de43e0f41 graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java 4d73c78ef graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCacheTest.java cac09d229 graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java PRE-CREATION graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessMapPropertyTest.java PRE-CREATION graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION graphdb/janus/src/test/resources/table-v-147504.json 898dce5df intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java 22cd55217 repository/src/main/java/org/apache/atlas/repository/migration/TypesStoringEdgeIds.java PRE-CREATION repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 3c84e3c22 repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java ec6e64a25 repository/src/test/java/org/apache/atlas/repository/migration/TypesStoringEdgeIdsTest.java PRE-CREATION Diff: https://reviews.apache.org/r/66928/diff/7/ Changes: https://reviews.apache.org/r/66928/diff/6-7/ Testing --- **Unit tests** Additional tests added. **Functional tests** Regular flow verified. Thanks, Ashutosh Mestry
Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66928/ --- (Updated May 11, 2018, 10:27 p.m.) Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath Subramanian. Changes --- Updates include: - Refactoring. - Minor modification to post processing. - Fix to new attribute assignment. Bugs: ATLAS-2637 https://issues.apache.org/jira/browse/ATLAS-2637 Repository: atlas Description --- **Approach** New class: _TypesStoringEdgeIds_: Navigates through all the types in the _typeRegistry_, returns map of entity type and properties that store edge ids. Modified: _DataMigrationService_: Uses output from class above and passes it down to migration. Modified: _PostProcessManager_: Uses the map generated above and uses it for post processing. Modified: _GraphSONUtility_: Improvement to check for vertex of type. This avoids potential exeception when a non-existent property is checked for presence. Added PostProcess framework. Added logic for handling new Array and Map representation. Diffs (updated) - graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 607baf664 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java c0b9c1741 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java 3636ca65a graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java aa0b6375d graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java ec320b03e graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java d0a65f7b1 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java e4e82649b graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java PRE-CREATION graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessMapProperty.java PRE-CREATION graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessor.java PRE-CREATION graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java de43e0f41 graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java 4d73c78ef graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java 794b5471e graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCacheTest.java cac09d229 graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java PRE-CREATION graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessMapPropertyTest.java PRE-CREATION graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION graphdb/janus/src/test/resources/table-v-147504.json 898dce5df intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java 22cd55217 repository/src/main/java/org/apache/atlas/repository/migration/TypesStoringEdgeIds.java PRE-CREATION repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 3c84e3c22 repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java ec6e64a25 repository/src/test/java/org/apache/atlas/repository/migration/TypesStoringEdgeIdsTest.java PRE-CREATION Diff: https://reviews.apache.org/r/66928/diff/8/ Changes: https://reviews.apache.org/r/66928/diff/7-8/ Testing --- **Unit tests** Additional tests added. **Functional tests** Regular flow verified. **[Pre-commit build](https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/382/)** Thanks, Ashutosh Mestry
Re: Review Request 66949: ATLAS-2523: changes to accept external GUIDs and manage homeIds
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66949/#review202564 --- Ship it! Ship It! - Ashutosh Mestry On May 4, 2018, 10:55 a.m., Graham Wallis wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/66949/ > --- > > (Updated May 4, 2018, 10:55 a.m.) > > > Review request for atlas, Ashutosh Mestry and Madhan Neethiraj. > > > Repository: atlas > > > Description > --- > > ATLAS-2523: Changes to accept external GUIDs and manage homeIds > > > Diffs > - > > common/src/main/java/org/apache/atlas/repository/Constants.java > c570be2ccbc41a426b0f093b4a33163092223f2f > intg/src/main/java/org/apache/atlas/model/instance/AtlasEntity.java > e6a7f19633a1642f6b415db99e20c0641df9463b > intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationship.java > d04daa5d283fdba90b917f38eba6c937021352df > intg/src/main/java/org/apache/atlas/type/AtlasTypeUtil.java > 1b6af94f24eb7d99953f72815167c868a9bd9efd > repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java > edf10da9533803234342dc3313ea1024c5e7030f > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1.java > 9fcba6ddaa1b74b2c81b980bcb455b614f4a4ed7 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphMapper.java > d51adad80e133d636ef84883d395126ae0150af5 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphRetriever.java > 3b9a287f6986ff6bca26fc20b2f94ec1700dbe1e > > > Diff: https://reviews.apache.org/r/66949/diff/1/ > > > Testing > --- > > Functional testing of these changes to save reference copies of entities and > relationships > > > Thanks, > > Graham Wallis > >
Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66928/ --- (Updated May 20, 2018, 4:30 p.m.) Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath Subramanian. Changes --- Updates include: - Addressed review comments. - Updated pre-commit build #. Bugs: ATLAS-2637 https://issues.apache.org/jira/browse/ATLAS-2637 Repository: atlas Description --- **Approach** New class: _TypesStoringEdgeIds_: Navigates through all the types in the _typeRegistry_, returns map of entity type and properties that store edge ids. Modified: _DataMigrationService_: Uses output from class above and passes it down to migration. Modified: _PostProcessManager_: Uses the map generated above and uses it for post processing. Modified: _GraphSONUtility_: Improvement to check for vertex of type. This avoids potential exeception when a non-existent property is checked for presence. Added PostProcess framework. Added logic for handling new Array and Map representation. Diffs (updated) - graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 607baf664 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java c0b9c1741 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java 16aecd5e2 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java ae119b0bc graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java ec320b03e graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java d0a65f7b1 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java e4e82649b graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java PRE-CREATION graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessor.java PRE-CREATION graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java e863d9fae graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java 4d73c78ef graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java 794b5471e graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCacheTest.java cac09d229 graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java PRE-CREATION graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION graphdb/janus/src/test/resources/edge-legacy-col3.json PRE-CREATION graphdb/janus/src/test/resources/edge-legacy-col4.json PRE-CREATION graphdb/janus/src/test/resources/table-v-147504.json 898dce5df intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b intg/src/test/java/org/apache/atlas/TestUtilsV2.java 886ce77f5 repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java 22cd55217 repository/src/main/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinder.java PRE-CREATION repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 3c84e3c22 repository/src/test/java/org/apache/atlas/repository/migration/ComplexAttributesTest.java PRE-CREATION repository/src/test/java/org/apache/atlas/repository/migration/HiveParititionTest.java ac0b79d38 repository/src/test/java/org/apache/atlas/repository/migration/HiveStocksTest.java ffbf3200b repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java ec6e64a25 repository/src/test/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinderTest.java PRE-CREATION repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityTestBase.java d810a1166 repository/src/test/resources/complex-attr_db/atlas-migration-data.json PRE-CREATION repository/src/test/resources/complex-attr_db/atlas-migration-typesdef.json PRE-CREATION repository/src/test/resources/parts_db/atlas-migration-data.json 1414ea160 Diff: https://reviews.apache.org/r/66928/diff/17/ Changes: https://reviews.apache.org/r/66928/diff/16-17/ Testing (updated) --- **Unit tests** Additional tests added. **Functional tests** Regular flow verified. **[Pre-commit build](https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/423/)** Thanks, Ashutosh Mestry
Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids
> On May 19, 2018, 12:36 a.m., Madhan Neethiraj wrote: > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java > > Line 67 (original), 74 (patched) > > <https://reviews.apache.org/r/66928/diff/16/?file=2025013#file2025013line74> > > > > This one handles removal/updte of properties for map type attributes. > > Aren't similar updates needed for array type attributes as well > > (primitive/struct/entity-ref)? Yes, only that the indexes are known only after all edges are created. I considered doing this during creation itself, it eded up becoming messy and number of computations creases drastically. There is an option that is possible if it is some how possible to store a temporary map of edge ids to indexes. - Ashutosh --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66928/#review203462 ------- On May 17, 2018, 8:48 p.m., Ashutosh Mestry wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/66928/ > --- > > (Updated May 17, 2018, 8:48 p.m.) > > > Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath > Subramanian. > > > Bugs: ATLAS-2637 > https://issues.apache.org/jira/browse/ATLAS-2637 > > > Repository: atlas > > > Description > --- > > **Approach** > New class: _TypesStoringEdgeIds_: Navigates through all the types in the > _typeRegistry_, returns map of entity type and properties that store edge ids. > Modified: _DataMigrationService_: Uses output from class above and passes it > down to migration. > Modified: _PostProcessManager_: Uses the map generated above and uses it for > post processing. > Modified: _GraphSONUtility_: Improvement to check for vertex of type. This > avoids potential exeception when a non-existent property is checked for > presence. > > Added PostProcess framework. > Added logic for handling new Array and Map representation. > > > Diffs > - > > > graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java > 607baf664 > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java > c0b9c1741 > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java > 16aecd5e2 > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java > ae119b0bc > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java > ec320b03e > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java > d0a65f7b1 > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java > e4e82649b > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java > PRE-CREATION > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessor.java > PRE-CREATION > > graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java > e863d9fae > > graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java > 4d73c78ef > > graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java > 794b5471e > > graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCacheTest.java > cac09d229 > > graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java > PRE-CREATION > graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION > graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION > graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION > graphdb/janus/src/test/resources/edge-legacy-col3.json PRE-CREATION > graphdb/janus/src/test/resources/edge-legacy-col4.json PRE-CREATION > graphdb/janus/src/test/resources/table-v-147504.json 898dce5df > intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b > intg/src/test/java/org/apache/atlas/TestUtilsV2.java 886ce77f5 > > repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationServ
Re: Review Request 67239: ATLAS-2710: basic-search doesn't return deleted entities though excludeDeletedEntities is False
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/67239/#review203543 --- Ship it! Ship It! - Ashutosh Mestry On May 21, 2018, 10:54 p.m., Madhan Neethiraj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/67239/ > --- > > (Updated May 21, 2018, 10:54 p.m.) > > > Review request for atlas, Apoorv Naik, Ashutosh Mestry, keval bhatt, Nixon > Rodrigues, and Sarath Subramanian. > > > Bugs: ATLAS-2710 > https://issues.apache.org/jira/browse/ATLAS-2710 > > > Repository: atlas > > > Description > --- > > This is caused by entity-delete removing vertex property that stores > associated classification names. This property is used by basic-search to > look for entities, hence the deleted entity was not returned. > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java > f417b4e95 > > > Diff: https://reviews.apache.org/r/67239/diff/1/ > > > Testing > --- > > Verified that basic-search returns deleted entities as well when > excludeDeletedEntities is False. > > > Thanks, > > Madhan Neethiraj > >
Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids
/migration/TypesWithCollectionsFinderTest.java PRE-CREATION repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityTestBase.java d810a1166 repository/src/test/resources/complex-attr_db/atlas-migration-data.json PRE-CREATION repository/src/test/resources/complex-attr_db/atlas-migration-typesdef.json PRE-CREATION repository/src/test/resources/parts_db/atlas-migration-data.json 1414ea160 Diff: https://reviews.apache.org/r/66928/diff/19/ Changes: https://reviews.apache.org/r/66928/diff/18-19/ Testing --- **Unit tests** Additional tests added. **Functional tests** Regular flow verified. **[Pre-commit build](https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/423/)** Thanks, Ashutosh Mestry
Re: Review Request 67273: ATLAS-2714: model updates - added relationship-def _AtlasUserProfile_savedsearches, updated few relationships to align end1/end2 with edge-direction, fixed cardinality of fe
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/67273/#review203719 --- Ship it! Ship It! - Ashutosh Mestry On May 23, 2018, 8:46 p.m., Madhan Neethiraj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/67273/ > --- > > (Updated May 23, 2018, 8:46 p.m.) > > > Review request for atlas, Apoorv Naik, Ashutosh Mestry, and Sarath > Subramanian. > > > Bugs: ATLAS-2714 > https://issues.apache.org/jira/browse/ATLAS-2714 > > > Repository: atlas > > > Description > --- > > - added relationship-def _AtlasUserProfile_savedsearches, to avoid WARN > messages in Atlas log file > - updated few relationships to align end1/end2 with edge-direction: > dataset_process_inputs, hive_table_db, hbase_table_namespace > - fixed cardinality of array attributes to LIST/SET (instead of SINGLE - > which causes confusion) > > > Diffs > - > > addons/models/-Area0/0010-base_model.json fe8001915 > addons/models/1000-Hadoop/1030-hive_model.json 552dfe483 > addons/models/1000-Hadoop/1060-hbase_model.json 392412aec > > repository/src/test/java/org/apache/atlas/repository/migration/ComplexAttributesTest.java > 9451fa263 > > repository/src/test/java/org/apache/atlas/repository/migration/HiveParititionTest.java > 7b23e59de > > repository/src/test/java/org/apache/atlas/repository/migration/HiveStocksTest.java > 721bb31fa > > repository/src/test/java/org/apache/atlas/repository/migration/PathTest.java > 5c8decca3 > > > Diff: https://reviews.apache.org/r/67273/diff/2/ > > > Testing > --- > > - verified that WARN message is no more present in Atlas log > - verified that hive_db/hive_table entities CRUD & lineage work as before > (i.e. no regression due to model changes) > - precommit test run: > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/444/ > > > Thanks, > > Madhan Neethiraj > >
Re: Review Request 67302: Data Migration: Handle Legacy Types that Reference Classifications
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/67302/ --- (Updated May 25, 2018, 8:21 p.m.) Review request for atlas, Madhan Neethiraj and Ruchi Solani. Changes --- Updates include: - Addressed case where cardinality was _list_ and _set_. Existing logic was not sufficient to handle this case. Bugs: ATLAS-2717 https://issues.apache.org/jira/browse/ATLAS-2717 Repository: atlas Description --- **Approach** - Attributes of classification types are pre-processed (before incoming types are processed). - New _STRUCT_ types are created with the classification name prefixed with _legacy_. So, classification of the name PII is used to create a _STRUCT_ type named _legacy_PII_. - Entity definitions are updated to reference these newly created types. - During edge processing, the type name of the in vertex is replaced with this new type. - Rest of the processing continues as is. **Refacoring** - Moved migration-specific functions to separate class _GraphDBMigrator_. Diffs (updated) - graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java e5316d80a graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/GraphDBMigrator.java PRE-CREATION graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java b4d6b33ea graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java c9d6067db graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ElementProcessors.java f51080a61 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphDBGraphSONMigrator.java PRE-CREATION graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java f1bbfcfcd graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/TypesDefScrubber.java PRE-CREATION graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java aee1b698c graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java 049fd6418 graphdb/janus/src/test/resources/entity-with-trait-type.json PRE-CREATION repository/src/main/java/org/apache/atlas/repository/impexp/MigrationProgressService.java 9620c1357 repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java 081376ff5 repository/src/test/java/org/apache/atlas/TestModules.java 23d128740 repository/src/test/java/org/apache/atlas/repository/migration/ComplexAttributesTest.java 800638d43 repository/src/test/java/org/apache/atlas/repository/migration/HiveParititionTest.java cae751215 repository/src/test/java/org/apache/atlas/repository/migration/HiveStocksTest.java 9f255189d repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java 4b02b7fb7 repository/src/test/java/org/apache/atlas/repository/migration/MigrationProgressServiceTest.java 8fc524e81 repository/src/test/java/org/apache/atlas/repository/migration/PathTest.java 468b5268c repository/src/test/java/org/apache/atlas/repository/migration/TypesDefScrubberTest.java PRE-CREATION repository/src/test/java/org/apache/atlas/repository/migration/TypesWithClassificationTest.java PRE-CREATION repository/src/test/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinderTest.java 4b236c1fe repository/src/test/resources/classification_defs/atlas-migration-data.json PRE-CREATION repository/src/test/resources/classification_defs/atlas-migration-typesdef.json PRE-CREATION repository/src/test/resources/legacy-typesdef.json PRE-CREATION Diff: https://reviews.apache.org/r/67302/diff/4/ Changes: https://reviews.apache.org/r/67302/diff/3-4/ Testing --- **Unit tests** - Added tests to verify the new flow. - Executed existing tests. **Test data** - In versions 2.5 onwards (post 2/13/2017) it is not possible to create entities of such types. Entity creation API throws a validation error. - The test data was thus hand-created. - This data verifies the cases where attribute type is: - Classification type - Array of classification type. - Map of classification type. **Build** [Pre-commit](https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/451/) File Attachments atlas-migration-typesdef.json https://reviews.apache.org/media/uploaded/files/2018/05/24/c5cc2171-27f5-4d26-acd9-08aac9e1e1ef__atlas-migration-typesdef.json atlas-migration-data.json https://reviews.apache.org/media/uploaded/files/2018/05/24/8230c44b-9f2e-4ae2-bc1a-04424dafc08c__atlas-migration-data.json Thanks, Ashutosh Mestry
Re: Review Request 67302: Data Migration: Handle Legacy Types that Reference Classifications
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/67302/ --- (Updated May 25, 2018, 9:26 p.m.) Review request for atlas, Madhan Neethiraj and Ruchi Solani. Changes --- Updates include: Scanning of struct type during scrub process. Bugs: ATLAS-2717 https://issues.apache.org/jira/browse/ATLAS-2717 Repository: atlas Description --- **Approach** - Attributes of classification types are pre-processed (before incoming types are processed). - New _STRUCT_ types are created with the classification name prefixed with _legacy_. So, classification of the name PII is used to create a _STRUCT_ type named _legacy_PII_. - Entity definitions are updated to reference these newly created types. - During edge processing, the type name of the in vertex is replaced with this new type. - Rest of the processing continues as is. **Refacoring** - Moved migration-specific functions to separate class _GraphDBMigrator_. Diffs (updated) - graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java e5316d80a graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/GraphDBMigrator.java PRE-CREATION graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java b4d6b33ea graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java c9d6067db graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ElementProcessors.java f51080a61 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphDBGraphSONMigrator.java PRE-CREATION graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java f1bbfcfcd graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/TypesDefScrubber.java PRE-CREATION graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java aee1b698c graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java 049fd6418 graphdb/janus/src/test/resources/entity-with-trait-type.json PRE-CREATION repository/src/main/java/org/apache/atlas/repository/impexp/MigrationProgressService.java 9620c1357 repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java 081376ff5 repository/src/test/java/org/apache/atlas/TestModules.java 23d128740 repository/src/test/java/org/apache/atlas/repository/migration/ComplexAttributesTest.java 800638d43 repository/src/test/java/org/apache/atlas/repository/migration/HiveParititionTest.java cae751215 repository/src/test/java/org/apache/atlas/repository/migration/HiveStocksTest.java 9f255189d repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java 4b02b7fb7 repository/src/test/java/org/apache/atlas/repository/migration/MigrationProgressServiceTest.java 8fc524e81 repository/src/test/java/org/apache/atlas/repository/migration/PathTest.java 468b5268c repository/src/test/java/org/apache/atlas/repository/migration/TypesDefScrubberTest.java PRE-CREATION repository/src/test/java/org/apache/atlas/repository/migration/TypesWithClassificationTest.java PRE-CREATION repository/src/test/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinderTest.java 4b236c1fe repository/src/test/resources/classification_defs/atlas-migration-data.json PRE-CREATION repository/src/test/resources/classification_defs/atlas-migration-typesdef.json PRE-CREATION repository/src/test/resources/legacy-typesdef.json PRE-CREATION Diff: https://reviews.apache.org/r/67302/diff/5/ Changes: https://reviews.apache.org/r/67302/diff/4-5/ Testing --- **Unit tests** - Added tests to verify the new flow. - Executed existing tests. **Test data** - In versions 2.5 onwards (post 2/13/2017) it is not possible to create entities of such types. Entity creation API throws a validation error. - The test data was thus hand-created. - This data verifies the cases where attribute type is: - Classification type - Array of classification type. - Map of classification type. **Build** [Pre-commit](https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/451/) File Attachments atlas-migration-typesdef.json https://reviews.apache.org/media/uploaded/files/2018/05/24/c5cc2171-27f5-4d26-acd9-08aac9e1e1ef__atlas-migration-typesdef.json atlas-migration-data.json https://reviews.apache.org/media/uploaded/files/2018/05/24/8230c44b-9f2e-4ae2-bc1a-04424dafc08c__atlas-migration-data.json Thanks, Ashutosh Mestry
Re: Review Request 67221: Data Migration: Documentation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/67221/ --- (Updated May 25, 2018, 1:44 a.m.) Review request for atlas, Madhan Neethiraj and Ruchi Solani. Changes --- Updates include: - Addressed review comments. - Some restructuring. - Addition information about classification as type handling. Bugs: ATLAS-2704 https://issues.apache.org/jira/browse/ATLAS-2704 Repository: atlas Description --- Documentation. Diffs (updated) - docs/src/site/twiki/Migration-0.8-to-1.0.twiki PRE-CREATION docs/src/site/twiki/index.twiki ed1461571 Diff: https://reviews.apache.org/r/67221/diff/2/ Changes: https://reviews.apache.org/r/67221/diff/1-2/ Testing --- Launched site using: _mvn site:run_ and verified the contents. Thanks, Ashutosh Mestry
