----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71426/#review217657 -----------------------------------------------------------
Fix it, then Ship it! distro/src/conf/atlas-application.properties Lines 69 (patched) <https://reviews.apache.org/r/71426/#comment304949> atlas.import.temporaryDirectory => atlas.import.temp.directory docs/src/site/twiki/Import-API-Options.twiki Lines 144 (patched) <https://reviews.apache.org/r/71426/#comment304950> "It is up to the user to ascertain" => "Please ensure " repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java Line 99 (original), 103 (patched) <https://reviews.apache.org/r/71426/#comment304943> '@Override' seems to be missing for few methods - like getTypesDef(), getExportResult(). Please review and update. repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceWithBackingDirectory.java Lines 74 (patched) <https://reviews.apache.org/r/71426/#comment304951> please move 'private' methods after all public and protected methods. repository/src/main/java/org/apache/atlas/repository/store/graph/v1/BulkImporterImpl.java Lines 52 (patched) <https://reviews.apache.org/r/71426/#comment304948> Please avoid statics *as much as possible*. Replace use of directoryBasedImportConfigured with: if (StringUtils.isEmpty(AtlasConfiguration.IMPORT_TEMP_DIRECTORY.getString())) { } repository/src/main/java/org/apache/atlas/repository/store/graph/v1/BulkImporterImpl.java Lines 220 (patched) <https://reviews.apache.org/r/71426/#comment304947> Consider replacing this block, #220 - #231, with a following: package org.apache.atlas; public enum AtlasConfiguration { ... IMPORT_TEMP_DIRECTORY("atlas.import.temp.directory", "") } repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java Line 102 (original), 103 (patched) <https://reviews.apache.org/r/71426/#comment304946> getInputStreamFrom(): this variation does return a ZipSource; so consider retaining current name, getZipSourceFrom(). - Madhan Neethiraj On Sept. 9, 2019, 5:35 a.m., Ashutosh Mestry wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71426/ > ----------------------------------------------------------- > > (Updated Sept. 9, 2019, 5:35 a.m.) > > > Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, > and Sarath Subramanian. > > > Bugs: ATLAS-3396 > https://issues.apache.org/jira/browse/ATLAS-3396 > > > Repository: atlas > > > Description > ------- > > **Background** > The approach adds another option to _ImportService_ to be able to optionally > use a directory on the server for storing data to be imported. > > **Approach** > This is a backward compatible. > > - - New: _ZipSourceWithBackingDirectory_: Uses a temporary directory for > storing contents of the zip file. > - Modified: _ImportService_ Now takes _InputStream_ as parameter instead of > _ZipSource_. > - Modified: _ImportRequest_ Additionl option takes _backingDirectory_ as > additional option. Specifying this will use the new > _ZipSourceWithBackingDirectory_ during import. > > > **CURL** > > Use pre-configured temporary directory, set _atlas.import.temporaryDirectory_ > in the applicatino properties. > > _CURL_ > ``` > curl -v -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H > "Cache-Control: no-cache" -F request=@../docs/import-options.json -F > data=@../docs/smalldb.zip http://localhost:21000/api/atlas/admin/import > ``` > > **Documentation** > Updated. > > > Diffs > ----- > > distro/src/conf/atlas-application.properties 654a9304e > docs/src/site/twiki/Import-API-Options.twiki 4004e7013 > > repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java > b5d8b7c39 > repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java > 016acd743 > > repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceWithBackingDirectory.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStreamForImport.java > 90ae15d1e > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/BulkImporterImpl.java > 26066920a > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityImportStream.java > d4b6c5505 > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java > a355297a8 > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java > eaf4602d5 > > repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java > 693a163f8 > > repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformsShaperTest.java > 06bdaa6d6 > > repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java > 868b732d3 > > repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java > a2a5f58dc > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java > 8417e7eb0 > > > Diff: https://reviews.apache.org/r/71426/diff/6/ > > > Testing > ------- > > **Unit tests** > Updated to exercise the new implementation. > > **Volume tests** > Imported 4GB ZIP file that took 26 hours. Memory and CPU stayed in normal > range during this operation. > > **Functional tests** > Accuracy tests preformed on small size data. > > **Pre-commit Build** > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1380/ > > > Thanks, > > Ashutosh Mestry > >
