GitHub user prj2113 opened a pull request:
https://github.com/apache/incubator-usergrid/pull/42
Import Feature
This pull request adds in the import rest api to usergrid. Docs
forthcoming. Current iteration suffers from two problems: the first is a
streaming api is needed for the files to properly have distributed processing
and the counts for each number of entities created per file isn't currently
working within the rx module. These errors are logged and will be corrected
with time.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/prj2113/incubator-usergrid master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-usergrid/pull/42.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #42
----
commit a2612b7a581c24805c44eebc0da500e7d736e19c
Author: Pooja Jain <[email protected]>
Date: 2014-07-10T22:56:26Z
changed the jcloud version from 1.6.2-incubating to 1.7.1
commit c11a3962598b85236decd5117a7210df898d016c
Author: Harish Rajagopal <[email protected]>
Date: 2014-07-11T16:20:37Z
import skeleton code
commit de5f6650306ee1f99df53e0aa20951a6076d66f3
Author: Pooja Jain <[email protected]>
Date: 2014-07-11T23:56:30Z
- Added implementation for Downloading CollectionFile and ApplicationFile
from S3
- Added Integration tests to test the above cases
- Added beans to the spring configuration
- Added the services to the setup file
commit 35df4be245a9493efbceac97e9608c431f796e9c
Author: Pooja Jain <[email protected]>
Date: 2014-07-14T20:50:34Z
- Added implmentation for importing all applications from an organization
- Fixed the regex for identifying file names
- Added test for testing organization import
commit db3ea6f8b1c25a5e065e9dcdd3a0051c1b94f5a5
Author: Harish Rajagopal <[email protected]>
Date: 2014-07-15T17:24:22Z
import entity skeleton code
commit 7aa5b33246d7dbf056adb59faec0e0775ef19458
Author: Pooja Jain <[email protected]>
Date: 2014-07-15T19:52:23Z
trying to code the file parsing logic
commit 653450751cb3a902799cd8314a926b12aa96ea3c
Author: Pooja Jain <[email protected]>
Date: 2014-07-15T22:52:29Z
- "added code to parse a json file and update collections"
TO DO:
-update connections and dictionaries
-write integration tests to check for update
commit 9cdf2037c3a6476b7660755c3537b6d8fc3a7eea
Author: Pooja Jain <[email protected]>
Date: 2014-07-15T23:52:11Z
added testcase to check for updated collection incase of importCollection
commit 870ef1f71e67c48ee546cfeffaa6fef8f4fc1d5a
Author: Pooja Jain <[email protected]>
Date: 2014-07-16T16:40:15Z
added createConnections in the service test
commit 3ce1a9c5552fd4e43acd20eb7882d4151e28ccd3
Author: Pooja Jain <[email protected]>
Date: 2014-07-16T18:45:14Z
Added implementation for parsing connections and dictionaries. Also, the
integration tests for collections tests this correctly
commit c1c40a7cccc9168d57dfd92b66d5955f5b42ab27
Author: Pooja Jain <[email protected]>
Date: 2014-07-16T21:20:37Z
refactored code and added test case for testing import organization
commit c0c2a89c7ecb1547117c18696135a71ae1ca1e47
Author: Pooja Jain <[email protected]>
Date: 2014-07-16T23:10:20Z
added code for testing import organization and restored files for
AppResource and OrgResource
commit 73ca07ee3a4b31715905d92fef583287bcee5f27
Author: Harish Rajagopal <[email protected]>
Date: 2014-07-16T23:30:43Z
application integration tests
commit c8898e5cbfba39820dd311c141b73281b7cb8080
Author: Pooja Jain <[email protected]>
Date: 2014-07-17T00:19:55Z
ResourceIT testcases working perfectly. Completed a very basic single
threaded import
commit de5298ca4ecd785b70e8bb11d7c083b1006de0eb
Author: Pooja Jain <[email protected]>
Date: 2014-07-17T00:56:00Z
added comments to ImportServiceIT and reorganized code so that all the
testcases can pass even if run together
commit 280c324eaa1f431dbd3e95e6ea3e7110758a137c
Author: Pooja Jain <[email protected]>
Date: 2014-07-17T17:30:44Z
fixed the integration tests and now deleting temp import files
commit b8854d198ac1623c10cd5ee639ad51d7d440a9c5
Author: Pooja Jain <[email protected]>
Date: 2014-07-17T19:40:40Z
fixed the ordering of methods in saveCollectionMember
and compeleted the test cases
commit ef8b5d3ab4bc130ac13c94184ec0b31ba6c18f82
Author: Harish Rajagopal <[email protected]>
Date: 2014-07-17T20:08:44Z
application level integration tests complete
commit 60dd2695f556e78a50507f311ca6ee6f1cbca469
Author: Pooja Jain <[email protected]>
Date: 2014-07-17T20:59:06Z
- Added implementation for deleting import temp files
- Added method to delete buckets
commit 1d2af7f6751887314702995a92fd1f11b55f167e
Author: Pooja Jain <[email protected]>
Date: 2014-07-17T22:03:33Z
added resourceTests
commit d9874b6bafbdbbc46fee74fa3eeeeab424406a75
Author: Harish Rajagopal <[email protected]>
Date: 2014-07-17T22:18:16Z
Resource nullpointer integration tests for import
commit 12b2a7386f2ea2a8c9b730f8118e2bd7f7c17542
Author: Harish Rajagopal <[email protected]>
Date: 2014-07-17T22:27:08Z
"Fix bug"
- USERGRID-193
- The saveConnections() and saveDictionaries() methods are called at the
end of SaveCollectionMembers() in file ExportServiceImpl.java, but if
collection for an entity was null, then it would just return and not export the
connections and dictionaries for that particular entity, thus calling these
functions in the start of the method ensures, that all connections and
dictionaries are always exported.
"Improve Export Test Cases"
- USERGRID-194
- Removed redundant code and added testcases for exportOrganization Null
pointer checks.
commit 045b5bfc6dd124633f01179294f184eee1c6bca6
Author: Pooja Jain <[email protected]>
Date: 2014-07-18T18:16:44Z
cleaned code and added comments
commit 2dd38f36b333a2a1ea3a73c540633fc5f9e832a0
Author: Harish Rajagopal <[email protected]>
Date: 2014-07-18T22:28:20Z
Merge remote-tracking branch 'origin/import-pooja' into import-harish
commit d7590e862d86c594fe457df60443c2d9620fc9c9
Author: Harish Rajagopal <[email protected]>
Date: 2014-07-18T22:32:11Z
blobstore does not throw an exception when it cannot find a bucket
commit d48001949cd23026621a41b98adf50b53865d00b
Author: Pooja Jain <[email protected]>
Date: 2014-07-18T22:40:40Z
added testcases for covergae for ImportServiceImpl and fixed a small bug
in importServiceImpl
commit 643045788951b79bbec6dc89a95378ba333d5e5f
Author: Pooja Jain <[email protected]>
Date: 2014-07-18T22:41:10Z
Merge remote-tracking branch 'origin/import-harish' into import-pooja
commit d8a6d366b64bfe1d3bbe0647c38da19d29a962cc
Author: Pooja Jain <[email protected]>
Date: 2014-07-22T21:07:39Z
- changed method signature for getErrorMessage as appID is not used at all
in the method. It requires the managment_app_ID to retrieve info of the import
job
- removed test case for the above
- Handled throwing exceptions to restart job which throws runtime exception.
- Handled different kind of exceptions and wrote test cases for a few.
- Fixed exception handling in copyFromS3
commit 75722674c2c1371d8462adc58110098cb2f580c2
Author: Pooja Jain <[email protected]>
Date: 2014-07-22T23:05:03Z
added code to store import job metadate and resume code
commit af7531902c81875762423ac05c8b66c29cdceed6
Author: Pooja Jain <[email protected]>
Date: 2014-07-23T19:04:38Z
Ignored test cases in the service layer which need s3 credentials
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
