Author: jsedding
Date: Mon Nov 30 15:30:52 2015
New Revision: 1717276
URL: http://svn.apache.org/viewvc?rev=1717276&view=rev
Log:
OAK-3685 - Support S3DataSource as the migration source
- patch provided by Tomek Rękawek, thanks!
Added:
jackrabbit/oak/trunk/oak-upgrade/src/test/java/org/apache/jackrabbit/oak/upgrade/cli/blob/S3ToFbsTest.java
- copied, changed from r1717263,
jackrabbit/oak/trunk/oak-upgrade/src/test/java/org/apache/jackrabbit/oak/upgrade/cli/blob/FbsToS3Test.java
jackrabbit/oak/trunk/oak-upgrade/src/test/java/org/apache/jackrabbit/oak/upgrade/cli/blob/StripConfigValueTest.java
- copied, changed from r1717263,
jackrabbit/oak/trunk/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/blob/FileBlobStoreFactory.java
Modified:
jackrabbit/oak/trunk/oak-doc/src/site/markdown/migration.md
jackrabbit/oak/trunk/oak-upgrade/pom.xml
jackrabbit/oak/trunk/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/CliUtils.java
jackrabbit/oak/trunk/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/blob/DummyBlobStoreFactory.java
jackrabbit/oak/trunk/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/blob/FileBlobStoreFactory.java
jackrabbit/oak/trunk/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/blob/FileDataStoreFactory.java
jackrabbit/oak/trunk/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/blob/S3DataStoreFactory.java
jackrabbit/oak/trunk/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/parser/OptionParserFactory.java
jackrabbit/oak/trunk/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/parser/StoreArguments.java
jackrabbit/oak/trunk/oak-upgrade/src/main/resources/upgrade_usage.txt
jackrabbit/oak/trunk/oak-upgrade/src/test/java/org/apache/jackrabbit/oak/upgrade/cli/blob/FbsToS3Test.java
jackrabbit/oak/trunk/oak-upgrade/src/test/java/org/apache/jackrabbit/oak/upgrade/cli/container/S3DataStoreContainer.java
Modified: jackrabbit/oak/trunk/oak-doc/src/site/markdown/migration.md
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-doc/src/site/markdown/migration.md?rev=1717276&r1=1717275&r2=1717276&view=diff
==============================================================================
--- jackrabbit/oak/trunk/oak-doc/src/site/markdown/migration.md (original)
+++ jackrabbit/oak/trunk/oak-doc/src/site/markdown/migration.md Mon Nov 30
15:30:52 2015
@@ -74,25 +74,25 @@ In order to upgrade Jackrabbit 2 reposit
By default, the `oak-upgrade` only copies the binary references, so you need
to reuse the same blob/data store in the new repository. However, it's also
possible to migrate binaries as well using the `--copy-binaries` parameter.
Following migration paths are possible for the binaries. The *internal* means
that the binaries are stored inside the segment or document node store:
-| From â To â | Internal | FileBlobStore | FileDataStore | S3 |
-| :-----------: | :------: | :-----------: | :-----------: | :--: |
-| Internal | Yes² | Yes | Yes | Yes |
-| FileBlobStore | Yes | Yes² | Yes | Yes |
-| FileDataStore | Yes | Yes | Yes² | Yes¹ |
-| S3 | \- | \- | \- | Yes² |
+| From â To â | Internal | FileBlobStore | FileDataStore | S3
|
+| :-----------: | :------: | :-----------: | :-----------: |
:--------------------: |
+| Internal | Yes² | Yes | Yes | Yes
|
+| FileBlobStore | Yes | Yes² | Yes | Yes
|
+| FileDataStore | Yes | Yes | Yes² | Yes (not
recommended)¹ |
+| S3 | Yes | Yes | Yes | Yes²
|
-¹ Online migration is available as well using Granite Amazon S3DataStore
Connector <br/>
+¹ The S3DataStore will take care of this migration automatically, no need to
use oak-upgrade <br/>
² The storage might be simple cloned without using oak2oak
Following parameters can be used to define the source and the destination blob
stores:
-| Blob store type | Source parameter | Destination
|
-| :-------------: | :--------------------------------: | :-------------------:
|
-| FileBlobStore | `--src-fileblobstore=...` | `--fileblobstore=...`
|
-| FileDataStore | `--src-datastore=...` | `--datastore=...`
|
-| S3 | `--s3config=... --s3datastore=...` | \-
|
+| Blob store type | Source parameter | Destination
|
+| :-------------: | :----------------------------------------: |
:--------------------------------: |
+| FileBlobStore | `--src-fileblobstore=...` |
`--fileblobstore=...` |
+| FileDataStore | `--src-datastore=...` |
`--datastore=...` |
+| S3 | `--src-s3config=... --src-s3datastore=...` |
`--s3config=... --s3datastore=...` |
-Use the `--copy-binaries` parameter to instruct the `oak-upgrade` to copy
binaries:
+Use the `--copy-binaries` parameter to instruct the `oak-upgrade` to copy
binaries.
Example:
@@ -103,6 +103,26 @@ Example:
/old/repository \
/new/repository
+#### S3 configuration
+
+Using S3DataStore as a source or destination for binaries requires passing two
arguments: `s3datastore` and `s3config`. The first one should point to the
datastore directory (eg. `crx-quickstart/repository/datastore`). The second
should be used to define the
`org.apache.jackrabbit.oak.plugins.blob.datastore.S3DataStore.cfg`
configuration file path. File should have following format:
+
+ accessKey=...
+ secretKey=...
+ s3Bucket=...
+ s3Region=eu-west-1
+ s3EndPoint=s3-eu-west-1.amazonaws.com
+
+ connectionTimeout=120000
+ socketTimeout=120000
+ maxConnections=40
+ writeThreads=30
+ maxErrorRetry=10
+
+For the region and endpoints please visit the [Amazon
documentation](http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region).
+
+Alternativly the `*.config` file format, described on the [Apache Sling
website](https://sling.apache.org/documentation/bundles/configuration-installer-factory.html#configuration-files),
can be used as it may be convenient for re-using existing OSGi configuration
files.
+
### Migrating a subtree
Modified: jackrabbit/oak/trunk/oak-upgrade/pom.xml
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-upgrade/pom.xml?rev=1717276&r1=1717275&r2=1717276&view=diff
==============================================================================
--- jackrabbit/oak/trunk/oak-upgrade/pom.xml (original)
+++ jackrabbit/oak/trunk/oak-upgrade/pom.xml Mon Nov 30 15:30:52 2015
@@ -156,6 +156,24 @@
<version>${h2.version}</version>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>com.amazonaws</groupId>
+ <artifactId>aws-java-sdk</artifactId>
+ <version>1.9.11</version>
+ <exclusions>
+ <exclusion>
+ <groupId>joda-time</groupId>
+ <artifactId>joda-time</artifactId>
+ </exclusion>
+ </exclusions>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>joda-time</groupId>
+ <artifactId>joda-time</artifactId>
+ <version>2.2</version>
+ <scope>test</scope>
+ </dependency>
</dependencies>
<!-- Workaround for http://bugs.java.com/view_bug.do?bug_id=6550655 -->
Modified:
jackrabbit/oak/trunk/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/CliUtils.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/CliUtils.java?rev=1717276&r1=1717275&r2=1717276&view=diff
==============================================================================
---
jackrabbit/oak/trunk/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/CliUtils.java
(original)
+++
jackrabbit/oak/trunk/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/CliUtils.java
Mon Nov 30 15:30:52 2015
@@ -33,7 +33,6 @@ import com.google.common.io.Closer;
import joptsimple.OptionParser;
import joptsimple.OptionSet;
-@SuppressWarnings("restriction")
public class CliUtils {
private static final Logger log =
LoggerFactory.getLogger(OakUpgrade.class);
Modified:
jackrabbit/oak/trunk/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/blob/DummyBlobStoreFactory.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/blob/DummyBlobStoreFactory.java?rev=1717276&r1=1717275&r2=1717276&view=diff
==============================================================================
---
jackrabbit/oak/trunk/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/blob/DummyBlobStoreFactory.java
(original)
+++
jackrabbit/oak/trunk/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/blob/DummyBlobStoreFactory.java
Mon Nov 30 15:30:52 2015
@@ -27,4 +27,8 @@ public class DummyBlobStoreFactory imple
return null;
}
+ @Override
+ public String toString() {
+ return "DummyBlobStore";
+ }
}
Modified:
jackrabbit/oak/trunk/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/blob/FileBlobStoreFactory.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/blob/FileBlobStoreFactory.java?rev=1717276&r1=1717275&r2=1717276&view=diff
==============================================================================
---
jackrabbit/oak/trunk/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/blob/FileBlobStoreFactory.java
(original)
+++
jackrabbit/oak/trunk/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/blob/FileBlobStoreFactory.java
Mon Nov 30 15:30:52 2015
@@ -34,4 +34,8 @@ public class FileBlobStoreFactory implem
return new FileBlobStore(directory);
}
+ @Override
+ public String toString() {
+ return String.format("FileBlobStore[%s]", directory);
+ }
}
Modified:
jackrabbit/oak/trunk/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/blob/FileDataStoreFactory.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/blob/FileDataStoreFactory.java?rev=1717276&r1=1717275&r2=1717276&view=diff
==============================================================================
---
jackrabbit/oak/trunk/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/blob/FileDataStoreFactory.java
(original)
+++
jackrabbit/oak/trunk/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/blob/FileDataStoreFactory.java
Mon Nov 30 15:30:52 2015
@@ -51,4 +51,9 @@ public class FileDataStoreFactory implem
}
};
}
+
+ @Override
+ public String toString() {
+ return String.format("FileDataStore[%s]", directory);
+ }
}
Modified:
jackrabbit/oak/trunk/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/blob/S3DataStoreFactory.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/blob/S3DataStoreFactory.java?rev=1717276&r1=1717275&r2=1717276&view=diff
==============================================================================
---
jackrabbit/oak/trunk/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/blob/S3DataStoreFactory.java
(original)
+++
jackrabbit/oak/trunk/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/blob/S3DataStoreFactory.java
Mon Nov 30 15:30:52 2015
@@ -20,10 +20,14 @@ import java.io.Closeable;
import java.io.File;
import java.io.FileReader;
import java.io.IOException;
+import java.util.HashSet;
import java.util.Properties;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
import javax.jcr.RepositoryException;
+import org.apache.commons.io.FileUtils;
import org.apache.commons.io.IOUtils;
import org.apache.jackrabbit.core.data.CachingDataStore;
import org.apache.jackrabbit.core.data.DataStoreException;
@@ -34,15 +38,20 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.google.common.io.Closer;
+import com.google.common.io.Files;
public class S3DataStoreFactory implements BlobStoreFactory {
private static final Logger log =
LoggerFactory.getLogger(S3DataStoreFactory.class);
+ private static final Pattern STRIP_VALUE_PATTERN =
Pattern.compile("^[TILFDXSCB]?\"(.*)\"\\W*$");
+
private final Properties props;
private final String directory;
+ private final File tempHomeDir;
+
public S3DataStoreFactory(String configuration, String directory) throws
IOException {
this.props = new Properties();
FileReader reader = new FileReader(new File(configuration));
@@ -51,24 +60,31 @@ public class S3DataStoreFactory implemen
} finally {
IOUtils.closeQuietly(reader);
}
+
+ for (Object key : new HashSet<Object>(props.keySet())) {
+ String value = props.getProperty((String) key);
+ props.put(key, stripValue(value));
+ }
+
this.directory = directory;
+ this.tempHomeDir = Files.createTempDir();
}
@Override
public BlobStore create(Closer closer) throws IOException {
S3DataStore delegate = new S3DataStore();
delegate.setProperties(props);
- delegate.setPath(props.getProperty("path"));
+ delegate.setPath(directory);
try {
- delegate.init(directory);
+ delegate.init(tempHomeDir.getPath());
} catch (RepositoryException e) {
throw new IOException(e);
}
- closer.register(asCloseable(delegate));
+ closer.register(asCloseable(delegate, tempHomeDir));
return new DataStoreBlobStore(delegate);
}
- private static Closeable asCloseable(final CachingDataStore store) {
+ private static Closeable asCloseable(final CachingDataStore store, final
File tempHomeDir) {
return new Closeable() {
@Override
public void close() throws IOException {
@@ -78,6 +94,7 @@ public class S3DataStoreFactory implemen
Thread.sleep(1000);
}
store.close();
+ FileUtils.deleteDirectory(tempHomeDir);
} catch (DataStoreException e) {
throw new IOException(e);
} catch (InterruptedException e) {
@@ -86,4 +103,18 @@ public class S3DataStoreFactory implemen
}
};
}
+
+ static String stripValue(String value) {
+ Matcher matcher = STRIP_VALUE_PATTERN.matcher(value);
+ if (matcher.matches()) {
+ return matcher.group(1);
+ } else {
+ return value;
+ }
+ }
+
+ @Override
+ public String toString() {
+ return String.format("S3DataStore[%s]", directory);
+ }
}
Modified:
jackrabbit/oak/trunk/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/parser/OptionParserFactory.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/parser/OptionParserFactory.java?rev=1717276&r1=1717275&r2=1717276&view=diff
==============================================================================
---
jackrabbit/oak/trunk/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/parser/OptionParserFactory.java
(original)
+++
jackrabbit/oak/trunk/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/parser/OptionParserFactory.java
Mon Nov 30 15:30:52 2015
@@ -46,6 +46,10 @@ public class OptionParserFactory {
public static final String SRC_FDS = "src-datastore";
+ public static final String SRC_S3 = "src-s3datastore";
+
+ public static final String SRC_S3_CONFIG = "src-s3config";
+
public static final String DST_FDS = "datastore";
public static final String DST_FBS = "fileblobstore";
@@ -72,10 +76,6 @@ public class OptionParserFactory {
addPathsOptions(op);
addVersioningOptions(op);
addMiscOptions(op);
-
-// op.nonOptions(
-//
"[/path/to/oak/repository|/path/to/crx2/repository|mongodb://host:port|<Jdbc
URI>] [/path/to/repository.xml]
{/path/to/oak/repository|mongodb://host:port|<Jdbc URI>}");
-
return op;
}
@@ -89,11 +89,14 @@ public class OptionParserFactory {
.ofType(String.class);
op.accepts(SRC_FBS, "Datastore directory to be used as a source
FileBlobStore").withRequiredArg()
.ofType(String.class);
+ op.accepts(SRC_S3, "Datastore directory to be used for the source
S3").withRequiredArg().ofType(String.class);
+ op.accepts(SRC_S3_CONFIG, "Configuration file for the source
S3DataStore").withRequiredArg()
+ .ofType(String.class);
op.accepts(DST_FDS, "Datastore directory to be used as a target
FileDataStore").withRequiredArg()
- .ofType(String.class);
+ .ofType(String.class);
op.accepts(DST_FBS, "Datastore directory to be used as a target
FileBlobStore").withRequiredArg()
.ofType(String.class);
- op.accepts(DST_S3, "Repository home to be used for the target
S3").withRequiredArg().ofType(String.class);
+ op.accepts(DST_S3, "Datastore directory to be used for the target
S3").withRequiredArg().ofType(String.class);
op.accepts(DST_S3_CONFIG, "Configuration file for the target
S3DataStore").withRequiredArg()
.ofType(String.class);
}
Modified:
jackrabbit/oak/trunk/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/parser/StoreArguments.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/parser/StoreArguments.java?rev=1717276&r1=1717275&r2=1717276&view=diff
==============================================================================
---
jackrabbit/oak/trunk/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/parser/StoreArguments.java
(original)
+++
jackrabbit/oak/trunk/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/parser/StoreArguments.java
Mon Nov 30 15:30:52 2015
@@ -34,6 +34,8 @@ import org.slf4j.LoggerFactory;
import static
org.apache.jackrabbit.oak.upgrade.cli.parser.OptionParserFactory.SRC_FBS;
import static
org.apache.jackrabbit.oak.upgrade.cli.parser.OptionParserFactory.SRC_FDS;
+import static
org.apache.jackrabbit.oak.upgrade.cli.parser.OptionParserFactory.SRC_S3;
+import static
org.apache.jackrabbit.oak.upgrade.cli.parser.OptionParserFactory.SRC_S3_CONFIG;
import static
org.apache.jackrabbit.oak.upgrade.cli.parser.OptionParserFactory.DST_FBS;
import static
org.apache.jackrabbit.oak.upgrade.cli.parser.OptionParserFactory.DST_FDS;
import static
org.apache.jackrabbit.oak.upgrade.cli.parser.OptionParserFactory.DST_S3;
@@ -83,15 +85,18 @@ public class StoreArguments {
return dst.getFactory(MigrationDirection.DST, parser);
}
- public BlobStoreFactory getSrcBlobStore() {
+ public BlobStoreFactory getSrcBlobStore() throws IOException {
BlobStoreFactory factory;
if (parser.hasOption(SRC_FBS)) {
factory = new FileBlobStoreFactory(parser.getOption(SRC_FBS));
+ } else if (parser.hasOption(SRC_S3_CONFIG) &&
parser.hasOption(SRC_S3)) {
+ factory = new S3DataStoreFactory(parser.getOption(SRC_S3_CONFIG),
parser.getOption(SRC_S3));
} else if (parser.hasOption(SRC_FDS)) {
factory = new FileDataStoreFactory(parser.getOption(SRC_FDS));
} else {
factory = new DummyBlobStoreFactory();
}
+ log.info("Source blob store: {}", factory);
return factory;
}
@@ -106,6 +111,7 @@ public class StoreArguments {
} else {
factory = new DummyBlobStoreFactory();
}
+ log.info("Destination blob store: {}", factory);
return factory;
}
Modified: jackrabbit/oak/trunk/oak-upgrade/src/main/resources/upgrade_usage.txt
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-upgrade/src/main/resources/upgrade_usage.txt?rev=1717276&r1=1717275&r2=1717276&view=diff
==============================================================================
--- jackrabbit/oak/trunk/oak-upgrade/src/main/resources/upgrade_usage.txt
(original)
+++ jackrabbit/oak/trunk/oak-upgrade/src/main/resources/upgrade_usage.txt Mon
Nov 30 15:30:52 2015
@@ -76,6 +76,8 @@ An descriptor of the Oak node store. Pos
FileDataStore
--src-fileblobstore Datastore directory to be used as a source
FileBlobStore
+--src-s3datastore Datastore directory to be used for the source S3
+--src-s3config Configuration file for the source S3DataStore
# destination blob store options
@@ -83,8 +85,8 @@ An descriptor of the Oak node store. Pos
FileDataStore
--fileblobstore Datastore directory to be used as a target
FileBlobStore
+--s3datastore Datastore directory to be used for the target S3
--s3config Configuration file for the target S3DataStore
---s3datastore Repository home to be used for the target S3
# other options
Modified:
jackrabbit/oak/trunk/oak-upgrade/src/test/java/org/apache/jackrabbit/oak/upgrade/cli/blob/FbsToS3Test.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-upgrade/src/test/java/org/apache/jackrabbit/oak/upgrade/cli/blob/FbsToS3Test.java?rev=1717276&r1=1717275&r2=1717276&view=diff
==============================================================================
---
jackrabbit/oak/trunk/oak-upgrade/src/test/java/org/apache/jackrabbit/oak/upgrade/cli/blob/FbsToS3Test.java
(original)
+++
jackrabbit/oak/trunk/oak-upgrade/src/test/java/org/apache/jackrabbit/oak/upgrade/cli/blob/FbsToS3Test.java
Mon Nov 30 15:30:52 2015
@@ -16,6 +16,8 @@
*/
package org.apache.jackrabbit.oak.upgrade.cli.blob;
+import java.io.IOException;
+
import org.apache.jackrabbit.oak.upgrade.cli.AbstractOak2OakTest;
import org.apache.jackrabbit.oak.upgrade.cli.container.BlobStoreContainer;
import org.apache.jackrabbit.oak.upgrade.cli.container.FileBlobStoreContainer;
@@ -36,7 +38,7 @@ public class FbsToS3Test extends Abstrac
private final NodeStoreContainer destination;
- public FbsToS3Test() {
+ public FbsToS3Test() throws IOException {
Assume.assumeTrue(S3_PROPERTIES != null);
sourceBlob = new FileBlobStoreContainer();
destinationBlob = new S3DataStoreContainer(S3_PROPERTIES);
Copied:
jackrabbit/oak/trunk/oak-upgrade/src/test/java/org/apache/jackrabbit/oak/upgrade/cli/blob/S3ToFbsTest.java
(from r1717263,
jackrabbit/oak/trunk/oak-upgrade/src/test/java/org/apache/jackrabbit/oak/upgrade/cli/blob/FbsToS3Test.java)
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-upgrade/src/test/java/org/apache/jackrabbit/oak/upgrade/cli/blob/S3ToFbsTest.java?p2=jackrabbit/oak/trunk/oak-upgrade/src/test/java/org/apache/jackrabbit/oak/upgrade/cli/blob/S3ToFbsTest.java&p1=jackrabbit/oak/trunk/oak-upgrade/src/test/java/org/apache/jackrabbit/oak/upgrade/cli/blob/FbsToS3Test.java&r1=1717263&r2=1717276&rev=1717276&view=diff
==============================================================================
---
jackrabbit/oak/trunk/oak-upgrade/src/test/java/org/apache/jackrabbit/oak/upgrade/cli/blob/FbsToS3Test.java
(original)
+++
jackrabbit/oak/trunk/oak-upgrade/src/test/java/org/apache/jackrabbit/oak/upgrade/cli/blob/S3ToFbsTest.java
Mon Nov 30 15:30:52 2015
@@ -16,6 +16,8 @@
*/
package org.apache.jackrabbit.oak.upgrade.cli.blob;
+import java.io.IOException;
+
import org.apache.jackrabbit.oak.upgrade.cli.AbstractOak2OakTest;
import org.apache.jackrabbit.oak.upgrade.cli.container.BlobStoreContainer;
import org.apache.jackrabbit.oak.upgrade.cli.container.FileBlobStoreContainer;
@@ -24,7 +26,7 @@ import org.apache.jackrabbit.oak.upgrade
import
org.apache.jackrabbit.oak.upgrade.cli.container.SegmentNodeStoreContainer;
import org.junit.Assume;
-public class FbsToS3Test extends AbstractOak2OakTest {
+public class S3ToFbsTest extends AbstractOak2OakTest {
private static final String S3_PROPERTIES =
System.getProperty("s3.properties");
@@ -36,10 +38,10 @@ public class FbsToS3Test extends Abstrac
private final NodeStoreContainer destination;
- public FbsToS3Test() {
+ public S3ToFbsTest() throws IOException {
Assume.assumeTrue(S3_PROPERTIES != null);
- sourceBlob = new FileBlobStoreContainer();
- destinationBlob = new S3DataStoreContainer(S3_PROPERTIES);
+ sourceBlob = new S3DataStoreContainer(S3_PROPERTIES);
+ destinationBlob = new FileBlobStoreContainer();
source = new SegmentNodeStoreContainer(sourceBlob);
destination = new SegmentNodeStoreContainer(destinationBlob);
}
@@ -56,8 +58,8 @@ public class FbsToS3Test extends Abstrac
@Override
protected String[] getArgs() {
- return new String[] { "--copy-binaries", "--src-fileblobstore",
sourceBlob.getDescription(), "--s3datastore",
- destinationBlob.getDescription(), "--s3config", S3_PROPERTIES,
source.getDescription(),
+ return new String[] { "--copy-binaries", "--src-s3datastore",
sourceBlob.getDescription(), "--src-s3config",
+ S3_PROPERTIES, "--fileblobstore",
destinationBlob.getDescription(), source.getDescription(),
destination.getDescription() };
}
}
Copied:
jackrabbit/oak/trunk/oak-upgrade/src/test/java/org/apache/jackrabbit/oak/upgrade/cli/blob/StripConfigValueTest.java
(from r1717263,
jackrabbit/oak/trunk/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/blob/FileBlobStoreFactory.java)
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-upgrade/src/test/java/org/apache/jackrabbit/oak/upgrade/cli/blob/StripConfigValueTest.java?p2=jackrabbit/oak/trunk/oak-upgrade/src/test/java/org/apache/jackrabbit/oak/upgrade/cli/blob/StripConfigValueTest.java&p1=jackrabbit/oak/trunk/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/blob/FileBlobStoreFactory.java&r1=1717263&r2=1717276&rev=1717276&view=diff
==============================================================================
---
jackrabbit/oak/trunk/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/blob/FileBlobStoreFactory.java
(original)
+++
jackrabbit/oak/trunk/oak-upgrade/src/test/java/org/apache/jackrabbit/oak/upgrade/cli/blob/StripConfigValueTest.java
Mon Nov 30 15:30:52 2015
@@ -16,22 +16,30 @@
*/
package org.apache.jackrabbit.oak.upgrade.cli.blob;
-import org.apache.jackrabbit.oak.spi.blob.BlobStore;
-import org.apache.jackrabbit.oak.spi.blob.FileBlobStore;
+import static
org.apache.jackrabbit.oak.upgrade.cli.blob.S3DataStoreFactory.stripValue;
+import static org.junit.Assert.assertEquals;
-import com.google.common.io.Closer;
+import org.junit.Test;
-public class FileBlobStoreFactory implements BlobStoreFactory {
+public class StripConfigValueTest {
- private final String directory;
+ @Test
+ public void testStripPrefixAndQuotes() {
+ assertEquals("123aaa", stripValue("B\"123aaa\""));
+ }
- public FileBlobStoreFactory(String directory) {
- this.directory = directory;
+ @Test
+ public void testStripPrefixQuotesAndWhitespaces() {
+ assertEquals("123aaa", stripValue("B\"123aaa\" "));
}
- @Override
- public BlobStore create(Closer closer) {
- return new FileBlobStore(directory);
+ @Test
+ public void testStripQuotes() {
+ assertEquals("123aaa", stripValue("\"123aaa\""));
}
+ @Test
+ public void testNoStrip() {
+ assertEquals("123aaa", stripValue("123aaa"));
+ }
}
Modified:
jackrabbit/oak/trunk/oak-upgrade/src/test/java/org/apache/jackrabbit/oak/upgrade/cli/container/S3DataStoreContainer.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-upgrade/src/test/java/org/apache/jackrabbit/oak/upgrade/cli/container/S3DataStoreContainer.java?rev=1717276&r1=1717275&r2=1717276&view=diff
==============================================================================
---
jackrabbit/oak/trunk/oak-upgrade/src/test/java/org/apache/jackrabbit/oak/upgrade/cli/container/S3DataStoreContainer.java
(original)
+++
jackrabbit/oak/trunk/oak-upgrade/src/test/java/org/apache/jackrabbit/oak/upgrade/cli/container/S3DataStoreContainer.java
Mon Nov 30 15:30:52 2015
@@ -17,76 +17,55 @@
package org.apache.jackrabbit.oak.upgrade.cli.container;
import java.io.File;
-import java.io.FileReader;
import java.io.IOException;
-import java.util.Properties;
-
-import javax.jcr.RepositoryException;
import org.apache.commons.io.FileUtils;
-import org.apache.commons.io.IOUtils;
-import org.apache.jackrabbit.core.data.DataStoreException;
-import org.apache.jackrabbit.oak.blob.cloud.aws.s3.S3DataStore;
-import org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore;
import org.apache.jackrabbit.oak.spi.blob.BlobStore;
+import org.apache.jackrabbit.oak.upgrade.cli.blob.S3DataStoreFactory;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import com.google.common.io.Closer;
import com.google.common.io.Files;
public class S3DataStoreContainer implements BlobStoreContainer {
private static final Logger log =
LoggerFactory.getLogger(S3DataStoreContainer.class);
- private final String configFile;
+ private final File directory;
- private final File homeDir;
+ private final S3DataStoreFactory factory;
- private S3DataStore s3DataStore;
+ private final Closer closer;
- public S3DataStoreContainer(String configFile) {
- this.configFile = configFile;
- this.homeDir = Files.createTempDir();
+ public S3DataStoreContainer(String configFile) throws IOException {
+ this.directory = Files.createTempDir();
+ this.factory = new S3DataStoreFactory(configFile, directory.getPath());
+ this.closer = Closer.create();
}
@Override
public BlobStore open() throws IOException {
- Properties props = new Properties();
- FileReader reader = new FileReader(new File(configFile));
- try {
- props.load(reader);
- } finally {
- IOUtils.closeQuietly(reader);
- }
- props.setProperty("path", new File(homeDir,
"repository/datastore").getPath());
-
- S3DataStore delegate = new S3DataStore();
- delegate.setProperties(props);
- try {
- delegate.init(homeDir.getPath());
- } catch (RepositoryException e) {
- throw new IOException(e);
- }
- return new DataStoreBlobStore(delegate);
+ return factory.create(closer);
}
@Override
public void close() {
try {
- s3DataStore.close();
- } catch (DataStoreException e) {
- log.error("Can't close s3datastore", e);
+ closer.close();
+ } catch (IOException e) {
+ log.error("Can't close store", e);
}
}
@Override
public void clean() throws IOException {
- FileUtils.deleteDirectory(homeDir);
+ FileUtils.deleteDirectory(directory);
}
@Override
public String getDescription() {
- return homeDir.getPath();
+ return directory.getPath();
}
}
\ No newline at end of file
