This is an automated email from the ASF dual-hosted git repository. spmallette pushed a commit to branch tinkergraph-storage in repository https://gitbox.apache.org/repos/asf/tinkerpop.git
commit 991bf87eca2576d5356d3f6a7da0e528f6cb7c8f Author: Stephen Mallette <[email protected]> AuthorDate: Tue Jul 14 21:44:15 2026 +0000 Add pluggable storage layer to TinkerStorageGraph TinkerStorageGraph can now durably persist committed transactions to disk through a pluggable storage engine, selected with the new gremlin.tinkergraph.storage config key with graphLocation as the storage directory. A GraphBinary write-ahead-log engine ships as the reference implementation; custom engines implement the TinkerStorage SPI. TinkerMemoryGraph is now purely in-memory: the old load-on-open / save-on-close behavior and the graphFormat key are removed. Use TinkerStorageGraph for durability or g.io() for interchange. SimpleAuthenticator now reads its credential store explicitly since the in-memory graph no longer auto-loads. Assisted-by: Claude Code:claude-opus-4-8 --- CHANGELOG.asciidoc | 2 + .../reference/implementations-tinkergraph.asciidoc | 91 +++-- docs/src/upgrade/release-4.x.x.asciidoc | 50 +++ gremlin-console/conf/tinkergraph-gryo.properties | 6 +- .../gremlin/server/auth/SimpleAuthenticator.java | 47 +++ .../tinkergraph/structure/AbstractTinkerGraph.java | 100 +++--- .../gremlin/tinkergraph/structure/TinkerGraph.java | 14 +- .../tinkergraph/structure/TinkerMemoryGraph.java | 20 +- .../tinkergraph/structure/TinkerStorageGraph.java | 36 +- .../tinkergraph/structure/TinkerTransaction.java | 39 +++ .../structure/storage/ByteBufferBuffer.java | 336 ++++++++++++++++++ .../structure/storage/DefaultStorage.java | 40 +++ .../structure/storage/GraphBinaryStorage.java | 379 +++++++++++++++++++++ .../structure/storage/TinkerStorage.java | 100 ++++++ .../structure/storage/TinkerStorageMutation.java | 67 ++++ .../tinkergraph/TinkerMemoryGraphProvider.java | 20 -- .../tinkergraph/TinkerMemoryGraphUUIDProvider.java | 6 - .../tinkergraph/TinkerStorageGraphProvider.java | 23 +- .../structure/TinkerMemoryGraphTest.java | 179 ---------- .../AbstractTinkerStorageConformanceTest.java | 242 +++++++++++++ .../structure/storage/GraphBinaryStorageTest.java | 92 +++++ 21 files changed, 1577 insertions(+), 312 deletions(-) diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc index 71ac4afe03..7d3155ec79 100644 --- a/CHANGELOG.asciidoc +++ b/CHANGELOG.asciidoc @@ -44,6 +44,8 @@ image::https://raw.githubusercontent.com/apache/tinkerpop/master/docs/static/ima * Fixed `gremlin-python` read timeout to derive from a single source (aiohttp `sock_read`), removing a redundant `async_timeout` read wrapper that could race it; a read timeout now deterministically raises `ReadTimeoutError` (a builtin `TimeoutError` subclass). * Made `TinkerGraph` an interface and renamed the in-memory implementation to `TinkerMemoryGraph`; `TinkerGraph.open()` and `gremlin.graph=...TinkerGraph` behave as before. *(breaking)* * Renamed `TinkerTransactionGraph` to `TinkerStorageGraph`. *(breaking)* +* Added a pluggable storage layer to `TinkerStorageGraph` that durably persists each committed transaction to disk, selected with the `gremlin.tinkergraph.storage` config key and shipping a GraphBinary engine. +* Removed automatic persistence from `TinkerMemoryGraph`, which is now purely in-memory and ignores `gremlin.tinkergraph.graphLocation`/`graphFormat`. Use `TinkerStorageGraph` for durability or `g.io()` for interchange. *(breaking)* * Removed `Transaction.open()` in favor of `begin()`, which is now the single transaction-start primitive across embedded and remote contexts. * Changed `begin()` and `close()` to be idempotent and calling it when a transaction is already in that state no longer throws. * Added `maxTransactionLifetime` setting to Gremlin Server, an absolute cap on the total age of an HTTP transaction that interrupts a running operation and rolls the transaction back when it fires (default 600000ms, set to `0` to disable). diff --git a/docs/src/reference/implementations-tinkergraph.asciidoc b/docs/src/reference/implementations-tinkergraph.asciidoc index 2dd96470c1..a7a6d2625a 100644 --- a/docs/src/reference/implementations-tinkergraph.asciidoc +++ b/docs/src/reference/implementations-tinkergraph.asciidoc @@ -39,13 +39,13 @@ under the License. </dependency> ---- -image:tinkerpop-character.png[width=100,float=left] TinkerGraph is a single machine, in-memory (with optional -persistence), graph engine that provides both OLTP and OLAP functionality. It is non-transactional by default but does +image:tinkerpop-character.png[width=100,float=left] TinkerGraph is a single machine, in-memory graph engine that +provides both OLTP and OLAP functionality. It is non-transactional by default but does have a lightweight transactional form that can be instantiated offering simple `ThreadLocal` transactions supporting -`read committed` transaction isolation. As of 4.0.0, `TinkerGraph` is an interface with two implementations: -`TinkerMemoryGraph`, the in-memory, non-transactional implementation that `TinkerGraph.open()` constructs, and -`TinkerStorageGraph`, the transactional implementation formerly named `TinkerTransactionGraph`. TinkerGraph is -deployed with TinkerPop and serves as the reference +`read committed` transaction isolation, with optional durable persistence to disk. As of 4.0.0, `TinkerGraph` is an +interface with two implementations: `TinkerMemoryGraph`, the in-memory, non-transactional implementation that +`TinkerGraph.open()` constructs, and `TinkerStorageGraph`, the transactional implementation formerly named +`TinkerTransactionGraph`. TinkerGraph is deployed with TinkerPop and serves as the reference implementation for other providers to study in order to understand the semantics of the various methods of the TinkerPop API. Its status as a reference implementation does not however imply that it is not suitable for production. TinkerGraph has many practical use cases in production applications and their development. Some examples of TinkerGraph @@ -171,19 +171,16 @@ TinkerGraph has several settings that can be provided on creation via `Configura |gremlin.tinkergraph.vertexPropertyIdManager |The `IdManager` implementation to use for vertex properties. |gremlin.tinkergraph.defaultVertexPropertyCardinality |The default `VertexProperty.Cardinality` to use when `Vertex.property(k,v)` is called. |gremlin.tinkergraph.allowNullPropertyValues |A boolean value that determines whether or not `null` property values are allowed and defaults to `false`. -|gremlin.tinkergraph.graphLocation |The path and file name for where TinkerGraph should persist the graph data. If a -value is specified here, the `gremlin.tinkergraph.graphFormat` should also be specified. If this value is not -included (default), then the graph will stay in-memory and not be loaded/persisted to disk. -|gremlin.tinkergraph.graphFormat |The format to use to serialize the graph which may be one of the following: -`graphml`, `graphson`, `gryo`, or a fully qualified class name that implements Io.Builder interface (which allows for -external third party graph reader/writer formats to be used for persistence). -If a value is specified here, then the `gremlin.tinkergraph.graphLocation` should -also be specified. If this value is not included (default), then the graph will stay in-memory and not be -loaded/persisted to disk. +|gremlin.tinkergraph.storage |The durable storage engine used by `TinkerStorageGraph` to persist committed transactions +to disk. The value is either a built-in engine name (`graphbinary`) or a fully qualified class name of a +`TinkerStorage` implementation. When not specified (default), the graph holds data only in memory. This setting is +only valid on `TinkerStorageGraph` and is ignored by the in-memory `TinkerMemoryGraph`. +|gremlin.tinkergraph.graphLocation |The directory in which `TinkerStorageGraph` stores its durable data. Required when +`gremlin.tinkergraph.storage` is set and ignored otherwise. |========================================================= -NOTE: To use <<tinkergraph-gremlin-tx, transactions>>, configure `gremlin.graph` as -`org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph`. +NOTE: To use <<tinkergraph-gremlin-tx, transactions>> and <<tinkergraph-gremlin-persistence, persistence>>, configure +`gremlin.graph` as `org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph`. The `IdManager` settings above refer to how TinkerGraph will control identifiers for vertices, edges and vertex properties. There are several options for each of these settings: `ANY`, `LONG`, `INTEGER`, `UUID`, `STRING` or the @@ -196,15 +193,10 @@ type as well as generate new identifiers with that specified type. TIP: Setting the `IdManager` to `ANY` also allows `String` type ID values to be used. -If the TinkerGraph is configured for persistence with `gremlin.tinkergraph.graphLocation` and -`gremlin.tinkergraph.graphFormat`, then the graph will be written to the specified location with the specified -format when `Graph.close()` is called. In addition, if these settings are present, TinkerGraph will attempt to -load the graph from the specified location. - -IMPORTANT: If choosing `graphson` as the `gremlin.tinkergraph.graphFormat`, be sure to also establish the various -`IdManager` settings as well to ensure that identifiers are properly coerced to the appropriate types as GraphSON -can lose the identifier's type during serialization (i.e. it will assume `Integer` when the default for TinkerGraph -is `Long`, which could lead to load errors that result in a message like, "Vertex with id already exists"). +Durable persistence is provided by `TinkerStorageGraph` through its pluggable storage layer and is described in the +<<tinkergraph-gremlin-persistence, persistence section>>. The in-memory `TinkerMemoryGraph` holds no data across JVM +restarts. Moving data in and out of any TinkerGraph in an interchange format is handled on demand by the `io()` step +rather than by graph configuration, as shown below. It is important to consider the data being imported to TinkerGraph with respect to `defaultVertexPropertyCardinality` setting. For example, if a `.gryo` file is known to contain multi-property data, be sure to set the default @@ -379,6 +371,53 @@ g.V().valueMap() <4> Add a second vertex without committing <5> Rollback the change +[[tinkergraph-gremlin-persistence]] +=== Persistence + +`TinkerStorageGraph` can durably persist to disk through a pluggable storage layer built on its transaction support. +When a storage engine is configured, the changeset of each committed transaction is written to disk, and the graph is +rebuilt from that data when it is opened again. The in-memory `TinkerMemoryGraph` does not retain data across restarts. + +A storage engine is selected with the `gremlin.tinkergraph.storage` configuration key, and +`gremlin.tinkergraph.graphLocation` names the directory that holds the durable data. The value of the storage key is +either a built-in engine name or the fully qualified class name of a `TinkerStorage` implementation, following the same +enum-name-or-class-name convention as the `IdManager` settings. The built-in `graphbinary` engine records committed +transactions as an append-only log serialized with GraphBinary and folds that log into a compact snapshot when the +graph is closed. + +[source,groovy] +---- +conf = new BaseConfiguration() +conf.setProperty("gremlin.graph", "org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph") +conf.setProperty("gremlin.tinkergraph.storage", "graphbinary") +conf.setProperty("gremlin.tinkergraph.graphLocation", "/data/mygraph") + +graph = TinkerStorageGraph.open(conf) +g = traversal().with(graph) +g.addV("person").property("name","marko").iterate() +g.tx().commit() +graph.close() + +// reopening the same location restores the committed data +graph = TinkerStorageGraph.open(conf) +g = traversal().with(graph) +g.V().count() +==>1 +---- + +The graph remains the authoritative in-memory copy and is mirrored to disk, so a persisted graph must still fit in +memory. The write to the storage log happens before the in-memory commit is applied, so a failure to persist aborts the +transaction and leaves the on-disk data and the in-memory graph consistent. + +Persistence is distinct from interchange. `TinkerMemoryGraph` is purely in-memory and does not persist. To move data in +or out of any TinkerGraph in an interchange format such as GraphML, GraphSON, or Gryo, use the `io()` step directly: + +[source,groovy] +---- +g.io("/tmp/graph.kryo").write().iterate() +g.io("/tmp/graph.kryo").read().iterate() +---- + [[tinkergraph-gql]] === Declarative Pattern Matching (TinkerGQL) diff --git a/docs/src/upgrade/release-4.x.x.asciidoc b/docs/src/upgrade/release-4.x.x.asciidoc index c81b13f69a..1504614d15 100644 --- a/docs/src/upgrade/release-4.x.x.asciidoc +++ b/docs/src/upgrade/release-4.x.x.asciidoc @@ -103,6 +103,56 @@ which affects providers that extended them. See: link:https://lists.apache.org/thread/2zt62kvfssh6xz5vnf2lk1g7cstq9vod[DISCUSS thread] +==== TinkerStorageGraph Pluggable Disk Storage + +`TinkerStorageGraph` gained the optional disk storage anticipated by its rename. A storage engine is selected with the +new `gremlin.tinkergraph.storage` configuration key, and `gremlin.tinkergraph.graphLocation` names the directory that +holds the durable data. When a storage engine is configured, each committed transaction is durably written to disk and +the graph is rebuilt from that data when it is opened again, so a graph survives a restart of the JVM. + +The reference engine, `graphbinary`, records committed transactions as an append-only log serialized with GraphBinary +and folds that log into a compact snapshot on close. The storage layer is pluggable: the value of the storage key may +also be the fully-qualified class name of a custom engine, following the same convention as the `IdManager` selection. + +[source,groovy] +---- +conf = new BaseConfiguration() +conf.setProperty('gremlin.graph', 'org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph') +conf.setProperty('gremlin.tinkergraph.storage', 'graphbinary') +conf.setProperty('gremlin.tinkergraph.graphLocation', '/data/mygraph') + +graph = TinkerStorageGraph.open(conf) +g = traversal().with(graph) +g.addV('person').property('name','marko').iterate() +g.tx().commit() +graph.close() + +// reopening the same location restores the committed data +graph = TinkerStorageGraph.open(conf) +g = traversal().with(graph) +g.V().count().next() +==>1 +---- + +The in-memory `TinkerMemoryGraph` no longer persists to disk. Earlier versions of TinkerGraph would automatically read +from `gremlin.tinkergraph.graphLocation` on open and write back to it on close, using the `gremlin.tinkergraph.graphFormat` +interchange format. That automatic behavior is removed, and `TinkerMemoryGraph` now ignores both keys and reports +`FEATURE_PERSISTENCE` as `false`. Durable persistence is the responsibility of `TinkerStorageGraph` and its storage +engine, while moving data in and out of any graph in an interchange format remains the job of the `io()` step: + +[source,groovy] +---- +// interchange, on demand, for any graph +g.io('/tmp/graph.kryo').write().iterate() +g.io('/tmp/graph.kryo').read().iterate() +---- + +Configurations that previously relied on the in-memory graph loading itself from `graphLocation` on open must either +call `io().read()` explicitly or switch to `TinkerStorageGraph` with a storage engine. The `gremlin.tinkergraph.graphFormat` +key is retired. + +See: <<tinkergraph-gremlin,TinkerGraph>> + ==== Standardizing GLV Connection Options TinkerPop 4.x standardizes connection option names and defaults across all five Gremlin Language Variants (Java, Python, diff --git a/gremlin-console/conf/tinkergraph-gryo.properties b/gremlin-console/conf/tinkergraph-gryo.properties index 4c2684237c..312dc3f9b6 100644 --- a/gremlin-console/conf/tinkergraph-gryo.properties +++ b/gremlin-console/conf/tinkergraph-gryo.properties @@ -15,7 +15,7 @@ # specific language governing permissions and limitations # under the License. -gremlin.graph=org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph +gremlin.graph=org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph -gremlin.tinkergraph.graphFormat=gryo -gremlin.tinkergraph.graphLocation=/tmp/tinkergraph.kryo +gremlin.tinkergraph.storage=graphbinary +gremlin.tinkergraph.graphLocation=/tmp/tinkergraph diff --git a/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/auth/SimpleAuthenticator.java b/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/auth/SimpleAuthenticator.java index afdae0d2de..435316cd08 100644 --- a/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/auth/SimpleAuthenticator.java +++ b/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/auth/SimpleAuthenticator.java @@ -21,14 +21,17 @@ package org.apache.tinkerpop.gremlin.server.auth; import org.apache.tinkerpop.gremlin.groovy.jsr223.dsl.credential.CredentialTraversal; import org.apache.tinkerpop.gremlin.groovy.jsr223.dsl.credential.CredentialTraversalDsl; import org.apache.tinkerpop.gremlin.groovy.jsr223.dsl.credential.CredentialTraversalSource; +import org.apache.commons.configuration2.Configuration; import org.apache.tinkerpop.gremlin.structure.Graph; import org.apache.tinkerpop.gremlin.structure.Vertex; +import org.apache.tinkerpop.gremlin.structure.io.IoCore; import org.apache.tinkerpop.gremlin.structure.util.GraphFactory; import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph; import org.mindrot.jbcrypt.BCrypt; import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import java.io.File; import java.net.InetAddress; import java.nio.charset.StandardCharsets; import java.util.Arrays; @@ -80,12 +83,56 @@ public class SimpleAuthenticator implements Authenticator { // have to create the indices because they are not stored in gryo final TinkerGraph tinkerGraph = (TinkerGraph) graph; tinkerGraph.createIndex(PROPERTY_USERNAME, Vertex.class); + + // TinkerGraph no longer auto-loads from graphLocation on open, so read the credential store here from + // the configured location/format. TinkerStorageGraph (which persists via its own storage engine) manages + // its own data and is left untouched. + loadCredentialStore(tinkerGraph); } credentialStore = graph.traversal(CredentialTraversalSource.class); logger.info("CredentialGraph initialized at {}", credentialStore); } + /** + * Reads the credential store into the supplied in-memory {@link TinkerGraph} from the {@code graphLocation} + * declared in its configuration, if any. TinkerGraph no longer loads from disk automatically on open, so the + * credential file must be read explicitly here. A {@code TinkerStorageGraph} configured with a durable storage + * engine manages its own data and is skipped (it has no {@code graphFormat}). + */ + private static void loadCredentialStore(final TinkerGraph graph) { + final Configuration conf = graph.configuration(); + final String location = conf.getString(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, null); + // a storage engine manages its own persistence and is not an interchange-format load + final String storage = conf.getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE, null); + if (null == location || storage != null) + return; + + final File f = new File(location); + if (!f.exists() || !f.isFile()) + return; + + final String format = conf.getString("gremlin.tinkergraph.graphFormat", "gryo"); + try { + switch (format) { + case "graphml": + graph.io(IoCore.graphml()).readGraph(location); + break; + case "graphson": + graph.io(IoCore.graphson()).readGraph(location); + break; + case "gryo": + graph.io(IoCore.gryo()).readGraph(location); + break; + default: + graph.io(IoCore.createIoBuilder(format)).readGraph(location); + break; + } + } catch (Exception ex) { + throw new IllegalStateException(String.format("Could not load credential store at %s with format %s", location, format), ex); + } + } + @Override public SaslNegotiator newSaslNegotiator(final InetAddress remoteAddress) { return new PlainTextSaslAuthenticator(); diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/AbstractTinkerGraph.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/AbstractTinkerGraph.java index f9845fd5d9..9e82e84d89 100644 --- a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/AbstractTinkerGraph.java +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/AbstractTinkerGraph.java @@ -27,7 +27,6 @@ import org.apache.tinkerpop.gremlin.structure.Transaction; import org.apache.tinkerpop.gremlin.structure.Vertex; import org.apache.tinkerpop.gremlin.structure.VertexProperty; import org.apache.tinkerpop.gremlin.structure.io.Io; -import org.apache.tinkerpop.gremlin.structure.io.IoCore; import org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONVersion; import org.apache.tinkerpop.gremlin.structure.io.gryo.GryoVersion; import org.apache.tinkerpop.gremlin.structure.util.StringFactory; @@ -35,8 +34,9 @@ import org.apache.tinkerpop.gremlin.tinkergraph.process.computer.TinkerGraphComp import org.apache.tinkerpop.gremlin.tinkergraph.process.computer.TinkerGraphComputerView; import org.apache.tinkerpop.gremlin.gql.GqlDeclarativeMatchStrategy; import org.apache.tinkerpop.gremlin.tinkergraph.services.TinkerServiceRegistry; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.storage.DefaultStorage; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.storage.TinkerStorage; -import java.io.File; import java.lang.reflect.InvocationTargetException; import java.util.Collections; import java.util.Iterator; @@ -72,7 +72,18 @@ public abstract class AbstractTinkerGraph implements TinkerGraph { protected Configuration configuration; protected String graphLocation; - protected String graphFormat; + + /** + * The pluggable durable storage engine, or {@code null} when the graph holds data only in memory. Only set by + * transactional implementations that support persistence. + */ + protected TinkerStorage storage; + + /** + * Guard set while a graph is replaying its storage log on open. While {@code true}, mutations must not be + * re-persisted, otherwise replay would append the loaded data back to the log. + */ + protected volatile boolean loading = false; /** * {@inheritDoc} @@ -238,54 +249,6 @@ public abstract class AbstractTinkerGraph implements TinkerGraph { return vertexProperties.containsKey(id); } - protected void loadGraph() { - final File f = new File(graphLocation); - if (f.exists() && f.isFile()) { - try { - if (graphFormat.equals("graphml")) { - io(IoCore.graphml()).readGraph(graphLocation); - } else if (graphFormat.equals("graphson")) { - io(IoCore.graphson()).readGraph(graphLocation); - } else if (graphFormat.equals("gryo")) { - io(IoCore.gryo()).readGraph(graphLocation); - } else { - io(IoCore.createIoBuilder(graphFormat)).readGraph(graphLocation); - } - } catch (Exception ex) { - throw new RuntimeException(String.format("Could not load graph at %s with %s", graphLocation, graphFormat), ex); - } - } - } - - protected void saveGraph() { - final File f = new File(graphLocation); - if (f.exists()) { - f.delete(); - } else { - final File parent = f.getParentFile(); - - // the parent would be null in the case of an relative path if the graphLocation was simply: "f.gryo" - if (parent != null && !parent.exists()) { - parent.mkdirs(); - } - } - - try { - if (graphFormat.equals("graphml")) { - io(IoCore.graphml()).writeGraph(graphLocation); - } else if (graphFormat.equals("graphson")) { - io(IoCore.graphson()).writeGraph(graphLocation); - } else if (graphFormat.equals("gryo")) { - io(IoCore.gryo()).writeGraph(graphLocation); - } else { - io(IoCore.createIoBuilder(graphFormat)).writeGraph(graphLocation); - } - } catch (Exception ex) { - throw new RuntimeException(String.format("Could not save graph at %s with %s", graphLocation, graphFormat), ex); - } - } - - @Override public <I extends Io> I io(final Io.Builder<I> builder) { if (builder.requiresVersion(GryoVersion.V1_0) || builder.requiresVersion(GraphSONVersion.V1_0)) @@ -332,13 +295,18 @@ public abstract class AbstractTinkerGraph implements TinkerGraph { } /** - * This method only has an effect if the {@link TinkerGraph#GREMLIN_TINKERGRAPH_GRAPH_LOCATION} is set, in which case the - * data in the graph is persisted to that location. This method may be called multiple times and does not release - * resources. + * Closes the graph, releasing any resources held by its {@link TinkerServiceRegistry}. This method may be called + * multiple times and is a no-op with respect to graph data for the in-memory implementation. Transactional + * implementations that are backed by a {@link org.apache.tinkerpop.gremlin.tinkergraph.structure.storage.TinkerStorage} + * engine flush and close that engine here. */ @Override public void close() { - if (graphLocation != null) saveGraph(); + if (storage != null) { + storage.flush(); + storage.compact(this); + storage.close(); + } serviceRegistry.close(); GqlDeclarativeMatchStrategy.evict(this); } @@ -483,6 +451,28 @@ public abstract class AbstractTinkerGraph implements TinkerGraph { } } + ///////////// Storage engine /////////////// + /** + * Construct a {@link TinkerStorage} engine from the TinkerGraph {@code Configuration}, or return {@code null} when + * no storage engine is configured. The configuration value is either a {@link DefaultStorage} enum name (matched + * case-insensitively, e.g. {@code graphbinary}) or the fully-qualified class name of a {@link TinkerStorage} + * implementation with a public no-argument constructor. Mirrors {@link #selectIdManager}. + */ + protected static TinkerStorage selectStorage(final Configuration config, final String configKey) { + final String storageConfigValue = config.getString(configKey, null); + if (null == storageConfigValue) + return null; + try { + return DefaultStorage.valueOf(storageConfigValue.toUpperCase()).get(); + } catch (IllegalArgumentException iae) { + try { + return (TinkerStorage) Class.forName(storageConfigValue).newInstance(); + } catch (Exception ex) { + throw new IllegalStateException(String.format("Could not configure TinkerGraph storage engine with %s", storageConfigValue), ex); + } + } + } + protected TinkerServiceRegistry.TinkerServiceFactory instantiate(final String className) { try { return (TinkerServiceRegistry.TinkerServiceFactory) Class.forName(className).getConstructor(AbstractTinkerGraph.class).newInstance(this); diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerGraph.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerGraph.java index 78a49a1667..c2632c7e8d 100644 --- a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerGraph.java +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerGraph.java @@ -45,8 +45,20 @@ public interface TinkerGraph extends Graph { String GREMLIN_TINKERGRAPH_EDGE_ID_MANAGER = "gremlin.tinkergraph.edgeIdManager"; String GREMLIN_TINKERGRAPH_VERTEX_PROPERTY_ID_MANAGER = "gremlin.tinkergraph.vertexPropertyIdManager"; String GREMLIN_TINKERGRAPH_DEFAULT_VERTEX_PROPERTY_CARDINALITY = "gremlin.tinkergraph.defaultVertexPropertyCardinality"; + /** + * The filesystem directory that a {@link TinkerStorageGraph} uses for its durable storage engine. Ignored by + * {@link TinkerMemoryGraph}, which is purely in-memory. Only meaningful when {@link #GREMLIN_TINKERGRAPH_STORAGE} + * is also set. + */ String GREMLIN_TINKERGRAPH_GRAPH_LOCATION = "gremlin.tinkergraph.graphLocation"; - String GREMLIN_TINKERGRAPH_GRAPH_FORMAT = "gremlin.tinkergraph.graphFormat"; + /** + * Selects the pluggable storage engine used by {@link TinkerStorageGraph} to durably persist transactions to the + * {@link #GREMLIN_TINKERGRAPH_GRAPH_LOCATION} directory. The value is either a + * {@code TinkerStorageGraph.DefaultStorage} enum name (e.g. {@code graphbinary}) or the fully-qualified class name + * of a {@code org.apache.tinkerpop.gremlin.tinkergraph.structure.storage.TinkerStorage} implementation. When unset, + * the graph holds data only in memory. Not valid on {@link TinkerMemoryGraph}. + */ + String GREMLIN_TINKERGRAPH_STORAGE = "gremlin.tinkergraph.storage"; String GREMLIN_TINKERGRAPH_ALLOW_NULL_PROPERTY_VALUES = "gremlin.tinkergraph.allowNullPropertyValues"; String GREMLIN_TINKERGRAPH_SERVICE = "gremlin.tinkergraph.service"; diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerMemoryGraph.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerMemoryGraph.java index 2f7bd6ff70..7667c2d76b 100644 --- a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerMemoryGraph.java +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerMemoryGraph.java @@ -49,8 +49,10 @@ import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.atomic.AtomicInteger; /** - * The in-memory (with optional persistence on calls to {@link #close()}) implementation of the {@link TinkerGraph} - * interface and the reference implementation of the property graph interfaces provided by TinkerPop. + * The purely in-memory implementation of the {@link TinkerGraph} interface and the reference implementation of the + * property graph interfaces provided by TinkerPop. This graph holds no data across JVM restarts; use + * {@code g.io(...).write()} / {@code g.io(...).read()} for interchange, or {@link TinkerStorageGraph} for durable + * persistence. * * @author Marko A. Rodriguez (http://markorodriguez.com) * @author Stephen Mallette (http://stephen.genoprime.com) @@ -92,15 +94,6 @@ public class TinkerMemoryGraph extends AbstractTinkerGraph { configuration.getString(GREMLIN_TINKERGRAPH_DEFAULT_VERTEX_PROPERTY_CARDINALITY, VertexProperty.Cardinality.single.name())); allowNullPropertyValues = configuration.getBoolean(GREMLIN_TINKERGRAPH_ALLOW_NULL_PROPERTY_VALUES, false); - graphLocation = configuration.getString(GREMLIN_TINKERGRAPH_GRAPH_LOCATION, null); - graphFormat = configuration.getString(GREMLIN_TINKERGRAPH_GRAPH_FORMAT, null); - - if ((graphLocation != null && null == graphFormat) || (null == graphLocation && graphFormat != null)) - throw new IllegalStateException(String.format("The %s and %s must both be specified if either is present", - GREMLIN_TINKERGRAPH_GRAPH_LOCATION, GREMLIN_TINKERGRAPH_GRAPH_FORMAT)); - - if (graphLocation != null) loadGraph(); - serviceRegistry = new TinkerServiceRegistry(this); configuration.getList(String.class, GREMLIN_TINKERGRAPH_SERVICE, Collections.emptyList()).forEach(serviceClass -> serviceRegistry.registerService(instantiate(serviceClass))); @@ -391,6 +384,11 @@ public class TinkerMemoryGraph extends AbstractTinkerGraph { return false; } + @Override + public boolean supportsPersistence() { + return false; + } + @Override public boolean supportsServiceCall() { return true; diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerStorageGraph.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerStorageGraph.java index 4474bf69f3..eb6c9c5b82 100644 --- a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerStorageGraph.java +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerStorageGraph.java @@ -89,17 +89,25 @@ public final class TinkerStorageGraph extends AbstractTinkerGraph { allowNullPropertyValues = configuration.getBoolean(GREMLIN_TINKERGRAPH_ALLOW_NULL_PROPERTY_VALUES, false); graphLocation = configuration.getString(GREMLIN_TINKERGRAPH_GRAPH_LOCATION, null); - graphFormat = configuration.getString(GREMLIN_TINKERGRAPH_GRAPH_FORMAT, null); + storage = selectStorage(configuration, GREMLIN_TINKERGRAPH_STORAGE); - if ((graphLocation != null && null == graphFormat) || (null == graphLocation && graphFormat != null)) - throw new IllegalStateException(String.format("The %s and %s must both be specified if either is present", - GREMLIN_TINKERGRAPH_GRAPH_LOCATION, GREMLIN_TINKERGRAPH_GRAPH_FORMAT)); - - if (graphLocation != null) loadGraph(); + if (storage != null && null == graphLocation) + throw new IllegalStateException(String.format("The %s must be specified when %s is set", + GREMLIN_TINKERGRAPH_GRAPH_LOCATION, GREMLIN_TINKERGRAPH_STORAGE)); serviceRegistry = new TinkerServiceRegistry(this); configuration.getList(String.class, GREMLIN_TINKERGRAPH_SERVICE, Collections.emptyList()).forEach(serviceClass -> serviceRegistry.registerService(instantiate(serviceClass))); + + if (storage != null) { + storage.open(this, configuration); + loading = true; + try { + storage.replay(this); + } finally { + loading = false; + } + } } /** @@ -282,6 +290,17 @@ public final class TinkerStorageGraph extends AbstractTinkerGraph { this.edges.clear(); } + /** + * Fold the durable storage log into a compact snapshot of the current committed state, reclaiming space. Has no + * effect when no storage engine is configured. + */ + public void compact() { + if (storage != null) { + storage.flush(); + storage.compact(this); + } + } + @Override public Transaction tx() { return transaction; @@ -478,6 +497,11 @@ public final class TinkerStorageGraph extends AbstractTinkerGraph { return true; } + @Override + public boolean supportsPersistence() { + return storage != null; + } + @Override public boolean supportsServiceCall() { return true; diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerTransaction.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerTransaction.java index 0bfcf09e46..7bef8a9264 100644 --- a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerTransaction.java +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerTransaction.java @@ -21,9 +21,12 @@ package org.apache.tinkerpop.gremlin.tinkergraph.structure; import org.apache.tinkerpop.gremlin.structure.Transaction; import org.apache.tinkerpop.gremlin.structure.util.AbstractThreadLocalTransaction; import org.apache.tinkerpop.gremlin.structure.util.TransactionException; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.storage.TinkerStorageMutation; +import java.util.ArrayList; import java.util.Collections; import java.util.HashSet; +import java.util.List; import java.util.Set; import java.util.concurrent.atomic.AtomicLong; @@ -174,6 +177,14 @@ final class TinkerTransaction extends AbstractThreadLocalTransaction { final TinkerTransactionalIndex edgeIndex = (TinkerTransactionalIndex) graph.edgeIndex; if (edgeIndex != null) edgeIndex.commit(changedEdges); + // write-ahead: durably persist the changeset before applying the in-memory commit, so a failure here + // aborts the commit (via the catch below) and leaves memory and disk consistent. Skipped while the graph + // is replaying its storage log on open. + if (graph.storage != null && !graph.loading) { + graph.storage.persist(txVersion, toVertexMutations(changedVertices), toEdgeMutations(changedEdges)); + graph.storage.flush(); + } + // commit all changes changedVertices.forEach(v -> v.commit(txVersion)); changedEdges.forEach(e -> e.commit(txVersion)); @@ -209,6 +220,34 @@ final class TinkerTransaction extends AbstractThreadLocalTransaction { } } + /** + * Convert the changed vertex containers into the storage-facing {@link TinkerStorageMutation} view. Called during + * commit, before {@code commit()} is applied to the containers, so the modified value is still available via + * {@link TinkerElementContainer#getModified()}. + */ + private static List<TinkerStorageMutation<TinkerVertex>> toVertexMutations(final Set<TinkerElementContainer<TinkerVertex>> changed) { + final List<TinkerStorageMutation<TinkerVertex>> mutations = new ArrayList<>(changed.size()); + for (final TinkerElementContainer<TinkerVertex> c : changed) { + mutations.add(c.isDeleted() + ? new TinkerStorageMutation<>(c.getElementId(), null) + : new TinkerStorageMutation<>(c.getElementId(), c.getModified())); + } + return mutations; + } + + /** + * Convert the changed edge containers into the storage-facing {@link TinkerStorageMutation} view. + */ + private static List<TinkerStorageMutation<TinkerEdge>> toEdgeMutations(final Set<TinkerElementContainer<TinkerEdge>> changed) { + final List<TinkerStorageMutation<TinkerEdge>> mutations = new ArrayList<>(changed.size()); + for (final TinkerElementContainer<TinkerEdge> c : changed) { + mutations.add(c.isDeleted() + ? new TinkerStorageMutation<>(c.getElementId(), null) + : new TinkerStorageMutation<>(c.getElementId(), c.getModified())); + } + return mutations; + } + /** * Rollback all changes made in current transaction. * Workflow: diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/ByteBufferBuffer.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/ByteBufferBuffer.java new file mode 100644 index 0000000000..8872afd404 --- /dev/null +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/ByteBufferBuffer.java @@ -0,0 +1,336 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.apache.tinkerpop.gremlin.structure.io.Buffer; + +import java.io.IOException; +import java.io.OutputStream; +import java.nio.ByteBuffer; +import java.util.Arrays; + +/** + * A self-contained, heap {@code byte[]}-backed implementation of the gremlin-core {@link Buffer} abstraction, used to + * drive {@code GraphBinaryWriter}/{@code GraphBinaryReader} without depending on gremlin-util's Netty-backed buffer. + * All multi-byte values are big-endian, matching the wire order the Netty implementation uses. The backing array grows + * as needed on write. This class is not thread-safe; a buffer is used by a single thread for a single + * serialize/deserialize. + */ +public final class ByteBufferBuffer implements Buffer { + + private static final int DEFAULT_CAPACITY = 256; + + private byte[] array; + private int readerIndex = 0; + private int writerIndex = 0; + private int markedWriterIndex = 0; + private int referenceCount = 1; + + public ByteBufferBuffer() { + this(DEFAULT_CAPACITY); + } + + public ByteBufferBuffer(final int initialCapacity) { + this.array = new byte[Math.max(initialCapacity, 1)]; + } + + /** + * Wraps an existing array for reading. The writer index is positioned at the end of the supplied data. + */ + public ByteBufferBuffer(final byte[] data) { + this.array = data; + this.writerIndex = data.length; + } + + /** + * Returns a copy of the readable-region bytes (from reader index to writer index). Does not change indexes. + */ + public byte[] toReadableArray() { + return Arrays.copyOfRange(array, readerIndex, writerIndex); + } + + /** + * Returns a copy of all written bytes (index 0 to writer index). Does not change indexes. + */ + public byte[] toWrittenArray() { + return Arrays.copyOfRange(array, 0, writerIndex); + } + + private void ensureWritable(final int additional) { + final int required = writerIndex + additional; + if (required <= array.length) + return; + int newCapacity = array.length; + while (newCapacity < required) + newCapacity <<= 1; + array = Arrays.copyOf(array, newCapacity); + } + + private void checkReadable(final int length) { + if (readerIndex + length > writerIndex) + throw new IndexOutOfBoundsException(String.format( + "Not enough readable bytes: need %d at index %d but writer index is %d", length, readerIndex, writerIndex)); + } + + @Override + public int readableBytes() { + return writerIndex - readerIndex; + } + + @Override + public int readerIndex() { + return readerIndex; + } + + @Override + public Buffer readerIndex(final int readerIndex) { + if (readerIndex < 0 || readerIndex > writerIndex) + throw new IndexOutOfBoundsException("readerIndex: " + readerIndex); + this.readerIndex = readerIndex; + return this; + } + + @Override + public int writerIndex() { + return writerIndex; + } + + @Override + public Buffer writerIndex(final int writerIndex) { + if (writerIndex < readerIndex) + throw new IndexOutOfBoundsException("writerIndex: " + writerIndex); + ensureWritable(writerIndex - this.writerIndex); + this.writerIndex = writerIndex; + return this; + } + + @Override + public Buffer markWriterIndex() { + this.markedWriterIndex = writerIndex; + return this; + } + + @Override + public Buffer resetWriterIndex() { + this.writerIndex = markedWriterIndex; + return this; + } + + @Override + public int capacity() { + return array.length; + } + + @Override + public boolean isDirect() { + return false; + } + + @Override + public boolean readBoolean() { + return readByte() != 0; + } + + @Override + public byte readByte() { + checkReadable(1); + return array[readerIndex++]; + } + + @Override + public short readShort() { + checkReadable(2); + return (short) (((array[readerIndex++] & 0xFF) << 8) | (array[readerIndex++] & 0xFF)); + } + + @Override + public int readInt() { + checkReadable(4); + return ((array[readerIndex++] & 0xFF) << 24) | + ((array[readerIndex++] & 0xFF) << 16) | + ((array[readerIndex++] & 0xFF) << 8) | + (array[readerIndex++] & 0xFF); + } + + @Override + public long readLong() { + checkReadable(8); + long value = 0; + for (int i = 0; i < 8; i++) + value = (value << 8) | (array[readerIndex++] & 0xFF); + return value; + } + + @Override + public float readFloat() { + return Float.intBitsToFloat(readInt()); + } + + @Override + public double readDouble() { + return Double.longBitsToDouble(readLong()); + } + + @Override + public Buffer readBytes(final byte[] destination) { + return readBytes(destination, 0, destination.length); + } + + @Override + public Buffer readBytes(final byte[] destination, final int dstIndex, final int length) { + checkReadable(length); + System.arraycopy(array, readerIndex, destination, dstIndex, length); + readerIndex += length; + return this; + } + + @Override + public Buffer readBytes(final ByteBuffer dst) { + final int length = dst.remaining(); + checkReadable(length); + dst.put(array, readerIndex, length); + readerIndex += length; + return this; + } + + @Override + public Buffer readBytes(final OutputStream out, final int length) throws IOException { + checkReadable(length); + out.write(array, readerIndex, length); + readerIndex += length; + return this; + } + + @Override + public Buffer writeBoolean(final boolean value) { + return writeByte(value ? 1 : 0); + } + + @Override + public Buffer writeByte(final int value) { + ensureWritable(1); + array[writerIndex++] = (byte) value; + return this; + } + + @Override + public Buffer writeShort(final int value) { + ensureWritable(2); + array[writerIndex++] = (byte) (value >>> 8); + array[writerIndex++] = (byte) value; + return this; + } + + @Override + public Buffer writeInt(final int value) { + ensureWritable(4); + array[writerIndex++] = (byte) (value >>> 24); + array[writerIndex++] = (byte) (value >>> 16); + array[writerIndex++] = (byte) (value >>> 8); + array[writerIndex++] = (byte) value; + return this; + } + + @Override + public Buffer writeLong(final long value) { + ensureWritable(8); + for (int i = 56; i >= 0; i -= 8) + array[writerIndex++] = (byte) (value >>> i); + return this; + } + + @Override + public Buffer writeFloat(final float value) { + return writeInt(Float.floatToIntBits(value)); + } + + @Override + public Buffer writeDouble(final double value) { + return writeLong(Double.doubleToLongBits(value)); + } + + @Override + public Buffer writeBytes(final byte[] src) { + return writeBytes(src, 0, src.length); + } + + @Override + public Buffer writeBytes(final ByteBuffer src) { + final int length = src.remaining(); + ensureWritable(length); + src.get(array, writerIndex, length); + writerIndex += length; + return this; + } + + @Override + public Buffer writeBytes(final byte[] src, final int srcIndex, final int length) { + ensureWritable(length); + System.arraycopy(src, srcIndex, array, writerIndex, length); + writerIndex += length; + return this; + } + + @Override + public boolean release() { + return --referenceCount <= 0; + } + + @Override + public Buffer retain() { + referenceCount++; + return this; + } + + @Override + public int referenceCount() { + return referenceCount; + } + + @Override + public int nioBufferCount() { + return 1; + } + + @Override + public ByteBuffer[] nioBuffers() { + return new ByteBuffer[] { nioBuffer() }; + } + + @Override + public ByteBuffer[] nioBuffers(final int index, final int length) { + return new ByteBuffer[] { nioBuffer(index, length) }; + } + + @Override + public ByteBuffer nioBuffer() { + return nioBuffer(readerIndex, readableBytes()); + } + + @Override + public ByteBuffer nioBuffer(final int index, final int length) { + return ByteBuffer.wrap(Arrays.copyOfRange(array, index, index + length)); + } + + @Override + public Buffer getBytes(final int index, final byte[] dst) { + System.arraycopy(array, index, dst, 0, dst.length); + return this; + } +} diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/DefaultStorage.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/DefaultStorage.java new file mode 100644 index 0000000000..49c4a2becf --- /dev/null +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/DefaultStorage.java @@ -0,0 +1,40 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import java.util.function.Supplier; + +/** + * The built-in {@link TinkerStorage} engines that can be selected by name via the + * {@code gremlin.tinkergraph.storage} configuration key. A fully-qualified class name may be used instead of one of + * these names to plug in a custom engine. + */ +public enum DefaultStorage implements Supplier<TinkerStorage> { + + /** + * A durable, append-only commit log ("write-ahead log") that serializes each committed transaction with + * GraphBinary. See {@link GraphBinaryStorage}. + */ + GRAPHBINARY { + @Override + public TinkerStorage get() { + return new GraphBinaryStorage(); + } + } +} diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorage.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorage.java new file mode 100644 index 0000000000..b07891a93c --- /dev/null +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorage.java @@ -0,0 +1,379 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.apache.commons.configuration2.Configuration; +import org.apache.tinkerpop.gremlin.structure.Edge; +import org.apache.tinkerpop.gremlin.structure.Vertex; +import org.apache.tinkerpop.gremlin.structure.io.binary.GraphBinaryReader; +import org.apache.tinkerpop.gremlin.structure.io.binary.GraphBinaryWriter; +import org.apache.tinkerpop.gremlin.structure.io.binary.TypeSerializerRegistry; +import org.apache.tinkerpop.gremlin.structure.util.Attachable; +import org.apache.tinkerpop.gremlin.structure.util.detached.DetachedEdge; +import org.apache.tinkerpop.gremlin.structure.util.detached.DetachedFactory; +import org.apache.tinkerpop.gremlin.structure.util.detached.DetachedVertex; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.AbstractTinkerGraph; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerEdge; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerVertex; + +import java.io.BufferedOutputStream; +import java.io.DataInputStream; +import java.io.DataOutputStream; +import java.io.EOFException; +import java.io.File; +import java.io.FileInputStream; +import java.io.FileOutputStream; +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.io.UncheckedIOException; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Iterator; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; + +/** + * A durable {@link TinkerStorage} engine that persists a {@code TinkerStorageGraph} as an append-only commit log + * ("write-ahead log") serialized with GraphBinary. On each committed transaction the changeset is appended to + * {@code log.gbin} as a single record; on open the optional {@code snapshot.gbin} is read followed by the log, with the + * folded result re-applied to the in-memory graph. {@link #compact(AbstractTinkerGraph)} rewrites the snapshot from the + * current committed state and truncates the log. + * <p/> + * The in-memory graph remains authoritative (write-through). This engine does not support graphs larger than memory. + */ +public final class GraphBinaryStorage implements TinkerStorage { + + /** + * Version byte prefixing every record, allowing the on-disk format to evolve. + */ + static final byte FORMAT_VERSION = 1; + + private static final byte OP_PUT_VERTEX = 1; + private static final byte OP_DEL_VERTEX = 2; + private static final byte OP_PUT_EDGE = 3; + private static final byte OP_DEL_EDGE = 4; + + static final String SNAPSHOT_FILE = "snapshot.gbin"; + static final String LOG_FILE = "log.gbin"; + + private final GraphBinaryWriter writer = new GraphBinaryWriter(TypeSerializerRegistry.INSTANCE); + private final GraphBinaryReader reader = new GraphBinaryReader(TypeSerializerRegistry.INSTANCE); + + private File directory; + private File snapshotFile; + private File logFile; + + private DataOutputStream logOut; + private boolean closed = false; + + @Override + public void open(final AbstractTinkerGraph graph, final Configuration config) { + final String location = config.getString(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, null); + if (null == location) + throw new IllegalStateException(String.format("%s must be set to use the GraphBinary storage engine", + TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION)); + this.directory = new File(location); + this.snapshotFile = new File(directory, SNAPSHOT_FILE); + this.logFile = new File(directory, LOG_FILE); + ensureDirectory(); + } + + /** + * Ensure the backing directory exists, creating it if necessary. Called on open and again before writing a + * snapshot, since {@code close()} may be invoked more than once and the directory may have been removed in between. + */ + private void ensureDirectory() { + if (directory.exists()) { + if (!directory.isDirectory()) + throw new IllegalStateException(String.format("Storage location %s exists but is not a directory", directory)); + } else if (!directory.mkdirs()) { + throw new IllegalStateException(String.format("Could not create storage directory %s", directory)); + } + } + + @Override + public void replay(final AbstractTinkerGraph graph) { + // Fold snapshot then log into final state: last write per id wins, deletes remove. + final Map<Object, DetachedVertex> vertices = new LinkedHashMap<>(); + final Map<Object, DetachedEdge> edges = new LinkedHashMap<>(); + + if (snapshotFile.exists()) + foldRecords(snapshotFile, vertices, edges); + if (logFile.exists()) + foldRecords(logFile, vertices, edges); + + if (vertices.isEmpty() && edges.isEmpty()) + return; + + // Attach vertices first so edges can find their endpoints, then commit once. + for (final DetachedVertex v : vertices.values()) + v.attach(Attachable.Method.getOrCreate(graph)); + for (final DetachedEdge e : edges.values()) + e.attach(Attachable.Method.getOrCreate(graph)); + + graph.tx().commit(); + } + + /** + * Read every record in a file, folding puts and deletes into the supplied maps. + */ + private void foldRecords(final File file, final Map<Object, DetachedVertex> vertices, final Map<Object, DetachedEdge> edges) { + try (final DataInputStream in = new DataInputStream(new java.io.BufferedInputStream(new FileInputStream(file)))) { + while (true) { + final byte[] record; + try { + record = readFrame(in); + } catch (EOFException eof) { + break; + } + if (record == null) + break; + applyRecord(record, vertices, edges); + } + } catch (IOException ex) { + throw new UncheckedIOException(String.format("Could not read storage file %s", file), ex); + } + } + + private void applyRecord(final byte[] record, final Map<Object, DetachedVertex> vertices, final Map<Object, DetachedEdge> edges) throws IOException { + final ByteBufferBuffer buffer = new ByteBufferBuffer(record); + final byte version = buffer.readByte(); + if (version != FORMAT_VERSION) + throw new IOException(String.format("Unsupported storage record version %d (expected %d)", version, FORMAT_VERSION)); + buffer.readLong(); // txVersion, retained for diagnostics/future use + final int entryCount = buffer.readInt(); + for (int i = 0; i < entryCount; i++) { + final byte op = buffer.readByte(); + switch (op) { + case OP_PUT_VERTEX: { + final Vertex v = reader.read(buffer); + vertices.put(v.id(), (DetachedVertex) v); + break; + } + case OP_DEL_VERTEX: { + final Object id = reader.read(buffer); + vertices.remove(id); + break; + } + case OP_PUT_EDGE: { + final Edge e = reader.read(buffer); + edges.put(e.id(), (DetachedEdge) e); + break; + } + case OP_DEL_EDGE: { + final Object id = reader.read(buffer); + edges.remove(id); + break; + } + default: + throw new IOException("Unknown storage op code: " + op); + } + } + } + + @Override + public void persist(final long txVersion, + final Collection<TinkerStorageMutation<TinkerVertex>> changedVertices, + final Collection<TinkerStorageMutation<TinkerEdge>> changedEdges) { + ensureLogOpen(); + try { + final byte[] frame = encodeRecord(txVersion, changedVertices, changedEdges); + writeFrame(logOut, frame); + } catch (IOException ex) { + throw new UncheckedIOException("Could not append transaction to storage log", ex); + } + } + + /** + * Serialize a commit record: version byte, txVersion, entry count, then each entry as an op byte followed by + * either the serialized element (put) or the serialized id (delete). + */ + private byte[] encodeRecord(final long txVersion, + final Collection<TinkerStorageMutation<TinkerVertex>> changedVertices, + final Collection<TinkerStorageMutation<TinkerEdge>> changedEdges) throws IOException { + final ByteBufferBuffer buffer = new ByteBufferBuffer(); + buffer.writeByte(FORMAT_VERSION); + buffer.writeLong(txVersion); + buffer.writeInt(changedVertices.size() + changedEdges.size()); + for (final TinkerStorageMutation<TinkerVertex> m : changedVertices) { + if (m.isDeleted()) { + buffer.writeByte(OP_DEL_VERTEX); + writer.write(m.id(), buffer); + } else { + buffer.writeByte(OP_PUT_VERTEX); + // detach to a stable form independent of the transactional element + writer.write(DetachedFactory.detach(m.element(), true), buffer); + } + } + for (final TinkerStorageMutation<TinkerEdge> m : changedEdges) { + if (m.isDeleted()) { + buffer.writeByte(OP_DEL_EDGE); + writer.write(m.id(), buffer); + } else { + buffer.writeByte(OP_PUT_EDGE); + writer.write(DetachedFactory.detach(m.element(), true), buffer); + } + } + return buffer.toWrittenArray(); + } + + @Override + public void flush() { + if (closed) + return; + if (logOut != null) { + try { + logOut.flush(); + } catch (IOException ex) { + throw new UncheckedIOException("Could not flush storage log", ex); + } + } + } + + @Override + public void compact(final AbstractTinkerGraph graph) { + if (closed) + return; + // Write a fresh snapshot of the current committed state, then truncate the log. + closeLog(); + ensureDirectory(); + final File tmp = new File(directory, SNAPSHOT_FILE + ".tmp"); + try (final DataOutputStream out = new DataOutputStream(new BufferedOutputStream(new FileOutputStream(tmp)))) { + final byte[] frame = encodeSnapshot(graph); + if (frame.length > 0) + writeFrame(out, frame); + out.flush(); + } catch (IOException ex) { + throw new UncheckedIOException("Could not write storage snapshot", ex); + } + + if (snapshotFile.exists() && !snapshotFile.delete()) + throw new UncheckedIOException(new IOException("Could not replace snapshot " + snapshotFile)); + if (!tmp.renameTo(snapshotFile)) + throw new UncheckedIOException(new IOException("Could not rename snapshot into place " + snapshotFile)); + + // truncate the log + if (logFile.exists() && !logFile.delete()) + throw new UncheckedIOException(new IOException("Could not truncate storage log " + logFile)); + } + + /** + * Serialize the entire current committed state of the graph as a single put-only record. + */ + private byte[] encodeSnapshot(final AbstractTinkerGraph graph) throws IOException { + final List<Vertex> vertexList = new ArrayList<>(); + final Iterator<Vertex> vertexIterator = graph.vertices(); + while (vertexIterator.hasNext()) + vertexList.add(vertexIterator.next()); + final List<Edge> edgeList = new ArrayList<>(); + final Iterator<Edge> edgeIterator = graph.edges(); + while (edgeIterator.hasNext()) + edgeList.add(edgeIterator.next()); + + if (vertexList.isEmpty() && edgeList.isEmpty()) + return new byte[0]; + + final ByteBufferBuffer buffer = new ByteBufferBuffer(); + buffer.writeByte(FORMAT_VERSION); + buffer.writeLong(0L); // snapshot has no single tx version + buffer.writeInt(vertexList.size() + edgeList.size()); + for (final Vertex v : vertexList) { + buffer.writeByte(OP_PUT_VERTEX); + writer.write(DetachedFactory.detach(v, true), buffer); + } + for (final Edge e : edgeList) { + buffer.writeByte(OP_PUT_EDGE); + writer.write(DetachedFactory.detach(e, true), buffer); + } + return buffer.toWrittenArray(); + } + + @Override + public void close() { + closeLog(); + closed = true; + } + + private void ensureLogOpen() { + if (logOut == null) { + try { + logOut = new DataOutputStream(new BufferedOutputStream(new FileOutputStream(logFile, true))); + } catch (IOException ex) { + throw new UncheckedIOException("Could not open storage log for append", ex); + } + } + } + + private void closeLog() { + if (logOut != null) { + try { + logOut.flush(); + logOut.close(); + } catch (IOException ex) { + throw new UncheckedIOException("Could not close storage log", ex); + } finally { + logOut = null; + } + } + } + + /** + * Write a length-prefixed frame: a 4-byte big-endian length followed by the payload. + */ + private static void writeFrame(final DataOutputStream out, final byte[] payload) throws IOException { + out.writeInt(payload.length); + out.write(payload); + } + + /** + * Read a length-prefixed frame, or return {@code null} on a clean end of file. A truncated final frame (from a + * crash mid-append) is treated as end of file so earlier committed records still load. + */ + private static byte[] readFrame(final DataInputStream in) throws IOException { + final int length; + try { + length = in.readInt(); + } catch (EOFException eof) { + return null; + } + if (length < 0) + throw new IOException("Corrupt storage frame length: " + length); + final byte[] payload = new byte[length]; + try { + readFully(in, payload); + } catch (EOFException eof) { + // partial trailing frame from an interrupted append — stop here + return null; + } + return payload; + } + + private static void readFully(final InputStream in, final byte[] dst) throws IOException { + int off = 0; + while (off < dst.length) { + final int read = in.read(dst, off, dst.length - off); + if (read < 0) + throw new EOFException(); + off += read; + } + } +} diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/TinkerStorage.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/TinkerStorage.java new file mode 100644 index 0000000000..49dcedb1ba --- /dev/null +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/TinkerStorage.java @@ -0,0 +1,100 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.apache.commons.configuration2.Configuration; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.AbstractTinkerGraph; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerEdge; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerVertex; + +import java.util.Collection; + +/** + * A pluggable durable storage engine for a transactional {@code TinkerStorageGraph}. Implementations persist the + * changeset of each committed transaction to disk and rebuild the in-memory graph on open. The graph remains the + * authoritative in-memory copy (write-through); the engine is a durable mirror. + * <p/> + * The lifecycle is: + * <ol> + * <li>{@link #open(AbstractTinkerGraph, Configuration)} — resolve the backing location and ready the store.</li> + * <li>{@link #replay(AbstractTinkerGraph)} — rebuild in-memory state from what was previously persisted.</li> + * <li>{@link #persist(long, Collection, Collection)} then {@link #flush()} — called on each transaction commit, + * before the in-memory state is committed, so a failure aborts the commit and leaves memory and disk consistent.</li> + * <li>{@link #compact(AbstractTinkerGraph)} — optionally fold accumulated changes into a compact snapshot.</li> + * <li>{@link #close()} — release resources.</li> + * </ol> + * A new engine can be selected by name (see {@code TinkerStorageGraph.DefaultStorage}) or by fully-qualified class + * name via the {@code gremlin.tinkergraph.storage} configuration key. Implementations must provide a public no-argument + * constructor. + */ +public interface TinkerStorage extends AutoCloseable { + + /** + * Prepare the storage engine for use, resolving its backing location from the supplied configuration. Called once + * during graph construction before {@link #replay(AbstractTinkerGraph)}. + * + * @param graph the graph that owns this engine + * @param config the graph configuration, including {@code gremlin.tinkergraph.graphLocation} + */ + void open(AbstractTinkerGraph graph, Configuration config); + + /** + * Rebuild the in-memory state of the graph from previously persisted data. Called once during graph construction. + * Implementations should re-apply persisted elements through the graph's own mutation API; the graph sets its + * {@code loading} guard for the duration so this does not re-persist. + * + * @param graph the graph to populate + */ + void replay(AbstractTinkerGraph graph); + + /** + * Durably record the changeset of a committing transaction. Called from within the transaction commit, while the + * changed elements are locked, before the in-memory commit is applied. Each mutation is either a put (added or + * modified element) or a delete (see {@link TinkerStorageMutation}). + * + * @param txVersion the version number of the committing transaction + * @param changedVertices the vertex mutations in this transaction + * @param changedEdges the edge mutations in this transaction + */ + void persist(long txVersion, + Collection<TinkerStorageMutation<TinkerVertex>> changedVertices, + Collection<TinkerStorageMutation<TinkerEdge>> changedEdges); + + /** + * Force any buffered writes to durable storage. Called after {@link #persist(long, Collection, Collection)} as the + * commit's durability point. + */ + void flush(); + + /** + * Fold accumulated changes into a compact representation of the current committed state, reclaiming space. Safe to + * call at any time; typically invoked on {@link #close()} or on a size/commit-count threshold. + * + * @param graph the graph whose current committed state should be snapshotted + */ + void compact(AbstractTinkerGraph graph); + + /** + * {@inheritDoc} + * <p/> + * Flush and release resources. Does not delete persisted data. + */ + @Override + void close(); +} diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/TinkerStorageMutation.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/TinkerStorageMutation.java new file mode 100644 index 0000000000..726c6862fd --- /dev/null +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/TinkerStorageMutation.java @@ -0,0 +1,67 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.apache.tinkerpop.gremlin.structure.Element; + +/** + * A single element change within a committing transaction, handed to a {@link TinkerStorage} engine to persist. This + * is the stable, public view of a change that decouples storage engines from TinkerGraph's internal transactional + * containers. A mutation is either a <em>put</em> (the element was added or modified, {@link #element()} is non-null) + * or a <em>delete</em> ({@link #element()} is {@code null} and {@link #isDeleted()} is {@code true}). + * + * @param <T> the element type ({@code TinkerVertex} or {@code TinkerEdge}) + */ +public final class TinkerStorageMutation<T extends Element> { + + private final Object id; + private final T element; + + /** + * Create a mutation for the given element id. + * + * @param id the element identifier (never {@code null}) + * @param element the committed element for a put, or {@code null} for a delete + */ + public TinkerStorageMutation(final Object id, final T element) { + this.id = id; + this.element = element; + } + + /** + * The identifier of the changed element. + */ + public Object id() { + return id; + } + + /** + * The committed element to persist, or {@code null} when this mutation is a deletion. + */ + public T element() { + return element; + } + + /** + * Returns {@code true} when this mutation deletes the element rather than adding or modifying it. + */ + public boolean isDeleted() { + return element == null; + } +} diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerMemoryGraphProvider.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerMemoryGraphProvider.java index fe17fd7d71..170b43f8f5 100644 --- a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerMemoryGraphProvider.java +++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerMemoryGraphProvider.java @@ -21,7 +21,6 @@ package org.apache.tinkerpop.gremlin.tinkergraph; import org.apache.commons.configuration2.Configuration; import org.apache.tinkerpop.gremlin.AbstractGraphProvider; import org.apache.tinkerpop.gremlin.LoadGraphWith; -import org.apache.tinkerpop.gremlin.TestHelper; import org.apache.tinkerpop.gremlin.structure.Graph; import org.apache.tinkerpop.gremlin.structure.GraphTest; import org.apache.tinkerpop.gremlin.structure.VertexProperty; @@ -39,7 +38,6 @@ import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerProperty; import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerVertex; import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerVertexProperty; -import java.io.File; import java.util.HashMap; import java.util.HashSet; import java.util.Map; @@ -73,10 +71,6 @@ public class TinkerMemoryGraphProvider extends AbstractGraphProvider { put(TinkerGraph.GREMLIN_TINKERGRAPH_VERTEX_PROPERTY_ID_MANAGER, idMaker); if (requiresListCardinalityAsDefault(loadGraphWith, test, testMethodName)) put(TinkerGraph.GREMLIN_TINKERGRAPH_DEFAULT_VERTEX_PROPERTY_CARDINALITY, VertexProperty.Cardinality.list.name()); - if (requiresPersistence(test, testMethodName)) { - put(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_FORMAT, "gryo"); - put(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION,TestHelper.makeTestDataFile(test, "temp", testMethodName + ".kryo")); - } }}; } @@ -84,13 +78,6 @@ public class TinkerMemoryGraphProvider extends AbstractGraphProvider { public void clear(final Graph graph, final Configuration configuration) throws Exception { if (graph != null) graph.close(); - - // in the even the graph is persisted we need to clean up - final String graphLocation = null != configuration ? configuration.getString(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, null) : null; - if (graphLocation != null) { - final File f = new File(graphLocation); - f.delete(); - } } @Override @@ -98,13 +85,6 @@ public class TinkerMemoryGraphProvider extends AbstractGraphProvider { return IMPLEMENTATION; } - /** - * Determines if a test requires TinkerGraph persistence to be configured with graph location and format. - */ - protected static boolean requiresPersistence(final Class<?> test, final String testMethodName) { - return test == GraphTest.class && testMethodName.equals("shouldPersistDataOnClose"); - } - /** * Determines if a test requires a different cardinality as the default or not. */ diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerMemoryGraphUUIDProvider.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerMemoryGraphUUIDProvider.java index 8a127e8330..ef37147838 100644 --- a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerMemoryGraphUUIDProvider.java +++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerMemoryGraphUUIDProvider.java @@ -20,13 +20,11 @@ package org.apache.tinkerpop.gremlin.tinkergraph; import org.apache.tinkerpop.gremlin.LoadGraphWith; -import org.apache.tinkerpop.gremlin.TestHelper; import org.apache.tinkerpop.gremlin.structure.Graph; import org.apache.tinkerpop.gremlin.structure.VertexProperty; import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph; import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerMemoryGraph; -import java.io.File; import java.util.HashMap; import java.util.Map; @@ -47,10 +45,6 @@ public class TinkerMemoryGraphUUIDProvider extends TinkerMemoryGraphProvider { put(TinkerGraph.GREMLIN_TINKERGRAPH_VERTEX_PROPERTY_ID_MANAGER, idMaker); if (requiresListCardinalityAsDefault(loadGraphWith, test, testMethodName)) put(TinkerGraph.GREMLIN_TINKERGRAPH_DEFAULT_VERTEX_PROPERTY_CARDINALITY, VertexProperty.Cardinality.list.name()); - if (requiresPersistence(test, testMethodName)) { - put(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_FORMAT, "gryo"); - put(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, TestHelper.makeTestDataFile(test, "temp", testMethodName + ".kryo")); - } }}; } } diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerStorageGraphProvider.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerStorageGraphProvider.java index 6f777a630c..a0b686266f 100644 --- a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerStorageGraphProvider.java +++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerStorageGraphProvider.java @@ -70,8 +70,9 @@ public class TinkerStorageGraphProvider extends AbstractGraphProvider { if (requiresListCardinalityAsDefault(loadGraphWith, test, testMethodName)) put(TinkerStorageGraph.GREMLIN_TINKERGRAPH_DEFAULT_VERTEX_PROPERTY_CARDINALITY, VertexProperty.Cardinality.list.name()); if (requiresPersistence(test, testMethodName)) { - put(TinkerStorageGraph.GREMLIN_TINKERGRAPH_GRAPH_FORMAT, "gryo"); - put(TinkerStorageGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION,TestHelper.makeTestDataFile(test, "temp", testMethodName + ".kryo")); + put(TinkerStorageGraph.GREMLIN_TINKERGRAPH_STORAGE, "graphbinary"); + put(TinkerStorageGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, + TestHelper.makeTestDataDirectory(test, "temp", testMethodName)); } }}; } @@ -81,14 +82,26 @@ public class TinkerStorageGraphProvider extends AbstractGraphProvider { if (graph != null) graph.close(); - // in the even the graph is persisted we need to clean up + // in the event the graph is persisted we need to clean up the storage directory final String graphLocation = null != configuration ? configuration.getString(TinkerStorageGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, null) : null; if (graphLocation != null) { - final File f = new File(graphLocation); - f.delete(); + deleteRecursively(new File(graphLocation)); } } + private static void deleteRecursively(final File file) { + if (!file.exists()) + return; + if (file.isDirectory()) { + final File[] children = file.listFiles(); + if (children != null) { + for (final File child : children) + deleteRecursively(child); + } + } + file.delete(); + } + @Override public Set<Class> getImplementations() { return IMPLEMENTATION; diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerMemoryGraphTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerMemoryGraphTest.java index 4861eef518..ccde3e09a2 100644 --- a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerMemoryGraphTest.java +++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerMemoryGraphTest.java @@ -21,7 +21,6 @@ package org.apache.tinkerpop.gremlin.tinkergraph.structure; import org.apache.commons.configuration2.BaseConfiguration; import org.apache.commons.configuration2.Configuration; import org.apache.tinkerpop.gremlin.GraphHelper; -import org.apache.tinkerpop.gremlin.TestHelper; import org.apache.tinkerpop.gremlin.process.computer.Computer; import org.apache.tinkerpop.gremlin.process.traversal.P; import org.apache.tinkerpop.gremlin.process.traversal.Traversal; @@ -38,7 +37,6 @@ import org.apache.tinkerpop.gremlin.structure.Graph; import org.apache.tinkerpop.gremlin.structure.T; import org.apache.tinkerpop.gremlin.structure.Vertex; import org.apache.tinkerpop.gremlin.structure.VertexProperty; -import org.apache.tinkerpop.gremlin.structure.io.Io; import org.apache.tinkerpop.gremlin.structure.io.GraphReader; import org.apache.tinkerpop.gremlin.structure.io.GraphWriter; import org.apache.tinkerpop.gremlin.structure.io.IoCore; @@ -62,11 +60,8 @@ import org.apache.tinkerpop.shaded.kryo.io.Output; import org.junit.Test; import java.awt.Color; -import java.io.BufferedOutputStream; import java.io.ByteArrayInputStream; import java.io.ByteArrayOutputStream; -import java.io.File; -import java.io.FileOutputStream; import java.io.InputStream; import java.util.ArrayList; import java.util.Arrays; @@ -78,7 +73,6 @@ import java.util.Random; import java.util.Set; import java.util.UUID; import java.util.concurrent.TimeUnit; -import java.util.function.Consumer; import java.util.function.Supplier; import static org.apache.tinkerpop.gremlin.process.traversal.AnonymousTraversalSource.traversal; @@ -91,7 +85,6 @@ import static org.junit.Assert.assertNotSame; import static org.junit.Assert.assertTrue; import static org.junit.Assert.fail; import static org.junit.Assume.assumeThat; -import static org.mockito.Mockito.mock; /** * @author Marko A. Rodriguez (http://markorodriguez.com) @@ -386,13 +379,6 @@ public class TinkerMemoryGraphTest { } } - @Test(expected = IllegalStateException.class) - public void shouldRequireGraphLocationIfFormatIsSet() { - final Configuration conf = new BaseConfiguration(); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_FORMAT, "graphml"); - TinkerGraph.open(conf); - } - @Test(expected = IllegalStateException.class) public void shouldNotModifyAVertexThatWasRemoved() { final TinkerGraph graph = TinkerGraph.open(); @@ -427,137 +413,6 @@ public class TinkerMemoryGraphTest { v.value("name"); } - @Test(expected = IllegalStateException.class) - public void shouldRequireGraphFormatIfLocationIsSet() { - final Configuration conf = new BaseConfiguration(); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, TestHelper.makeTestDataDirectory(TinkerMemoryGraphTest.class)); - TinkerGraph.open(conf); - } - - @Test - public void shouldPersistToGraphML() { - final String graphLocation = TestHelper.makeTestDataFile(TinkerMemoryGraphTest.class, "shouldPersistToGraphML.xml"); - final File f = new File(graphLocation); - if (f.exists() && f.isFile()) f.delete(); - - final Configuration conf = new BaseConfiguration(); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_FORMAT, "graphml"); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, graphLocation); - final TinkerGraph graph = TinkerGraph.open(conf); - TinkerFactory.generateModern(graph); - graph.close(); - - final TinkerGraph reloadedGraph = TinkerGraph.open(conf); - IoTest.assertModernGraph(reloadedGraph, true, true); - reloadedGraph.close(); - } - - @Test - public void shouldPersistToGraphSON() { - final String graphLocation = TestHelper.makeTestDataFile(TinkerMemoryGraphTest.class, "shouldPersistToGraphSON.json"); - final File f = new File(graphLocation); - if (f.exists() && f.isFile()) f.delete(); - - final Configuration conf = new BaseConfiguration(); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_FORMAT, "graphson"); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, graphLocation); - final TinkerGraph graph = TinkerGraph.open(conf); - TinkerFactory.generateModern(graph); - graph.close(); - - final TinkerGraph reloadedGraph = TinkerGraph.open(conf); - IoTest.assertModernGraph(reloadedGraph, true, false); - reloadedGraph.close(); - } - - @Test - public void shouldPersistToGryo() { - final String graphLocation = TestHelper.makeTestDataFile(TinkerMemoryGraphTest.class, "shouldPersistToGryo.kryo"); - final File f = new File(graphLocation); - if (f.exists() && f.isFile()) f.delete(); - - final Configuration conf = new BaseConfiguration(); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_FORMAT, "gryo"); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, graphLocation); - final TinkerGraph graph = TinkerGraph.open(conf); - TinkerFactory.generateModern(graph); - graph.close(); - - final TinkerGraph reloadedGraph = TinkerGraph.open(conf); - IoTest.assertModernGraph(reloadedGraph, true, false); - reloadedGraph.close(); - } - - @Test - public void shouldPersistToGryoAndHandleMultiProperties() { - final String graphLocation = TestHelper.makeTestDataFile(TinkerMemoryGraphTest.class, "shouldPersistToGryoMulti.kryo"); - final File f = new File(graphLocation); - if (f.exists() && f.isFile()) f.delete(); - - final Configuration conf = new BaseConfiguration(); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_FORMAT, "gryo"); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, graphLocation); - final TinkerGraph graph = TinkerGraph.open(conf); - TinkerFactory.generateTheCrew(graph); - graph.close(); - - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_DEFAULT_VERTEX_PROPERTY_CARDINALITY, VertexProperty.Cardinality.list.toString()); - final TinkerGraph reloadedGraph = TinkerGraph.open(conf); - IoTest.assertCrewGraph(reloadedGraph, false); - reloadedGraph.close(); - } - - @Test - public void shouldPersistWithRelativePath() { - final String graphLocation = TestHelper.convertToRelative(TinkerMemoryGraphTest.class, - TestHelper.makeTestDataPath(TinkerMemoryGraphTest.class)) - + "shouldPersistToGryoRelative.kryo"; - final File f = new File(graphLocation); - if (f.exists() && f.isFile()) f.delete(); - - final Configuration conf = new BaseConfiguration(); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_FORMAT, "gryo"); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, graphLocation); - final TinkerGraph graph = TinkerGraph.open(conf); - TinkerFactory.generateModern(graph); - graph.close(); - - final TinkerGraph reloadedGraph = TinkerGraph.open(conf); - IoTest.assertModernGraph(reloadedGraph, true, false); - reloadedGraph.close(); - } - - @Test - public void shouldPersistToAnyGraphFormat() { - final String graphLocation = TestHelper.makeTestDataFile(TinkerMemoryGraphTest.class, "shouldPersistToAnyGraphFormat.dat"); - final File f = new File(graphLocation); - if (f.exists() && f.isFile()) f.delete(); - - final Configuration conf = new BaseConfiguration(); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_FORMAT, TestIoBuilder.class.getName()); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, graphLocation); - final TinkerGraph graph = TinkerGraph.open(conf); - TinkerFactory.generateModern(graph); - - //Test write graph - graph.close(); - assertEquals(TestIoBuilder.calledOnMapper, 1); - assertEquals(TestIoBuilder.calledGraph, 1); - assertEquals(TestIoBuilder.calledCreate, 1); - - try (BufferedOutputStream os = new BufferedOutputStream(new FileOutputStream(f))){ - os.write("dummy string".getBytes()); - } catch (Exception e) { - e.printStackTrace(); - } - - //Test read graph - final TinkerGraph readGraph = TinkerGraph.open(conf); - assertEquals(TestIoBuilder.calledOnMapper, 1); - assertEquals(TestIoBuilder.calledGraph, 1); - assertEquals(TestIoBuilder.calledCreate, 1); - } - @Test public void shouldSerializeWithColorClassResolverToTinkerGraph() throws Exception { final Map<String,Color> colors = new HashMap<>(); @@ -1026,38 +881,4 @@ public class TinkerMemoryGraphTest { } } - public static class TestIoBuilder implements Io.Builder { - - static int calledGraph, calledCreate, calledOnMapper; - - public TestIoBuilder(){ - //Looks awkward to reset static vars inside a constructor, but makes sense from testing perspective - calledGraph = 0; - calledCreate = 0; - calledOnMapper = 0; - } - - @Override - public Io.Builder<? extends Io> onMapper(final Consumer onMapper) { - calledOnMapper++; - return this; - } - - @Override - public Io.Builder<? extends Io> graph(final Graph graph) { - calledGraph++; - return this; - } - - @Override - public Io create() { - calledCreate++; - return mock(Io.class); - } - - @Override - public boolean requiresVersion(final Object version) { - return false; - } - } } diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/AbstractTinkerStorageConformanceTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/AbstractTinkerStorageConformanceTest.java new file mode 100644 index 0000000000..561320bcff --- /dev/null +++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/AbstractTinkerStorageConformanceTest.java @@ -0,0 +1,242 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.apache.commons.configuration2.BaseConfiguration; +import org.apache.commons.configuration2.Configuration; +import org.apache.tinkerpop.gremlin.structure.Edge; +import org.apache.tinkerpop.gremlin.structure.Graph; +import org.apache.tinkerpop.gremlin.structure.T; +import org.apache.tinkerpop.gremlin.structure.Vertex; +import org.apache.tinkerpop.gremlin.structure.VertexProperty; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph; +import org.junit.Before; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +import java.util.Iterator; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; + +/** + * Engine-agnostic conformance suite ("TCK") for pluggable {@link TinkerStorage} engines. A concrete engine is tested + * by subclassing this and returning the {@code gremlin.tinkergraph.storage} value that selects it (an engine name or a + * fully-qualified class name). Every test opens a {@link TinkerStorageGraph} backed by a fresh temporary directory, + * mutates it, reopens from the same configuration, and asserts the data survived. A new engine drops in by adding one + * subclass. + */ +public abstract class AbstractTinkerStorageConformanceTest { + + @Rule + public TemporaryFolder tempFolder = new TemporaryFolder(); + + private String location; + + /** + * The {@code gremlin.tinkergraph.storage} configuration value that selects the engine under test. + */ + protected abstract String storageEngine(); + + @Before + public void setUp() throws Exception { + location = tempFolder.newFolder("storage").getAbsolutePath(); + } + + protected Configuration config() { + final Configuration conf = new BaseConfiguration(); + conf.setProperty(Graph.GRAPH, TinkerStorageGraph.class.getName()); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE, storageEngine()); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, location); + return conf; + } + + protected TinkerStorageGraph open() { + return TinkerStorageGraph.open(config()); + } + + @Test + public void shouldPersistVerticesAndEdgesAcrossReopen() { + TinkerStorageGraph graph = open(); + final Vertex marko = graph.addVertex(T.id, 1, T.label, "person", "name", "marko", "age", 29); + final Vertex lop = graph.addVertex(T.id, 3, T.label, "software", "name", "lop", "lang", "java"); + marko.addEdge("created", lop, T.id, 9, "weight", 0.4); + graph.tx().commit(); + graph.close(); + + graph = open(); + assertEquals(2, countOf(graph.vertices())); + assertEquals(1, countOf(graph.edges())); + final Vertex reMarko = graph.vertices(1).next(); + assertEquals("marko", reMarko.value("name")); + assertEquals(Integer.valueOf(29), reMarko.value("age")); + final Edge reCreated = graph.edges(9).next(); + assertEquals("created", reCreated.label()); + assertEquals(0.4, reCreated.<Double>value("weight"), 0.0001); + assertEquals(Integer.valueOf(1), reCreated.outVertex().id()); + assertEquals(Integer.valueOf(3), reCreated.inVertex().id()); + graph.close(); + } + + @Test + public void shouldPersistAcrossMultipleCommits() { + TinkerStorageGraph graph = open(); + for (int i = 0; i < 10; i++) { + graph.addVertex(T.id, i, "value", i); + graph.tx().commit(); + } + graph.close(); + + graph = open(); + assertEquals(10, countOf(graph.vertices())); + for (int i = 0; i < 10; i++) + assertEquals(Integer.valueOf(i), graph.vertices(i).next().value("value")); + graph.close(); + } + + @Test + public void shouldPersistModificationsWithLastWriteWinning() { + TinkerStorageGraph graph = open(); + final Vertex v = graph.addVertex(T.id, 1, "name", "original"); + graph.tx().commit(); + v.property("name", "updated"); + graph.tx().commit(); + graph.close(); + + graph = open(); + assertEquals("updated", graph.vertices(1).next().value("name")); + graph.close(); + } + + @Test + public void shouldNotPersistRemovedElements() { + TinkerStorageGraph graph = open(); + final Vertex a = graph.addVertex(T.id, 1); + final Vertex b = graph.addVertex(T.id, 2); + final Edge e = a.addEdge("knows", b, T.id, 10); + graph.tx().commit(); + e.remove(); + b.remove(); + graph.tx().commit(); + graph.close(); + + graph = open(); + assertEquals(1, countOf(graph.vertices())); + assertEquals(0, countOf(graph.edges())); + assertNotNull(graph.vertices(1).next()); + assertFalse(graph.vertices(2).hasNext()); + graph.close(); + } + + @Test + public void shouldNotPersistRolledBackTransaction() { + TinkerStorageGraph graph = open(); + graph.addVertex(T.id, 1, "name", "committed"); + graph.tx().commit(); + graph.addVertex(T.id, 2, "name", "rolledback"); + graph.tx().rollback(); + graph.close(); + + graph = open(); + assertEquals(1, countOf(graph.vertices())); + assertNotNull(graph.vertices(1).next()); + assertFalse(graph.vertices(2).hasNext()); + graph.close(); + } + + @Test + public void shouldPersistMetaPropertiesAndMultiProperties() { + final Configuration conf = config(); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_DEFAULT_VERTEX_PROPERTY_CARDINALITY, VertexProperty.Cardinality.list.name()); + TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + final Vertex v = graph.addVertex(T.id, 1); + final VertexProperty<String> vp = v.property(VertexProperty.Cardinality.list, "name", "marko"); + vp.property("acl", "public"); + v.property(VertexProperty.Cardinality.list, "name", "marko a. rodriguez"); + graph.tx().commit(); + graph.close(); + + graph = TinkerStorageGraph.open(conf); + final Vertex reV = graph.vertices(1).next(); + assertEquals(2, countOf(reV.properties("name"))); + final Iterator<VertexProperty<Object>> props = reV.properties("name"); + boolean foundAcl = false; + while (props.hasNext()) { + final VertexProperty<Object> p = props.next(); + if (p.properties("acl").hasNext()) { + assertEquals("public", p.properties("acl").next().value()); + foundAcl = true; + } + } + assertTrue("meta-property should survive persistence", foundAcl); + graph.close(); + } + + @Test + public void shouldPreserveStateAfterCompact() { + TinkerStorageGraph graph = open(); + for (int i = 0; i < 5; i++) { + graph.addVertex(T.id, i, "value", i); + graph.tx().commit(); + } + graph.compact(); + // keep writing after compaction to exercise the truncated log + graph.addVertex(T.id, 100, "value", 100); + graph.tx().commit(); + graph.close(); + + graph = open(); + assertEquals(6, countOf(graph.vertices())); + assertEquals(Integer.valueOf(100), graph.vertices(100).next().value("value")); + assertEquals(Integer.valueOf(3), graph.vertices(3).next().value("value")); + graph.close(); + } + + @Test + public void shouldReopenEmptyGraph() { + TinkerStorageGraph graph = open(); + graph.close(); + + graph = open(); + assertEquals(0, countOf(graph.vertices())); + assertEquals(0, countOf(graph.edges())); + graph.close(); + } + + @Test + public void shouldReportPersistenceFeature() { + final TinkerStorageGraph graph = open(); + assertTrue(graph.features().graph().supportsPersistence()); + graph.close(); + } + + private static long countOf(final Iterator<?> it) { + long count = 0; + while (it.hasNext()) { + it.next(); + count++; + } + return count; + } +} diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorageTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorageTest.java new file mode 100644 index 0000000000..43c53d66f0 --- /dev/null +++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorageTest.java @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.apache.tinkerpop.gremlin.structure.T; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph; +import org.junit.Test; + +import java.io.File; +import java.io.RandomAccessFile; +import java.util.Iterator; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +/** + * Runs the shared {@link AbstractTinkerStorageConformanceTest} suite against the {@link GraphBinaryStorage} engine and + * adds engine-specific tests for the on-disk log layout. + */ +public class GraphBinaryStorageTest extends AbstractTinkerStorageConformanceTest { + + @Override + protected String storageEngine() { + return "graphbinary"; + } + + @Test + public void shouldWriteSnapshotAndLogFiles() { + final TinkerStorageGraph graph = open(); + graph.addVertex(T.id, 1); + graph.tx().commit(); + final String location = graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION); + assertTrue(new File(location, GraphBinaryStorage.LOG_FILE).exists()); + graph.close(); + // close compacts, producing a snapshot and truncating the log + assertTrue(new File(location, GraphBinaryStorage.SNAPSHOT_FILE).exists()); + } + + @Test + public void shouldRecoverFromTruncatedTrailingFrame() throws Exception { + TinkerStorageGraph graph = open(); + final String location = graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION); + graph.addVertex(T.id, 1, "value", 1); + graph.tx().commit(); + graph.addVertex(T.id, 2, "value", 2); + graph.tx().commit(); + // do NOT close (avoid compaction) so the raw log is preserved + graph.tx().close(); + + // simulate a crash mid-append by appending a partial (garbage) trailing frame to the log + final File logFile = new File(location, GraphBinaryStorage.LOG_FILE); + try (final RandomAccessFile raf = new RandomAccessFile(logFile, "rw")) { + raf.seek(raf.length()); + // a length prefix promising 100 bytes, but only a couple follow — a torn write + raf.writeInt(100); + raf.write(new byte[]{ 0x01, 0x02 }); + } + + // reopening must recover the two fully-committed vertices and ignore the torn frame + graph = open(); + assertEquals(2, countOf(graph.vertices())); + assertEquals(Integer.valueOf(1), graph.vertices(1).next().value("value")); + assertEquals(Integer.valueOf(2), graph.vertices(2).next().value("value")); + graph.close(); + } + + private static long countOf(final Iterator<?> it) { + long count = 0; + while (it.hasNext()) { + it.next(); + count++; + } + return count; + } +}
