This is an automated email from the ASF dual-hosted git repository.
Cole-Greer pushed a commit to branch 3.7-dev
in repository https://gitbox.apache.org/repos/asf/tinkerpop.git
The following commit(s) were added to refs/heads/3.7-dev by this push:
new bece4a34c7 TINKERPOP-3278 Disable Java serialization on Gryo IO reads
(#3578)
bece4a34c7 is described below
commit bece4a34c7eef753e1ce5ac3b6c95ee585f016b4
Author: Guian Gumpac <[email protected]>
AuthorDate: Wed Jul 29 14:56:46 2026 -0700
TINKERPOP-3278 Disable Java serialization on Gryo IO reads (#3578)
Several Gryo type registrations, mostly `TraversalStrategy` implementations
and `TraversalExplanation`,
were bound to Kryo's `JavaSerializer`, which deserializes via
`java.io.ObjectInputStream.readObject()`.
Reading a Gryo document from an untrusted source could therefore
reconstruct and run an arbitrary
`Serializable` object graph during decoding, before the graph layer accepts
or rejects anything.
This change stops the graph-document IO paths from reaching that sink,
while leaving full fidelity
available for trusted, in-process use.
Assisted-by: Kiro: Claude Opus 4.8
---
CHANGELOG.asciidoc | 1 +
THREAT_MODEL.md | 21 +-
docs/src/upgrade/release-3.7.x.asciidoc | 43 +++
.../process/traversal/step/sideEffect/IoStep.java | 6 +-
.../gremlin/structure/io/gryo/GryoIo.java | 4 +-
.../gremlin/structure/io/gryo/GryoMapper.java | 52 ++-
.../gremlin/structure/io/gryo/GryoReader.java | 5 +-
.../gremlin/structure/io/gryo/GryoWriter.java | 5 +-
.../gremlin/structure/io/gryo/GryoMapperTest.java | 352 +++++++++++++++++++++
.../gremlin/hadoop/structure/io/HadoopPools.java | 2 +
.../hadoop/structure/io/gryo/GryoRecordReader.java | 4 +-
.../hadoop/structure/io/gryo/GryoRecordWriter.java | 4 +-
.../io/gryo/GryoRecordReaderWriterTest.java | 88 ++++++
13 files changed, 571 insertions(+), 16 deletions(-)
diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc
index 37f9e7e4c5..8b6a157413 100644
--- a/CHANGELOG.asciidoc
+++ b/CHANGELOG.asciidoc
@@ -25,6 +25,7 @@
image::https://raw.githubusercontent.com/apache/tinkerpop/master/docs/static/ima
[[release-3-7-7]]
=== TinkerPop 3.7.7 (Release Date: NOT OFFICIALLY RELEASED YET)
+* Disabled unsafe Java deserialization on the Gryo IO paths and added
`GryoMapper.Builder.javaSerializationAllowed(boolean)` to control it.
* Fixed `subgraph()` to throw a descriptive error identifying the required
`Edge` input instead of an internal `ClassCastException` when the traversal
produces a non-edge value.
* Fixed `where(P)` to throw a descriptive error identifying the required
String scope key (and suggesting `is(P)` for value comparisons) instead of an
internal `ClassCastException` when given a non-String predicate value.
* Fixed `PeerPressure.property_name` in `gremlin-python` incorrectly mapping
to the `pageRank` property name token.
diff --git a/THREAT_MODEL.md b/THREAT_MODEL.md
index 0f1fb753c1..1b2f80bc4f 100644
--- a/THREAT_MODEL.md
+++ b/THREAT_MODEL.md
@@ -247,7 +247,7 @@ Per-surface trust table:
| Gremlin Server — bytecode/traversal request | serialized traversal bytecode
| **yes**, within privileges | auth; traversal-step allow-list; resource limits
|
| Gremlin Server — session id (`SessionOpProcessor`) | client-supplied session
string | **yes** | keyed by the string with no owning-user check, so any client
presenting the id shares the session (see §11b) |
| Request deserialization (GraphSON / GraphBinary) | serialized bytes |
**yes** (pre-auth) | robustness of the wire serializers |
-| Graph IO — Gryo/GraphSON/GraphML files (`io()` step, persistence, OLAP) |
on-disk / cluster bytes | only if the caller loads untrusted files | GraphSON,
locked Gryo (`registrationRequired=true`), and GraphML with the default XML
factory owe deserializer integrity. Unlocked Gryo and a caller-supplied
unhardened `XMLInputFactory` (XXE) are the caller's responsibility |
+| Graph IO — Gryo/GraphSON/GraphML files (`io()` step, persistence, OLAP) |
on-disk / cluster bytes | only if the caller loads untrusted files | GraphSON,
the hardened Gryo mappers the IO paths build (`registrationRequired=true` plus
`javaSerializationAllowed=false`), and GraphML with the default XML factory owe
deserializer integrity. Unlocked Gryo, a Gryo mapper that keeps Java
serialization, and a caller-supplied unhardened `XMLInputFactory` (XXE) are the
caller's responsibility |
| Gremlin string parser (`gremlin-language` ANTLR) | Gremlin string | **yes**
| parser robustness, no crash/hang/OOM on malformed input and no grammar
breakout / step injection (distinct from execution cost, §8/Q7) |
| Any string the grammar accepts as an argument (e.g. a `regex` pattern) |
Gremlin string / bytecode | **yes** | a grammatically valid string must not
enable DoS (e.g. ReDoS via a pathological pattern), the Q7
super-linear-amplification carve-out (§8) |
| GLV (client) — server response | serialized bytes from the server | yes if
the server is malicious/compromised, or a MITM (TLS off / cert not validated) |
response-deserialization robustness; TLS with cert validation |
@@ -337,14 +337,18 @@ Per-surface trust table:
Groovy string by concatenation, which is the calling application's concern
(§9). *Violation symptom:*
grammar breakout / step injection from a value that should stay a literal.
*Severity:* critical.
-- **Deserializer integrity.** The wire deserializers (GraphSON, GraphBinary)
and **Gryo in its locked default
- (`registrationRequired=true`)** reading attacker bytes do not lead to
arbitrary object instantiation / code
- execution beyond the registered type set. Because `inject()` and value
arguments let a request carry any
+- **Deserializer integrity.** The wire deserializers (GraphSON, GraphBinary)
and **the hardened Gryo mappers the
+ IO paths build** (`registrationRequired=true` plus
`javaSerializationAllowed=false`, i.e. `io()`, `GryoReader`,
+ `GryoWriter`, `GryoIo`, and the Hadoop Gryo input/output formats) reading
attacker bytes do not reach native Java
+ deserialization
+ (`ObjectInputStream.readObject()`). Because `inject()` and value arguments
let a request carry any
supported type, a bug in a **registered** type's (de)serializer that
crashes/OOMs the reader is also
in-model, on **both** the server (request) and the GLV (response) side. The
GraphML reader disables
external entities and DTDs by default (XXE-safe). *Violation symptom:*
deserialization gadget / RCE / XXE,
or a registered-type serializer crashing/OOMing either end. *Severity:*
critical. Gryo is not on the wire,
- and unlocked Gryo or a caller-supplied unhardened XML factory is
out-of-model (user responsibility, §9).
+ and unlocked Gryo (as run by `spark-gremlin` and the Hadoop object pools), a
directly built `GryoMapper` or
+ `GryoPool` that keeps Java serialization, or a caller-supplied unhardened
XML factory is out-of-model
+ (user responsibility, §9).
- **Resource bounds — split.** Malformed/pre-auth input that
crashes/OOMs/hangs the server is **in-model**
(above). Ordinary expensive traversals / large results are **operator
capacity**, NOT in-model, unless a
@@ -394,7 +398,12 @@ Per-surface trust table:
allow-list (`registrationRequired=true`), so it is not an
arbitrary-instantiation sink, and a break within
that locked config is a `VALID` bug like any deserializer. **Running Gryo
unlocked
(`registrationRequired=false`) is not a safe boundary against untrusted
bytes and is the user's
- responsibility.** (A few registered types use Java native serialization, a
gadget caveat even when locked.)
+ responsibility.** A few registered types are also serialized with Kryo's
`JavaSerializer`, which reads by way of
+ `ObjectInputStream.readObject()` and is a gadget caveat even when locked.
The mappers the IO paths build
+ (`io()`, `GryoReader`, `GryoWriter`, `GryoIo`, and the Hadoop Gryo
input/output formats) drop those registrations
+ (`GryoMapper.Builder.javaSerializationAllowed(boolean)` selects the
behavior), so a break there is `VALID`. A
+ directly built `GryoMapper` and `GryoPool` keep them, as do the
`spark-gremlin` and Hadoop object pools that
+ additionally run unlocked; those remain the user's responsibility.
Gryo is not on the wire, so this is an IO/file-surface concern (`io()` step,
persistence, OLAP).
- **A `TraversalStrategy` is not an access-control boundary on its own.** A
remote request can remove or
diff --git a/docs/src/upgrade/release-3.7.x.asciidoc
b/docs/src/upgrade/release-3.7.x.asciidoc
index 2c8f82cade..703eccf077 100644
--- a/docs/src/upgrade/release-3.7.x.asciidoc
+++ b/docs/src/upgrade/release-3.7.x.asciidoc
@@ -33,6 +33,49 @@ complete list of all the modifications that are part of this
release.
=== Upgrading for Users
+==== Gryo IO Disables Java Serialization
+
+A small number of Gryo type registrations, mostly `TraversalStrategy`
implementations, were registered with Kryo's
+`JavaSerializer`, which reads by way of
`java.io.ObjectInputStream.readObject()`. Reading a Gryo document from an
+untrusted source was therefore unsafe, as a crafted document could carry an
arbitrary Java object graph that runs
+during decoding.
+
+The paths that consume graph documents, `io()`, `GryoReader`, `GryoWriter` and
`GryoIo` (including `gryo` graph
+persistence) and the Hadoop Gryo input/output formats, no longer install those
registrations. A graph document only needs graph structure, so nothing is lost
+in practice. A stream that does present one of the affected type ids now fails
with an unregistered class id:
+
+[source,text]
+----
+org.apache.tinkerpop.shaded.kryo.KryoException: Encountered unregistered class
ID: 187
+----
+
+The affected types are `PartitionStrategy`, `SubgraphStrategy`,
`SeedStrategy`, `VertexProgramStrategy`,
+`ProductiveByStrategy`, `OptionsStrategy` and `TraversalExplanation` in both
Gryo 1.0 and 3.0, plus
+`GroupStep.GroupBiOperator` and `OrderGlobalStep.OrderBiOperator` in Gryo 1.0.
+
+A mapper built directly is unchanged and keeps full fidelity for trusted,
in-process work such as OLAP. The new
+`GryoMapper.Builder.javaSerializationAllowed(boolean)` selects the behavior:
+
+[source,java]
+----
+// full fidelity, unchanged, appropriate for trusted bytes only
+GryoMapper.build().create();
+
+// hardened, what io() and GryoReader/GryoWriter now use by default
+GryoMapper.build().javaSerializationAllowed(false).create();
+----
+
+Callers that supply their own mapper to `GryoReader` or `GryoWriter` are
unaffected and should add
+`javaSerializationAllowed(false)` if they read bytes they do not control.
`GryoIo` accepts an `onMapper` consumer, so
+full fidelity can be restored where the bytes are trusted:
+
+[source,java]
+----
+GryoIo.build().graph(graph).onMapper(m -> ((GryoMapper.Builder)
m).javaSerializationAllowed(true)).create();
+----
+
+See: link:https://issues.apache.org/jira/browse/TINKERPOP-3278[TINKERPOP-3278]
+
==== conjoin() Step Null Handling
The `conjoin()` step previously returned `null` when elements in the incoming
list are `null`. This behavior has
diff --git
a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/sideEffect/IoStep.java
b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/sideEffect/IoStep.java
index e469e65c68..d42d477f88 100644
---
a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/sideEffect/IoStep.java
+++
b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/sideEffect/IoStep.java
@@ -151,7 +151,8 @@ public class IoStep<S> extends AbstractStep<S,S> implements
ReadWriting {
detectRegistries().forEach(builder::addRegistry);
return
GraphSONReader.build().mapper(builder.create()).create();
} else if (objectOrClass.equals(IO.gryo)){
- final GryoMapper.Builder builder = GryoMapper.build();
+ // io() reads bytes the caller may not control, so refuse
native Java serialization; a graph document only needs graph structure
+ final GryoMapper.Builder builder =
GryoMapper.build().javaSerializationAllowed(false);
detectRegistries().forEach(builder::addRegistry);
return GryoReader.build().mapper(builder.create()).create();
} else if (objectOrClass.equals(IO.graphml))
@@ -185,7 +186,8 @@ public class IoStep<S> extends AbstractStep<S,S> implements
ReadWriting {
detectRegistries().forEach(builder::addRegistry);
return
GraphSONWriter.build().mapper(builder.create()).create();
} else if (objectOrClass.equals(IO.gryo)){
- final GryoMapper.Builder builder = GryoMapper.build();
+ // kept symmetrical with the reader above so that io() cannot
write a document it will not read
+ final GryoMapper.Builder builder =
GryoMapper.build().javaSerializationAllowed(false);
detectRegistries().forEach(builder::addRegistry);
return GryoWriter.build().mapper(builder.create()).create();
} else if (objectOrClass.equals(IO.graphml))
diff --git
a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/structure/io/gryo/GryoIo.java
b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/structure/io/gryo/GryoIo.java
index 4c9fc71934..e608f6518d 100644
---
a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/structure/io/gryo/GryoIo.java
+++
b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/structure/io/gryo/GryoIo.java
@@ -70,7 +70,9 @@ public final class GryoIo implements Io<GryoReader.Builder,
GryoWriter.Builder,
*/
@Override
public GryoMapper.Builder mapper() {
- final GryoMapper.Builder builder = GryoMapper.build().version(version);
+ // as with GryoReader/GryoWriter, graph documents may be untrusted, so
native Java serialization is
+ // disabled. An onMapper consumer can restore it for trusted,
in-process work.
+ final GryoMapper.Builder builder =
GryoMapper.build().version(version).javaSerializationAllowed(false);
onMapper.ifPresent(c -> c.accept(builder));
return builder;
}
diff --git
a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/structure/io/gryo/GryoMapper.java
b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/structure/io/gryo/GryoMapper.java
index aeaa417a04..502cc52b7d 100644
---
a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/structure/io/gryo/GryoMapper.java
+++
b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/structure/io/gryo/GryoMapper.java
@@ -29,6 +29,7 @@ import org.apache.tinkerpop.shaded.kryo.Kryo;
import org.apache.tinkerpop.shaded.kryo.Serializer;
import org.apache.tinkerpop.shaded.kryo.util.DefaultStreamFactory;
import org.apache.tinkerpop.shaded.kryo.util.MapReferenceResolver;
+import org.apache.tinkerpop.shaded.kryo.serializers.JavaSerializer;
import org.javatuples.Pair;
import java.util.ArrayList;
@@ -78,10 +79,12 @@ public final class GryoMapper implements Mapper<Kryo> {
private final List<TypeRegistration<?>> typeRegistrations;
private final boolean registrationRequired;
private final boolean referenceTracking;
+ private final boolean javaSerializationAllowed;
private final Supplier<ClassResolver> classResolver;
private final GryoVersion version;
private GryoMapper(final Builder builder) {
+ this.javaSerializationAllowed = builder.javaSerializationAllowed;
this.typeRegistrations = builder.typeRegistrations;
this.version = builder.version;
validate();
@@ -97,11 +100,30 @@ public final class GryoMapper implements Mapper<Kryo> {
kryo.addDefaultSerializer(Map.Entry.class, new
UtilSerializers.EntrySerializer());
kryo.setRegistrationRequired(registrationRequired);
kryo.setReferences(referenceTracking);
- for (TypeRegistration tr : typeRegistrations)
+ for (TypeRegistration tr : typeRegistrations) {
+ // drop any registration that resolves to a JavaSerializer; the
Function- and default-serializer
+ // forms can only be inspected once a Kryo exists, which is why
this is done here rather than up front
+ if (!javaSerializationAllowed && resolvesToJavaSerializer(tr,
kryo)) continue;
tr.registerWith(kryo);
+ }
return kryo;
}
+ /**
+ * Detects a registration whose serializer is, or resolves to, Kryo's
{@code JavaSerializer}. A directly supplied
+ * serializer is known statically, while a {@code Function} or a class
default serializer can only be resolved
+ * from a live {@link Kryo} instance, which is why this runs at {@link
#createMapper()} time.
+ */
+ private static boolean resolvesToJavaSerializer(final TypeRegistration<?>
tr, final Kryo kryo) {
+ if (null != tr.getShadedSerializer())
+ return tr.getShadedSerializer() instanceof JavaSerializer;
+ if (null != tr.getFunctionOfShadedKryo())
+ return tr.getFunctionOfShadedKryo().apply(kryo) instanceof
JavaSerializer;
+ if (!tr.hasSerializer())
+ return kryo.getDefaultSerializer(tr.getTargetClass()) instanceof
JavaSerializer;
+ return false;
+ }
+
public GryoVersion getVersion() {
return version;
}
@@ -110,6 +132,11 @@ public final class GryoMapper implements Mapper<Kryo> {
return
this.typeRegistrations.stream().map(TypeRegistration::getTargetClass).collect(Collectors.toList());
}
+ /**
+ * Note that when {@link Builder#javaSerializationAllowed(boolean)} is
{@code false}, registrations whose
+ * serializer is (or resolves to) a {@code JavaSerializer} are still
listed here even though
+ * {@link #createMapper()} skips installing them.
+ */
public List<TypeRegistration<?>> getTypeRegistrations() {
return typeRegistrations;
}
@@ -155,6 +182,7 @@ public final class GryoMapper implements Mapper<Kryo> {
private boolean registrationRequired = true;
private boolean referenceTracking = true;
+ private boolean javaSerializationAllowed = true;
private Supplier<ClassResolver> classResolver;
private Builder() {
@@ -248,6 +276,28 @@ public final class GryoMapper implements Mapper<Kryo> {
return this;
}
+ /**
+ * When set to {@code false}, every registration whose serializer is
Kryo's {@code JavaSerializer} is dropped,
+ * producing a mapper suited to reading bytes from an untrusted
source. By default this value is {@code true}.
+ * <p/>
+ * That serializer reads by way of {@code
java.io.ObjectInputStream.readObject()}, which reconstructs and runs
+ * an arbitrary {@code Serializable} object graph while decoding,
before the graph layer can accept or reject
+ * anything. The affected types are mostly {@code TraversalStrategy}
implementations that a graph document does
+ * not need; a stream that carries one now fails with an unregistered
class id. Registrations contributed
+ * through an {@link IoRegistry} or {@code addCustom(...)} are covered
on the same terms, including those
+ * whose serializer is a {@code Function} or a class default that
resolves to a {@code JavaSerializer}. This
+ * relies on the default {@link #registrationRequired(boolean)} of
{@code true}. Callers that need the full
+ * fidelity for trusted, in-process work should leave this value at
{@code true}.
+ *
+ * @param javaSerializationAllowed set to {@code false} to drop the
{@code JavaSerializer} registrations or
+ * {@code true} to keep them
+ * @see #registrationRequired(boolean)
+ */
+ public Builder javaSerializationAllowed(final boolean
javaSerializationAllowed) {
+ this.javaSerializationAllowed = javaSerializationAllowed;
+ return this;
+ }
+
/**
* By default, each appearance of an object in the graph after the
first is stored as an integer ordinal.
* This allows multiple references to the same object and cyclic
graphs to be serialized. This has a small
diff --git
a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/structure/io/gryo/GryoReader.java
b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/structure/io/gryo/GryoReader.java
index 68f130a26e..09739f53a9 100644
---
a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/structure/io/gryo/GryoReader.java
+++
b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/structure/io/gryo/GryoReader.java
@@ -274,9 +274,10 @@ public final class GryoReader implements GraphReader {
private long batchSize = 10000;
/**
- * Always use the most recent gryo version by default
+ * Always use the most recent gryo version by default. Graph documents
read here may come from an untrusted
+ * source, so native Java serialization is disabled. Supply a mapper
explicitly to opt back in.
*/
- private Mapper<Kryo> gryoMapper = GryoMapper.build().create();
+ private Mapper<Kryo> gryoMapper =
GryoMapper.build().javaSerializationAllowed(false).create();
private Builder() {
}
diff --git
a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/structure/io/gryo/GryoWriter.java
b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/structure/io/gryo/GryoWriter.java
index 8172ba90b3..3451e0b270 100644
---
a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/structure/io/gryo/GryoWriter.java
+++
b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/structure/io/gryo/GryoWriter.java
@@ -161,9 +161,10 @@ public final class GryoWriter implements GraphWriter {
public final static class Builder implements WriterBuilder<GryoWriter> {
/**
- * Always creates the most current version available.
+ * Always creates the most current version available. Native Java
serialization is disabled here as it is on
+ * the {@link GryoReader} default, so that a document written here can
always be read back.
*/
- private Mapper<Kryo> gryoMapper = GryoMapper.build().create();
+ private Mapper<Kryo> gryoMapper =
GryoMapper.build().javaSerializationAllowed(false).create();
private Builder() {
}
diff --git
a/gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/structure/io/gryo/GryoMapperTest.java
b/gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/structure/io/gryo/GryoMapperTest.java
index 8951281053..5de7fa31ec 100644
---
a/gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/structure/io/gryo/GryoMapperTest.java
+++
b/gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/structure/io/gryo/GryoMapperTest.java
@@ -22,28 +22,36 @@ import
org.apache.tinkerpop.gremlin.process.remote.traversal.DefaultRemoteTraver
import org.apache.tinkerpop.gremlin.process.traversal.Bytecode;
import org.apache.tinkerpop.gremlin.process.traversal.Merge;
import org.apache.tinkerpop.gremlin.process.traversal.TextP;
+import
org.apache.tinkerpop.gremlin.process.traversal.strategy.decoration.OptionsStrategy;
import
org.apache.tinkerpop.gremlin.process.traversal.util.TraversalExplanation;
import org.apache.tinkerpop.gremlin.structure.Vertex;
+import org.apache.tinkerpop.gremlin.structure.io.Io;
import org.apache.tinkerpop.gremlin.structure.io.IoX;
import org.apache.tinkerpop.gremlin.structure.io.IoXIoRegistry;
import org.apache.tinkerpop.gremlin.structure.io.IoY;
import org.apache.tinkerpop.gremlin.structure.io.IoYIoRegistry;
import org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTokens;
import org.apache.tinkerpop.gremlin.structure.util.detached.DetachedVertex;
+import org.apache.tinkerpop.gremlin.structure.util.empty.EmptyGraph;
import org.apache.tinkerpop.shaded.kryo.ClassResolver;
import org.apache.tinkerpop.shaded.kryo.Kryo;
import org.apache.tinkerpop.shaded.kryo.Registration;
import org.apache.tinkerpop.shaded.kryo.Serializer;
import org.apache.tinkerpop.shaded.kryo.io.Input;
import org.apache.tinkerpop.shaded.kryo.io.Output;
+import org.apache.tinkerpop.shaded.kryo.serializers.JavaSerializer;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
+import java.io.IOException;
import java.io.InputStream;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
import java.io.OutputStream;
+import java.io.Serializable;
import java.net.InetAddress;
import java.nio.ByteBuffer;
import java.sql.Timestamp;
@@ -65,13 +73,17 @@ import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.function.Function;
import java.util.function.Supplier;
import static org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.__.__;
import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.greaterThan;
import static org.hamcrest.core.Is.is;
import static org.hamcrest.core.IsInstanceOf.instanceOf;
import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
import static org.junit.Assert.assertNotEquals;
import static org.junit.Assert.assertNotSame;
import static org.junit.Assert.fail;
@@ -95,6 +107,23 @@ public class GryoMapperTest {
@Parameterized.Parameter(value = 1)
public Supplier<GryoMapper.Builder> builder;
+ /**
+ * The Gryo type id of {@code OptionsStrategy}, registered with the shaded
{@code JavaSerializer} in both
+ * {@link GryoVersion#V1_0} and {@link GryoVersion#V3_0}.
+ */
+ private static final int OPTIONS_STRATEGY_GRYO_ID = 187;
+
+ /**
+ * Kryo shifts written class ids to leave room for its {@code NULL} and
{@code NAME} markers, as
+ * {@link AbstractGryoClassResolver#readClass(Input)} shows.
+ */
+ private static final int CLASS_ID_OFFSET = 2;
+
+ /**
+ * Kryo's reference marker for an object being seen for the first time.
+ */
+ private static final int KRYO_NOT_NULL = 1;
+
@Test
public void shouldMakeNewInstance() {
final GryoMapper.Builder b = GryoMapper.build();
@@ -349,6 +378,306 @@ public class GryoMapperTest {
assertEquals(te.toString(), serializeDeserialize(te,
TraversalExplanation.class).toString());
}
+ /**
+ * Gryo registers a handful of types with the shaded Kryo {@code
JavaSerializer}, which deserializes by way of
+ * {@code java.io.ObjectInputStream.readObject()}. A stream that presents
one of those type ids therefore
+ * reconstructs whatever {@code Serializable} object graph follows and
runs its {@code readObject()} methods,
+ * which is an unsafe-deserialization sink on caller-supplied bytes. The
canary used here carries no
+ * payload at all: the mere execution of its {@code readObject()} is the
proof.
+ */
+ @Test
+ public void shouldNotInvokeJavaDeserializationOnGryoRead() throws
Exception {
+ final Kryo kryo =
builder.get().javaSerializationAllowed(false).create().createMapper();
+
+ // Kryo frames an object as varint(class id + 2) followed by a
NOT_NULL reference marker. Confirm that
+ // framing against a known registration (HashMap is id 11 in both V1_0
and V3_0) rather than trusting a
+ // hard coded Kryo internal, so the crafted stream below cannot
silently stop reaching the serializer.
+ final int hashMapId = kryo.getRegistration(HashMap.class).getId();
+ assertEquals(11, hashMapId);
+ final Output probe = new Output(64, -1);
+ kryo.writeClassAndObject(probe, new HashMap<String, Object>());
+ probe.flush();
+ final Input probeInput = new Input(probe.toBytes());
+ assertEquals(hashMapId + CLASS_ID_OFFSET, probeInput.readVarInt(true));
+ assertEquals(KRYO_NOT_NULL, probeInput.readVarInt(true));
+
+ final byte[] malicious = maliciousGryoBytes();
+
+ DeserializationCanary.FIRED = false;
+ try {
+ kryo.readClassAndObject(new Input(new
ByteArrayInputStream(malicious)));
+ } catch (Exception ignored) {
+ // refusing the stream outright is the expected outcome. what
matters is that nothing was deserialized on
+ // the way to that decision.
+ }
+
+ assertFalse("Reading Gryo " + name + " bytes must not invoke
ObjectInputStream.readObject() on the stream, " +
+ "since a crafted stream presenting a
JavaSerializer-backed type id such as " +
+ OPTIONS_STRATEGY_GRYO_ID + " (OptionsStrategy) could
otherwise carry an arbitrary Java " +
+ "object graph",
+ DeserializationCanary.FIRED);
+ }
+
+ /**
+ * Positive control for {@link
#shouldNotInvokeJavaDeserializationOnGryoRead()}. Without hardening, the same
bytes
+ * do reach {@code ObjectInputStream.readObject()}, which is what gives
the assertion there any meaning: were the
+ * crafted framing ever to stop selecting the {@code JavaSerializer}, this
test would fail and say so. It also
+ * documents that a directly built mapper stays full fidelity and must
only be pointed at trusted bytes.
+ */
+ @Test
+ public void shouldInvokeJavaDeserializationOnDefaultMapperRead() throws
Exception {
+ final Kryo kryo = builder.get().create().createMapper();
+ final byte[] malicious = maliciousGryoBytes();
+
+ DeserializationCanary.FIRED = false;
+ try {
+ kryo.readClassAndObject(new Input(new
ByteArrayInputStream(malicious)));
+ } catch (Exception ignored) {
+ // the payload deserializes to the canary rather than to an
OptionsStrategy, so a failure is possible
+ // here, but it would come after readObject() has already run
+ }
+
+ assertTrue("the crafted stream must reach
ObjectInputStream.readObject() on a full fidelity mapper, " +
+ "otherwise the hardened assertions prove nothing",
+ DeserializationCanary.FIRED);
+ }
+
+ /**
+ * The same crafted stream fed through the reader that {@code io()} and
graph persistence use, whose default
+ * mapper is hardened.
+ */
+ @Test
+ public void shouldNotInvokeJavaDeserializationOnGryoReaderRead() throws
Exception {
+ final GryoReader reader = GryoReader.build().create();
+
+ DeserializationCanary.FIRED = false;
+ try (final InputStream stream = new
ByteArrayInputStream(maliciousGryoBytes())) {
+ reader.readObject(stream, Object.class);
+ } catch (Exception ignored) {
+ // as above, refusing the stream is the expected outcome
+ }
+
+ assertFalse("GryoReader must not invoke ObjectInputStream.readObject()
on the bytes it reads",
+ DeserializationCanary.FIRED);
+ }
+
+ /**
+ * Hardening the mapper must not cost anything on the graph structure that
a Gryo document actually carries.
+ */
+ @Test
+ public void shouldRoundTripGraphStructureWithJavaSerializationDisabled()
throws Exception {
+ final Kryo kryo =
builder.get().javaSerializationAllowed(false).create().createMapper();
+
+ final Map<String, Object> props = new HashMap<>();
+ final List<Map<String, Object>> propertyNames = new ArrayList<>(1);
+ final Map<String, Object> propertyName = new HashMap<>();
+ propertyName.put(GraphSONTokens.ID, "x");
+ propertyName.put(GraphSONTokens.KEY, "x");
+ propertyName.put(GraphSONTokens.VALUE, "no-way-this-will-ever-work");
+ propertyNames.add(propertyName);
+ props.put("x", propertyNames);
+
+ final Output out = new Output(1024, -1);
+ kryo.writeClassAndObject(out, new DetachedVertex(100,
Vertex.DEFAULT_LABEL, props));
+ out.flush();
+
+ final DetachedVertex readX = (DetachedVertex) kryo.readClassAndObject(
+ new Input(new ByteArrayInputStream(out.toBytes())));
+ assertEquals("no-way-this-will-ever-work", readX.value("x"));
+ }
+
+ /**
+ * A Gryo stream that presents {@code OptionsStrategy}'s type id and then
a raw Java-serialized payload. Crafting
+ * it needs no cooperation from the Gryo writer, which is why the sink was
reachable from untrusted bytes.
+ */
+ private byte[] maliciousGryoBytes() throws Exception {
+ final ByteArrayOutputStream javaPayload = new ByteArrayOutputStream();
+ try (final ObjectOutputStream oos = new
ObjectOutputStream(javaPayload)) {
+ oos.writeObject(new DeserializationCanary());
+ }
+
+ final Output malicious = new Output(javaPayload.size() + 64, -1);
+ malicious.writeVarInt(OPTIONS_STRATEGY_GRYO_ID + CLASS_ID_OFFSET,
true);
+ malicious.writeVarInt(KRYO_NOT_NULL, true);
+ malicious.writeBytes(javaPayload.toByteArray());
+ malicious.flush();
+ return malicious.toBytes();
+ }
+
+ /**
+ * Companion to {@link #shouldNotInvokeJavaDeserializationOnGryoRead()}
that covers the whole sink surface rather
+ * than one carrier type. The assertion is made against the {@code Kryo}
instance that actually decodes bytes, so
+ * that it cannot pass merely by asking the same question of the same
metadata the filter itself used.
+ */
+ @Test
+ public void shouldNotRegisterTypesWithJavaSerializerWhenDisabled() {
+ final Kryo hardened =
builder.get().javaSerializationAllowed(false).create().createMapper();
+
+ for (final TypeRegistration<?> tr : javaSerializedRegistrations()) {
+ final Class<?> clazz = tr.getTargetClass();
+ try {
+ hardened.getRegistration(clazz);
+ fail(clazz.getSimpleName() + " must not be registered on a
hardened mapper");
+ } catch (IllegalArgumentException expected) {
+ // Kryo refuses an unregistered class while registration is
required
+ }
+ }
+ }
+
+ /**
+ * A custom type contributed with Kryo's {@code JavaSerializer} is
filtered on the same terms as the defaults,
+ * since an {@link org.apache.tinkerpop.gremlin.structure.io.IoRegistry}
is an untrusted-input path too.
+ */
+ @Test
+ public void shouldNotRegisterCustomTypesWithJavaSerializerWhenDisabled() {
+ final Kryo hardened = builder.get().addCustom(IoX.class, new
JavaSerializer()).
+ javaSerializationAllowed(false).create().createMapper();
+
+ try {
+ hardened.getRegistration(IoX.class);
+ fail("a custom JavaSerializer registration must not survive on a
hardened mapper");
+ } catch (IllegalArgumentException expected) {
+ // as above
+ }
+ }
+
+ /**
+ * A custom type whose serializer is supplied as a {@code Function}
resolving to a {@code JavaSerializer} can only
+ * be recognized once a {@code Kryo} exists, so it is dropped at
mapper-creation time rather than at build time.
+ */
+ @Test
+ public void
shouldNotRegisterCustomFunctionTypesWithJavaSerializerWhenDisabled() {
+ final Kryo hardened = builder.get().addCustom(IoX.class,
(Function<Kryo, Serializer>) k -> new JavaSerializer()).
+ javaSerializationAllowed(false).create().createMapper();
+
+ try {
+ hardened.getRegistration(IoX.class);
+ fail("a custom Function supplied JavaSerializer must not survive
on a hardened mapper");
+ } catch (IllegalArgumentException expected) {
+ // Kryo refuses an unregistered class while registration is
required
+ }
+ }
+
+ /**
+ * A type carrying {@code @DefaultSerializer(JavaSerializer.class)} and
registered without an explicit serializer
+ * resolves to a {@code JavaSerializer} through Kryo's default, which is
likewise dropped at mapper-creation time.
+ */
+ @Test
+ public void
shouldNotRegisterDefaultSerializerJavaSerializerTypesWhenDisabled() {
+ final Kryo hardened =
builder.get().addCustom(JavaSerializedByDefault.class).
+ javaSerializationAllowed(false).create().createMapper();
+
+ try {
+ hardened.getRegistration(JavaSerializedByDefault.class);
+ fail("a @DefaultSerializer(JavaSerializer) registration must not
survive on a hardened mapper");
+ } catch (IllegalArgumentException expected) {
+ // as above
+ }
+ }
+
+ /**
+ * The full fidelity mapper is unchanged and remains available for
trusted, in-process round-trips. This test
+ * documents which registrations that leaves on native Java serialization.
+ */
+ @Test
+ public void shouldRegisterTypesWithJavaSerializerByDefault() {
+ final List<String> found = new ArrayList<>();
+ for (final TypeRegistration<?> tr : javaSerializedRegistrations())
+ found.add(String.format("%s(%d)",
tr.getTargetClass().getSimpleName(), tr.getId()));
+
+ final List<String> expected = name.equals("1_0") ?
+ Arrays.asList("TraversalExplanation(106)",
"GroupBiOperator(117)", "OrderBiOperator(118)",
+ "PartitionStrategy(140)", "SubgraphStrategy(141)",
"SeedStrategy(192)",
+ "VertexProgramStrategy(142)",
"ProductiveByStrategy(195)", "OptionsStrategy(187)") :
+ Arrays.asList("PartitionStrategy(140)",
"SubgraphStrategy(141)", "SeedStrategy(192)",
+ "VertexProgramStrategy(142)",
"ProductiveByStrategy(195)", "OptionsStrategy(187)",
+ "TraversalExplanation(106)");
+ assertEquals(expected, found);
+ }
+
+ /**
+ * The inverse of {@link
#shouldNotRegisterTypesWithJavaSerializerWhenDisabled()}. Setting the value
explicitly
+ * keeps the affected types usable, which is what a trusted, in-process
round-trip relies on.
+ */
+ @Test
+ public void shouldRoundTripStrategyWhenJavaSerializationAllowed() throws
Exception {
+ final Kryo kryo =
builder.get().javaSerializationAllowed(true).create().createMapper();
+
+ final Output out = new Output(1024, -1);
+ kryo.writeClassAndObject(out, OptionsStrategy.build().with("some-key",
"some-value").create());
+ out.flush();
+
+ final OptionsStrategy read = (OptionsStrategy) kryo.readClassAndObject(
+ new Input(new ByteArrayInputStream(out.toBytes())));
+ assertEquals("some-value", read.getOptions().get("some-key"));
+ }
+
+ /**
+ * {@link GryoIo} hardens its mapper, and the {@code onMapper} consumer is
the documented way to restore full
+ * fidelity where the bytes are trusted. This pins the form shown in the
upgrade documentation, including the cast.
+ */
+ @Test
+ public void shouldRestoreJavaSerializationThroughGryoIoOnMapper() {
+ final Io.Builder<GryoIo> io = GryoIo.build(gryoVersion());
+ io.graph(EmptyGraph.instance());
+ io.onMapper(m -> ((GryoMapper.Builder)
m).javaSerializationAllowed(true));
+
+ final Kryo restored = io.create().mapper().create().createMapper();
+ assertEquals(OPTIONS_STRATEGY_GRYO_ID,
restored.getRegistration(OptionsStrategy.class).getId());
+ }
+
+ /**
+ * Without such a consumer, {@link GryoIo} is hardened like the reader and
writer defaults.
+ */
+ @Test
+ public void shouldNotRegisterTypesWithJavaSerializerOnGryoIoDefault() {
+ final Io.Builder<GryoIo> io = GryoIo.build(gryoVersion());
+ io.graph(EmptyGraph.instance());
+
+ final Kryo hardened = io.create().mapper().create().createMapper();
+ try {
+ hardened.getRegistration(OptionsStrategy.class);
+ fail("GryoIo must not register OptionsStrategy by default");
+ } catch (IllegalArgumentException expected) {
+ // Kryo refuses an unregistered class while registration is
required
+ }
+ }
+
+ /**
+ * The writer default is hardened too, so a document carrying one of the
dropped types cannot be produced by the
+ * paths that could not read it back.
+ */
+ @Test
+ public void shouldNotWriteTypesWithJavaSerializerOnGryoWriterDefault()
throws Exception {
+ final GryoWriter writer = GryoWriter.build().create();
+
+ try (final OutputStream stream = new ByteArrayOutputStream()) {
+ writer.writeObject(stream,
OptionsStrategy.build().with("some-key", "some-value").create());
+ fail("the GryoWriter default must not write a JavaSerializer
backed type");
+ } catch (IllegalArgumentException expected) {
+ // as above, Kryo refuses the unregistered class
+ }
+ }
+
+ private GryoVersion gryoVersion() {
+ return name.equals("1_0") ? GryoVersion.V1_0 : GryoVersion.V3_0;
+ }
+
+ /**
+ * The registrations that the full fidelity mapper of the version under
test backs with Kryo's
+ * {@code JavaSerializer}, each of which is a carrier for the sink.
+ */
+ private List<TypeRegistration<?>> javaSerializedRegistrations() {
+ final List<TypeRegistration<?>> found = new ArrayList<>();
+ for (final TypeRegistration<?> tr :
builder.get().create().getTypeRegistrations()) {
+ if (tr.getShadedSerializer() instanceof JavaSerializer)
found.add(tr);
+ }
+
+ // if detection ever breaks, the tests that loop over this would pass
without checking anything
+ assertThat(found.size(), greaterThan(0));
+ return found;
+ }
+
@Test
public void shouldHandleBytecode() throws Exception {
final Bytecode bytecode =
__().out().outV().outE().asAdmin().getBytecode();
@@ -456,4 +785,27 @@ public class GryoMapperTest {
}
}
+ /**
+ * A type that resolves to Kryo's {@code JavaSerializer} through the
class-level {@code @DefaultSerializer}
+ * annotation rather than an explicit registration, exercising the
default-serializer branch of the filter.
+ */
+ @org.apache.tinkerpop.shaded.kryo.DefaultSerializer(JavaSerializer.class)
+ public static class JavaSerializedByDefault implements Serializable {
+ private static final long serialVersionUID = 1L;
+ }
+
+ /**
+ * A deliberately inert {@code Serializable} used to detect whether native
Java deserialization ran during a Gryo
+ * read. It touches nothing outside this class: no process execution, no
filesystem, no reflection.
+ */
+ private static class DeserializationCanary implements Serializable {
+ private static final long serialVersionUID = 1L;
+
+ static volatile boolean FIRED = false;
+
+ private void readObject(final ObjectInputStream in) throws
IOException, ClassNotFoundException {
+ in.defaultReadObject();
+ FIRED = true;
+ }
+ }
}
diff --git
a/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/structure/io/HadoopPools.java
b/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/structure/io/HadoopPools.java
index 2284f279f6..7201b4acd2 100644
---
a/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/structure/io/HadoopPools.java
+++
b/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/structure/io/HadoopPools.java
@@ -47,6 +47,8 @@ public final class HadoopPools {
poolSize(configuration.getInt(GryoPool.CONFIG_IO_GRYO_POOL_SIZE, 256)).
version(GryoVersion.valueOf(configuration.getString(GryoPool.CONFIG_IO_GRYO_VERSION,
GryoPool.CONFIG_IO_GRYO_POOL_VERSION_DEFAULT.name()))).
ioRegistries(configuration.getList(IoRegistry.IO_REGISTRY,
Collections.emptyList())).
+ // intra-job shuffle path (VertexWritable), not untrusted
input, so it is left unlocked and keeps the
+ // JavaSerializer registrations. The Gryo IO formats are
the hardened untrusted-file path.
initializeMapper(m -> m.registrationRequired(false)).
create();
}
diff --git
a/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/structure/io/gryo/GryoRecordReader.java
b/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/structure/io/gryo/GryoRecordReader.java
index 8fead07f4b..f904eac0f1 100644
---
a/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/structure/io/gryo/GryoRecordReader.java
+++
b/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/structure/io/gryo/GryoRecordReader.java
@@ -72,8 +72,10 @@ public final class GryoRecordReader extends
RecordReader<NullWritable, VertexWri
final Configuration configuration = context.getConfiguration();
if (configuration.get(Constants.GREMLIN_HADOOP_GRAPH_FILTER, null) !=
null)
this.graphFilter =
VertexProgramHelper.deserialize(ConfUtil.makeApacheConfiguration(configuration),
Constants.GREMLIN_HADOOP_GRAPH_FILTER);
+ // reads only graph Elements from untrusted split bytes, never the
JavaSerializer backed types, so native
+ // Java serialization is disabled here (unlike the intra-job MapReduce
shuffle path, left unlocked).
this.gryoReader = GryoReader.build().mapper(
-
GryoMapper.build().addRegistries(IoRegistryHelper.createRegistries(ConfUtil.makeApacheConfiguration(configuration))).create()).create();
+
GryoMapper.build().javaSerializationAllowed(false).addRegistries(IoRegistryHelper.createRegistries(ConfUtil.makeApacheConfiguration(configuration))).create()).create();
long start = split.getStart();
final Path file = split.getPath();
if (null != new CompressionCodecFactory(configuration).getCodec(file))
{
diff --git
a/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/structure/io/gryo/GryoRecordWriter.java
b/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/structure/io/gryo/GryoRecordWriter.java
index 983e05c5cb..34340297ca 100644
---
a/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/structure/io/gryo/GryoRecordWriter.java
+++
b/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/structure/io/gryo/GryoRecordWriter.java
@@ -46,8 +46,10 @@ public final class GryoRecordWriter extends
RecordWriter<NullWritable, VertexWri
public GryoRecordWriter(final DataOutputStream outputStream, final
Configuration configuration) {
this.outputStream = outputStream;
this.hasEdges =
configuration.getBoolean(Constants.GREMLIN_HADOOP_GRAPH_WRITER_HAS_EDGES, true);
+ // as with GryoRecordReader, this writer only serializes graph
Elements, which never include the
+ // JavaSerializer backed types, so native Java serialization is
disabled and reads/writes stay symmetric.
this.gryoWriter = GryoWriter.build().mapper(
-
GryoMapper.build().addRegistries(IoRegistryHelper.createRegistries(ConfUtil.makeApacheConfiguration(configuration))).create()).create();
+
GryoMapper.build().javaSerializationAllowed(false).addRegistries(IoRegistryHelper.createRegistries(ConfUtil.makeApacheConfiguration(configuration))).create()).create();
}
@Override
diff --git
a/hadoop-gremlin/src/test/java/org/apache/tinkerpop/gremlin/hadoop/structure/io/gryo/GryoRecordReaderWriterTest.java
b/hadoop-gremlin/src/test/java/org/apache/tinkerpop/gremlin/hadoop/structure/io/gryo/GryoRecordReaderWriterTest.java
index fb6282d942..1a45f9a556 100644
---
a/hadoop-gremlin/src/test/java/org/apache/tinkerpop/gremlin/hadoop/structure/io/gryo/GryoRecordReaderWriterTest.java
+++
b/hadoop-gremlin/src/test/java/org/apache/tinkerpop/gremlin/hadoop/structure/io/gryo/GryoRecordReaderWriterTest.java
@@ -18,11 +18,30 @@
*/
package org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.LocalFileSystem;
+import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.mapreduce.InputFormat;
import org.apache.hadoop.mapreduce.OutputFormat;
+import org.apache.hadoop.mapreduce.TaskAttemptContext;
+import org.apache.hadoop.mapreduce.TaskAttemptID;
+import org.apache.hadoop.mapreduce.TaskType;
+import org.apache.hadoop.mapreduce.lib.input.FileSplit;
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl;
+import org.apache.tinkerpop.gremlin.features.TestFiles;
import org.apache.tinkerpop.gremlin.hadoop.structure.io.RecordReaderWriterTest;
import org.apache.tinkerpop.gremlin.hadoop.structure.io.VertexWritable;
+import
org.apache.tinkerpop.gremlin.process.traversal.strategy.decoration.OptionsStrategy;
+import org.apache.tinkerpop.gremlin.structure.io.gryo.GryoReader;
+import org.apache.tinkerpop.shaded.kryo.Kryo;
+import org.junit.Test;
+
+import java.io.File;
+import java.lang.reflect.Field;
+import java.util.UUID;
+
+import static org.junit.Assert.fail;
/**
* @author Marko A. Rodriguez (http://markorodriguez.com)
@@ -44,4 +63,73 @@ public class GryoRecordReaderWriterTest extends
RecordReaderWriterTest {
protected Class<? extends OutputFormat<NullWritable, VertexWritable>>
getOutputFormat() {
return GryoOutputFormat.class;
}
+
+ /**
+ * The reader builds a hardened mapper, i.e. one without the {@code
JavaSerializer} backed registrations. The
+ * general hardening behavior is covered in gremlin-core's {@code
GryoMapperTest}; this pins the wiring here.
+ */
+ @Test
+ public void gryoRecordReaderShouldBuildAHardenedMapper() throws Exception {
+ final GryoRecordReader reader = new GryoRecordReader();
+ final File file = new File(TestFiles.PATHS.get(getInputFilename()));
+
+ final Configuration config = new Configuration(false);
+ config.set("fs.file.impl", LocalFileSystem.class.getName());
+ config.set("fs.defaultFS", "file:///");
+ final TaskAttemptContext job = new TaskAttemptContextImpl(config,
+ new TaskAttemptID(UUID.randomUUID().toString(), 0,
TaskType.MAP, 0, 0));
+
+ reader.initialize(new FileSplit(new Path(file.toURI()), 0,
file.length(), null), job);
+
+ assertMapperRefusesOptionsStrategy(readerKryo(reader));
+ }
+
+ /**
+ * The writer builds a hardened mapper on the same terms, kept symmetric
with the reader.
+ */
+ @Test
+ public void gryoRecordWriterShouldBuildAHardenedMapper() throws Exception {
+ final File outputDirectory = new
File(System.getProperty("java.io.tmpdir"),
+ "gryo-record-writer-hardening-" + UUID.randomUUID());
+ final Configuration config = new Configuration(false);
+ config.set("fs.file.impl", LocalFileSystem.class.getName());
+ config.set("fs.defaultFS", "file:///");
+ config.set("mapreduce.output.fileoutputformat.outputdir",
outputDirectory.toURI().toString());
+ final TaskAttemptContext job = new TaskAttemptContextImpl(config,
+ new TaskAttemptID(UUID.randomUUID().toString(), 0,
TaskType.REDUCE, 0, 0));
+
+ final GryoRecordWriter writer = (GryoRecordWriter) new
GryoOutputFormat().getRecordWriter(job);
+ try {
+ assertMapperRefusesOptionsStrategy(writerKryo(writer));
+ } finally {
+ writer.close(job);
+ }
+ }
+
+ private static void assertMapperRefusesOptionsStrategy(final Kryo kryo) {
+ try {
+ kryo.getRegistration(OptionsStrategy.class);
+ fail("a hardened Hadoop Gryo mapper must not register
OptionsStrategy");
+ } catch (IllegalArgumentException expected) {
+ // Kryo refuses an unregistered class while registration is
required
+ }
+ }
+
+ private static Kryo readerKryo(final GryoRecordReader reader) throws
Exception {
+ final Field gryoReaderField =
GryoRecordReader.class.getDeclaredField("gryoReader");
+ gryoReaderField.setAccessible(true);
+ final GryoReader gryoReader = (GryoReader) gryoReaderField.get(reader);
+ final Field kryoField = GryoReader.class.getDeclaredField("kryo");
+ kryoField.setAccessible(true);
+ return (Kryo) kryoField.get(gryoReader);
+ }
+
+ private static Kryo writerKryo(final GryoRecordWriter writer) throws
Exception {
+ final Field gryoWriterField =
GryoRecordWriter.class.getDeclaredField("gryoWriter");
+ gryoWriterField.setAccessible(true);
+ final Object gryoWriter = gryoWriterField.get(writer);
+ final Field kryoField = gryoWriter.getClass().getDeclaredField("kryo");
+ kryoField.setAccessible(true);
+ return (Kryo) kryoField.get(gryoWriter);
+ }
}
