[
https://issues.apache.org/jira/browse/OPENNLP-1991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18126304#comment-18126304
]
ASF GitHub Bot commented on OPENNLP-1991:
-----------------------------------------
krickert commented on code in PR #1323:
URL: https://github.com/apache/opennlp/pull/1323#discussion_r4247888619
##########
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/depparse/FeedforwardDependencyTrainer.java:
##########
@@ -0,0 +1,1176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.depparse;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.function.Function;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import opennlp.tools.util.ObjectStream;
+import opennlp.tools.util.ParamChecks;
+
+/**
+ * Trains the {@link FeedforwardDependencyModel}: oracle-derived transition
examples,
+ * minibatch AdaGrad over a softmax cross-entropy loss, cube activation, and
inverted
+ * dropout on the hidden layer, the training recipe of
+ * <a href="https://aclanthology.org/D14-1082/";>Chen and Manning (2014)</a>.
+ *
+ * <p>Words below the frequency cutoff share a learned unknown embedding;
absent
+ * template positions share a learned padding embedding. Non-projective
samples have no
+ * arc-standard derivation and are skipped. Training is deterministic for a
fixed
+ * {@link Settings#seed()}.</p>
+ *
+ * @since 3.0.0
+ */
+public final class FeedforwardDependencyTrainer {
+
+ private static final Logger logger =
+ LoggerFactory.getLogger(FeedforwardDependencyTrainer.class);
+
+ /** The AdaGrad denominator offset that keeps the first steps finite. */
+ private static final double ADAGRAD_EPSILON = 1e-6;
+
+ /** The derivative factor of the cube activation, {@code d/dx x^3 = 3x^2}. */
+ private static final double CUBE_DERIVATIVE_FACTOR = 3.0;
+
+ /** The numerator of the Glorot uniform bound, {@code sqrt(6 / (fanIn +
fanOut))}. */
+ private static final double GLOROT_NUMERATOR = 6.0;
+
+ /** The half-width of the uniform range embeddings are drawn from. */
+ private static final double EMBEDDING_INIT_SCALE = 0.01;
+
+ /** The smallest probability the reported training loss takes the log of. */
+ private static final double LOSS_PROBABILITY_FLOOR = 1e-12;
+
+ /** The sentence loss reported when refinement made no update for a
sentence. */
+ private static final double NO_UPDATE = -1.0;
+
+ /** Special-symbol rows present before any training vocabulary is added. */
+ private static final int MIN_VOCABULARY_ROWS = 8;
+
+ /** SHIFT and at least one RIGHT_ARC are required to parse a sentence. */
+ private static final int MIN_TRANSITIONS = 2;
+
+ /** Prevents construction of this utility class. */
+ private FeedforwardDependencyTrainer() {
+ }
+
+ /**
+ * The training hyperparameters.
+ *
+ * @param embeddingSize The embedding dimensionality. Must be positive.
+ * @param hiddenSize The hidden layer width. Must be positive.
+ * @param epochs The number of passes over the examples. Must be positive.
+ * @param batchSize The minibatch size. Must be positive.
+ * @param learningRate The AdaGrad step size. Must be positive.
+ * @param l2 The L2 penalty applied to the dense weights. Must not be
negative.
+ * @param dropout The hidden dropout probability. Must be in {@code [0, 1)}.
+ * @param wordCutoff The minimum frequency for a word to get its own
embedding. Must
+ * not be negative.
+ * @param seed The random seed making a run reproducible.
+ */
+ public record Settings(int embeddingSize, int hiddenSize, int epochs, int
batchSize,
+ double learningRate, double l2, double dropout, int wordCutoff, long
seed) {
+
+ /** The default embedding dimensionality. */
+ private static final int DEFAULT_EMBEDDING_SIZE = 50;
+
+ /** The default hidden layer width. */
+ private static final int DEFAULT_HIDDEN_SIZE = 200;
+
+ /** The default number of epochs. */
+ private static final int DEFAULT_EPOCHS = 10;
+
+ /** The default minibatch size. */
+ private static final int DEFAULT_BATCH_SIZE = 256;
+
+ /** The default AdaGrad step size. */
+ private static final double DEFAULT_LEARNING_RATE = 0.02;
+
+ /** The default L2 penalty. */
+ private static final double DEFAULT_L2 = 1e-8;
+
+ /** The default hidden dropout probability. */
+ private static final double DEFAULT_DROPOUT = 0.5;
+
+ /** The default minimum word frequency for an embedding row. */
+ private static final int DEFAULT_WORD_CUTOFF = 2;
+
+ /** The default random seed. */
+ static final long DEFAULT_SEED = 17L;
+
+ /**
+ * Validates the hyperparameters.
+ *
+ * @throws IllegalArgumentException Thrown if a value is outside its
documented
+ * range.
+ */
+ public Settings {
+ requirePositive(embeddingSize, "embeddingSize");
+ requirePositive(hiddenSize, "hiddenSize");
+ requirePositive(epochs, "epochs");
+ requirePositive(batchSize, "batchSize");
+ if (embeddingSize > FeedforwardDependencyModel.MAX_EMBEDDING_SIZE) {
+ throw new IllegalArgumentException("embeddingSize exceeds the model
format limit: "
+ + embeddingSize);
+ }
+ if (hiddenSize > FeedforwardDependencyModel.MAX_HIDDEN_SIZE) {
+ throw new IllegalArgumentException("hiddenSize exceeds the model
format limit: "
+ + hiddenSize);
+ }
+ final long minimumModelValues = modelFloatValues(MIN_VOCABULARY_ROWS,
+ MIN_TRANSITIONS, embeddingSize, hiddenSize);
+ if (minimumModelValues >
FeedforwardDependencyModel.MAX_MODEL_FLOAT_VALUES) {
+ throw new IllegalArgumentException("embeddingSize and hiddenSize
exceed the model "
+ + "format allocation limit");
+ }
+ if (ParamChecks.requireFinite(learningRate, "learningRate") <= 0.0) {
+ throw new IllegalArgumentException("learningRate must be positive: " +
learningRate);
+ }
+ ParamChecks.requireNonNegative(ParamChecks.requireFinite(l2, "l2"),
"l2");
+ if (!(dropout >= 0.0 && dropout < 1.0)) {
+ throw new IllegalArgumentException("dropout must be in [0, 1): " +
dropout);
+ }
+ ParamChecks.requireNonNegative(wordCutoff, "wordCutoff");
+ }
+
+ /**
+ * Rejects a hyperparameter that is not positive.
+ *
+ * @param value The value to check.
+ * @param name The hyperparameter name used in the exception message.
+ * @throws IllegalArgumentException Thrown if {@code value} is not
positive.
+ */
+ private static void requirePositive(int value, String name) {
+ if (value <= 0) {
+ throw new IllegalArgumentException(name + " must be positive: " +
value);
+ }
+ }
+
+ /**
+ * @return The default hyperparameters. Never {@code null}.
+ */
+ public static Settings defaults() {
+ return new Settings(DEFAULT_EMBEDDING_SIZE, DEFAULT_HIDDEN_SIZE,
DEFAULT_EPOCHS,
+ DEFAULT_BATCH_SIZE, DEFAULT_LEARNING_RATE, DEFAULT_L2,
DEFAULT_DROPOUT,
+ DEFAULT_WORD_CUTOFF, DEFAULT_SEED);
+ }
+ }
+
+ /**
+ * Trains a model from dependency samples.
+ *
+ * @param samples The training samples. Must not be {@code null}.
+ * @param settings The hyperparameters. Must not be {@code null}.
+ * @return A trained {@link FeedforwardDependencyModel}. Never {@code null}.
+ * @throws IOException Thrown if reading the samples fails.
+ * @throws IllegalArgumentException Thrown if a parameter is {@code null} or
no
+ * trainable example can be derived from the samples.
+ */
+ public static FeedforwardDependencyModel
train(ObjectStream<DependencySample> samples,
Review Comment:
Done. Both `train` overloads document `IllegalStateException` for a
diverging epoch and `IllegalArgumentException` for the cell limit, the
vocabulary and symbol limits, and wrongly sized or non-finite pretrained
vectors; `refine` documents the same `IllegalStateException` plus the beam-size
floor and unknown transitions.
##########
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/depparse/FeedforwardDependencyTrainer.java:
##########
@@ -0,0 +1,1176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.depparse;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.function.Function;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import opennlp.tools.util.ObjectStream;
+import opennlp.tools.util.ParamChecks;
+
+/**
+ * Trains the {@link FeedforwardDependencyModel}: oracle-derived transition
examples,
+ * minibatch AdaGrad over a softmax cross-entropy loss, cube activation, and
inverted
+ * dropout on the hidden layer, the training recipe of
+ * <a href="https://aclanthology.org/D14-1082/";>Chen and Manning (2014)</a>.
+ *
+ * <p>Words below the frequency cutoff share a learned unknown embedding;
absent
+ * template positions share a learned padding embedding. Non-projective
samples have no
+ * arc-standard derivation and are skipped. Training is deterministic for a
fixed
+ * {@link Settings#seed()}.</p>
+ *
+ * @since 3.0.0
+ */
+public final class FeedforwardDependencyTrainer {
+
+ private static final Logger logger =
+ LoggerFactory.getLogger(FeedforwardDependencyTrainer.class);
+
+ /** The AdaGrad denominator offset that keeps the first steps finite. */
+ private static final double ADAGRAD_EPSILON = 1e-6;
+
+ /** The derivative factor of the cube activation, {@code d/dx x^3 = 3x^2}. */
+ private static final double CUBE_DERIVATIVE_FACTOR = 3.0;
+
+ /** The numerator of the Glorot uniform bound, {@code sqrt(6 / (fanIn +
fanOut))}. */
+ private static final double GLOROT_NUMERATOR = 6.0;
+
+ /** The half-width of the uniform range embeddings are drawn from. */
+ private static final double EMBEDDING_INIT_SCALE = 0.01;
+
+ /** The smallest probability the reported training loss takes the log of. */
+ private static final double LOSS_PROBABILITY_FLOOR = 1e-12;
+
+ /** The sentence loss reported when refinement made no update for a
sentence. */
+ private static final double NO_UPDATE = -1.0;
+
+ /** Special-symbol rows present before any training vocabulary is added. */
+ private static final int MIN_VOCABULARY_ROWS = 8;
+
+ /** SHIFT and at least one RIGHT_ARC are required to parse a sentence. */
+ private static final int MIN_TRANSITIONS = 2;
+
+ /** Prevents construction of this utility class. */
+ private FeedforwardDependencyTrainer() {
+ }
+
+ /**
+ * The training hyperparameters.
+ *
+ * @param embeddingSize The embedding dimensionality. Must be positive.
+ * @param hiddenSize The hidden layer width. Must be positive.
+ * @param epochs The number of passes over the examples. Must be positive.
+ * @param batchSize The minibatch size. Must be positive.
+ * @param learningRate The AdaGrad step size. Must be positive.
+ * @param l2 The L2 penalty applied to the dense weights. Must not be
negative.
+ * @param dropout The hidden dropout probability. Must be in {@code [0, 1)}.
+ * @param wordCutoff The minimum frequency for a word to get its own
embedding. Must
+ * not be negative.
+ * @param seed The random seed making a run reproducible.
+ */
+ public record Settings(int embeddingSize, int hiddenSize, int epochs, int
batchSize,
+ double learningRate, double l2, double dropout, int wordCutoff, long
seed) {
+
+ /** The default embedding dimensionality. */
+ private static final int DEFAULT_EMBEDDING_SIZE = 50;
+
+ /** The default hidden layer width. */
+ private static final int DEFAULT_HIDDEN_SIZE = 200;
+
+ /** The default number of epochs. */
+ private static final int DEFAULT_EPOCHS = 10;
+
+ /** The default minibatch size. */
+ private static final int DEFAULT_BATCH_SIZE = 256;
+
+ /** The default AdaGrad step size. */
+ private static final double DEFAULT_LEARNING_RATE = 0.02;
+
+ /** The default L2 penalty. */
+ private static final double DEFAULT_L2 = 1e-8;
+
+ /** The default hidden dropout probability. */
+ private static final double DEFAULT_DROPOUT = 0.5;
+
+ /** The default minimum word frequency for an embedding row. */
+ private static final int DEFAULT_WORD_CUTOFF = 2;
+
+ /** The default random seed. */
+ static final long DEFAULT_SEED = 17L;
+
+ /**
+ * Validates the hyperparameters.
+ *
+ * @throws IllegalArgumentException Thrown if a value is outside its
documented
+ * range.
+ */
+ public Settings {
+ requirePositive(embeddingSize, "embeddingSize");
+ requirePositive(hiddenSize, "hiddenSize");
+ requirePositive(epochs, "epochs");
+ requirePositive(batchSize, "batchSize");
+ if (embeddingSize > FeedforwardDependencyModel.MAX_EMBEDDING_SIZE) {
+ throw new IllegalArgumentException("embeddingSize exceeds the model
format limit: "
+ + embeddingSize);
+ }
+ if (hiddenSize > FeedforwardDependencyModel.MAX_HIDDEN_SIZE) {
+ throw new IllegalArgumentException("hiddenSize exceeds the model
format limit: "
+ + hiddenSize);
+ }
+ final long minimumModelValues = modelFloatValues(MIN_VOCABULARY_ROWS,
+ MIN_TRANSITIONS, embeddingSize, hiddenSize);
+ if (minimumModelValues >
FeedforwardDependencyModel.MAX_MODEL_FLOAT_VALUES) {
+ throw new IllegalArgumentException("embeddingSize and hiddenSize
exceed the model "
+ + "format allocation limit");
+ }
+ if (ParamChecks.requireFinite(learningRate, "learningRate") <= 0.0) {
+ throw new IllegalArgumentException("learningRate must be positive: " +
learningRate);
+ }
+ ParamChecks.requireNonNegative(ParamChecks.requireFinite(l2, "l2"),
"l2");
+ if (!(dropout >= 0.0 && dropout < 1.0)) {
+ throw new IllegalArgumentException("dropout must be in [0, 1): " +
dropout);
+ }
+ ParamChecks.requireNonNegative(wordCutoff, "wordCutoff");
+ }
+
+ /**
+ * Rejects a hyperparameter that is not positive.
+ *
+ * @param value The value to check.
+ * @param name The hyperparameter name used in the exception message.
+ * @throws IllegalArgumentException Thrown if {@code value} is not
positive.
+ */
+ private static void requirePositive(int value, String name) {
Review Comment:
Done. `ParamChecks.requirePositive`, with int and double overloads and
tests, is used for every positive check in the parser and trainer. The local
helper is removed.
##########
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/depparse/FeedforwardDependencyModel.java:
##########
@@ -0,0 +1,942 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.depparse;
+
+import java.io.BufferedInputStream;
+import java.io.BufferedOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReferenceArray;
+
+import opennlp.tools.commons.ThreadSafe;
+import opennlp.tools.util.InvalidFormatException;
+import opennlp.tools.util.ParamChecks;
+import opennlp.tools.util.StringUtil;
+
+/**
+ * The weights of the feedforward transition parser: embeddings for words,
tags, and arc
+ * labels, one hidden layer with cube activation, and a transition output
layer, stored
+ * in a versioned binary format. The architecture follows
+ * <a href="https://aclanthology.org/D14-1082/";>Chen and Manning (2014)</a>.
+ *
+ * <p>The same class scores configurations during training and decoding.
Unknown words
+ * use a learned fallback row; words are matched case-insensitively after
+ * {@link #normalize(String) normalization}.</p>
+ *
+ * <p>Instances returned by {@link FeedforwardDependencyTrainer} are immutable
and safe
+ * to share between threads. {@link FeedforwardDependencyTrainer#refine
refine} updates
+ * an independent copy.</p>
+ *
+ * @see FeedforwardDependencyParser
+ * @see FeedforwardDependencyTrainer
+ * @since 3.0.0
+ */
+@ThreadSafe
+public class FeedforwardDependencyModel {
+
+ /** The format header written before every serialized model; it carries the
format version. */
+ static final String MAGIC = "ONLP-FFDP-1";
+
+ /** The maximum number of (template position, embedding row) pairs the
scoring cache holds. */
+ static final int CACHE_ENTRY_LIMIT = 32_768;
+
+ /** The vocabulary label used in load errors for the word map. */
+ private static final String WORD_VOCABULARY = "word vocabulary";
Review Comment:
Inlined at the throw sites.
##########
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/depparse/FeedforwardDependencyTrainer.java:
##########
@@ -0,0 +1,1176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.depparse;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.function.Function;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import opennlp.tools.util.ObjectStream;
+import opennlp.tools.util.ParamChecks;
+
+/**
+ * Trains the {@link FeedforwardDependencyModel}: oracle-derived transition
examples,
+ * minibatch AdaGrad over a softmax cross-entropy loss, cube activation, and
inverted
+ * dropout on the hidden layer, the training recipe of
+ * <a href="https://aclanthology.org/D14-1082/";>Chen and Manning (2014)</a>.
+ *
+ * <p>Words below the frequency cutoff share a learned unknown embedding;
absent
+ * template positions share a learned padding embedding. Non-projective
samples have no
+ * arc-standard derivation and are skipped. Training is deterministic for a
fixed
+ * {@link Settings#seed()}.</p>
+ *
+ * @since 3.0.0
+ */
+public final class FeedforwardDependencyTrainer {
+
+ private static final Logger logger =
+ LoggerFactory.getLogger(FeedforwardDependencyTrainer.class);
+
+ /** The AdaGrad denominator offset that keeps the first steps finite. */
+ private static final double ADAGRAD_EPSILON = 1e-6;
+
+ /** The derivative factor of the cube activation, {@code d/dx x^3 = 3x^2}. */
+ private static final double CUBE_DERIVATIVE_FACTOR = 3.0;
+
+ /** The numerator of the Glorot uniform bound, {@code sqrt(6 / (fanIn +
fanOut))}. */
+ private static final double GLOROT_NUMERATOR = 6.0;
+
+ /** The half-width of the uniform range embeddings are drawn from. */
+ private static final double EMBEDDING_INIT_SCALE = 0.01;
+
+ /** The smallest probability the reported training loss takes the log of. */
+ private static final double LOSS_PROBABILITY_FLOOR = 1e-12;
+
+ /** The sentence loss reported when refinement made no update for a
sentence. */
+ private static final double NO_UPDATE = -1.0;
+
+ /** Special-symbol rows present before any training vocabulary is added. */
+ private static final int MIN_VOCABULARY_ROWS = 8;
Review Comment:
The reserved symbol arrays are defined once in `FeedforwardNetwork` and the
trainer, the vocabulary check and `isReservedSymbol` all read them.
`MIN_VOCABULARY_ROWS` is replaced by `RESERVED_ROWS`, derived from the arrays.
> Add a feedforward neural dependency parser
> ------------------------------------------
>
> Key: OPENNLP-1991
> URL: https://issues.apache.org/jira/browse/OPENNLP-1991
> Project: OpenNLP
> Issue Type: Improvement
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Add a feedforward neural implementation of the dependency parser API
> introduced by OPENNLP-547. Use OpenNLP’s standard model container, artifact
> serializers, TrainingParameters, factory, and CLI conventions. Include
> deterministic training, greedy and beam decoding, model round-trip tests, and
> held-out Universal Dependencies evaluation. Address the deferred review
> feedback on shared scoring/training logic and parser-owned caches. Implement
> this without new dependencies.
> The list above may still get this ticket to further split. For now this is a
> placeholder to capture the code that was moved out of OPENNLP-547. I'll edit
> after 547 is merged.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
