thomasmueller commented on code in PR #1716:
URL: https://github.com/apache/jackrabbit-oak/pull/1716#discussion_r1760834470
##########
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/NodeDocumentCodec.java:
##########
@@ -49,44 +56,109 @@
* <li>Allows estimating the size of the document while reading it, which
will have a negligible overhead (as compared
* with doing an additional traverse of the object structure to compute the
size).</li>
* </ul>
- *
+ * <p>
* This class must be thread-safe, Mongo uses a single coded implementation
across multiple threads.
- *
*/
public class NodeDocumentCodec implements Codec<NodeDocument> {
+ private final static Logger LOG =
LoggerFactory.getLogger(NodeDocumentCodec.class);
+
+ public static final String
OAK_INDEXER_PIPELINED_NODE_DOCUMENT_FILTER_FILTERED_PATH =
"oak.indexer.pipelined.nodeDocument.filter.filteredPath";
+ public static final String
OAK_INDEXER_PIPELINED_NODE_DOCUMENT_FILTER_SUFFIXES_TO_SKIP =
"oak.indexer.pipelined.nodeDocument.filter.suffixesToSkip";
+ private final String filteredPath =
ConfigHelper.getSystemPropertyAsString(OAK_INDEXER_PIPELINED_NODE_DOCUMENT_FILTER_FILTERED_PATH,
"");
+ private final List<String> suffixesToSkip =
ConfigHelper.getSystemPropertyAsStringList(OAK_INDEXER_PIPELINED_NODE_DOCUMENT_FILTER_SUFFIXES_TO_SKIP,
"",';');
+
// The estimated size is stored in the NodeDocument itself
public final static String SIZE_FIELD = "_ESTIMATED_SIZE_";
+
+ private static class NodeDocumentDecoderContext {
+ long docsDecoded = 0;
+ long dataDownloaded = 0;
+ int estimatedSizeOfCurrentObject = 0;
+ }
+
+ private final NodeDocument emptyNodeDocument;
+
private final MongoDocumentStore store;
private final Collection<NodeDocument> collection;
private final BsonTypeCodecMap bsonTypeCodecMap;
private final DecoderContext decoderContext =
DecoderContext.builder().build();
-
private final Codec<String> stringCoded;
private final Codec<Long> longCoded;
private final Codec<Boolean> booleanCoded;
+ private final NodeDocumentFilter fieldFilter = new
NodeDocumentFilter(filteredPath, suffixesToSkip);
+
+ // Statistics
+ private final AtomicLong totalDocsDecoded = new AtomicLong(0);
+ private final AtomicLong totalDataDownloaded = new AtomicLong(0);
+ private final ThreadLocal<NodeDocumentDecoderContext> perThreadContext =
ThreadLocal.withInitial(NodeDocumentDecoderContext::new);
Review Comment:
I'm not a very big fan of using ThreadLocal because of the problems they can
cause. One is out-of-memory (in an non-obvious way), another is that some web
servers, and possibly also OSGi, can try to reset them (using things like
"setAccessible(true)")... Because if they don't, then the class
(NodeDocumentDecoderContext here) can't be unloaded, and so the classloader
can't be unloaded, and so you get out-of-memory that way.
I think here it is OK. But I wouldn't want to use ThreadLocal in other parts
of Oak. Just FYI. See the comments here:
https://stackoverflow.com/questions/60487341/are-there-any-general-drawbacks-from-using-java-lang-threadlocal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
