Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/5645#discussion_r29211734
--- Diff:
streaming/src/main/java/org/apache/spark/streaming/util/WriteAheadLogSegment.java
---
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.util;
+
+/**
+ * This is an interface that represent the information required by any
implementation of
+ * a WriteAheadLog to read a written record.
+ */
[email protected]
+public interface WriteAheadLogSegment extends java.io.Serializable {
--- End diff --
1. Well, for advanced users who want to implement their own WAL
implementation will have to ensure that the segment info is serializable, no
matter whether we expose an interface or a bytebuffer. In fact, exposing an
interface avoids them from writing the code to serialize and return a
bytebuffer in a usual case, which is easier to user. Also this interface is
expected to be called not faster than 100s of time per second. So does not
require super high serialization efficiency. Even if they want, they can always
make the implementation extend Externalizable.
2. That is a good point. There are easy workaround even if we dont make
this a ByteBuffer. They can put a bytebuffer within their implementation `class
MyWALSegment(byteBuffer: ByteBuffer) extends WALSegment`. Now for people who
dont care about backward compatibility, making it a bytebuffer make it harder
for them to implement. For others who do care about backward compatibility,
they will have to write custom externalization logic either way, while
returning bytebuffer or returning MyWALSegment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
