wonook commented on a change in pull request #18: [NEMO-45] Distributed
Nemo-Spark
URL: https://github.com/apache/incubator-nemo/pull/18#discussion_r190471382
##########
File path:
compiler/frontend/spark/src/main/java/edu/snu/nemo/compiler/frontend/spark/source/SparkDatasetBoundedSourceVertex.java
##########
@@ -74,25 +73,30 @@ public SparkBoundedSourceVertex getClone() {
}
/**
- * A Readable for SparkBoundedSourceReadablesWrapper.
+ * A Readable wrapper for Spark Dataset.
*/
- private final class SparkBoundedSourceReadable implements Readable<T> {
+ private final class SparkDatasetBoundedSourceReadable implements Readable<T>
{
private final LinkedHashMap<String, Object[]> commands;
private final Map<String, String> sessionInitialConf;
private final int partitionIndex;
+ private final List<String> locations;
/**
* Constructor.
- * @param commands list of commands needed to build the dataset.
+ *
+ * @param partition the partition to wrap.
+ * @param commands list of commands needed to build the dataset.
* @param sessionInitialConf spark session's initial configuration.
- * @param partitionIndex partition for this readable.
+ * @param partitionIndex partition for this readable.
*/
- private SparkBoundedSourceReadable(final LinkedHashMap<String, Object[]>
commands,
- final Map<String, String>
sessionInitialConf,
- final int partitionIndex) {
+ private SparkDatasetBoundedSourceReadable(final Partition partition,
Review comment:
I think in the long-run, this class should also use the SparkContext as in
the way in the TextFileSourceVertex. Can you add a todo note and open an issue
regarding this bit for me to re-investigate in this part of code?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
