Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/3070#discussion_r19754487
--- Diff: mllib/pom.xml ---
@@ -46,6 +46,11 @@
<version>${project.version}</version>
</dependency>
<dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-sql_${scala.binary.version}</artifactId>
--- End diff --
@srowen Yes, it feels weird if we say ML depends on SQL, the "query
language". Spark SQL provides RDD with schema support and execution plan
optimization, both of which are need by MLlib. We need flexible table-like
datasets and I/O support, and operations that "carry over" additional columns
during the training phrase. It is natural to say that ML depends on RDD with
schema support and execution plan optimization.
I agree that we should factor the common part out or make SchemaRDD a
first-class citizen in Core, but that definitely takes time for both design and
development. This dependence change has no effect on the content we deliver to
users, and UDTs are internal to Spark.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
